Functional Data Analysis

• Channel: The chat-link FDA group

Introduction

The statistical analysis of spatial time series requires additional methods of data analysis. First, we suppose time is continuous, put the state space changes \(\frac{d\mathbf{x}}{dt}\), and use neural ordinary and stochastic differential equations. Second, we analyze a multivariate and multidimensional time series and use the tensor representation and tensor analysis. Third, since the time series have significant cross-correlation, we model them in the Riemannian space. Fourth, medical time series are periodic; the base model is the pendulum model, \(\frac{d^2x}{dt^2}=-c\sin{x}\). We use physics-informed neural networks to approximate data. Fifth, the practical experiments involve multiple data sources. We use canonical correlation analysis with a latent state space. This space aligns the source and target spaces and generates data in the source and target manifolds.

Applications

This field of Machine Learning applies to any field where the measurements have continuous time and space data acquired from multimodal sources: climate modeling, neural interfaces, solid-state physics, electronics, fluid dynamics, and many more. We will carefully collect both the theory and its practice.

Foundation models for scientific research

Foundation AI models are universal models to solve a wide set of problems. This project proposes to investigate the theoretical properties of foundation models. The domain to model is a spatial-time series. These data are used in various scientific disciplines and serve to generalise scientific knowledge and make forecasts. The essential problems, formulated as user requests that solve a foundation model, are forecasting and generation of time series; analysis and classification of time series; detection of change point, and causal inference. To solve these problems, the foundation AI models are trained on massive datasets. The main goal of this project is to compare various architectures of foundation models to find an optimal architecture that solves the listed problems for a wide range of spatial time series.

Topics to discuss

1. State Space Models, Convolution, SSA, SSM (Spectral Submanifolds)
2. Neural and Controlled ODE, Neural PDE, Geometric Learning
3. Operator Learning, Physics-informed learning, and multimodeling
4. Spatial-Temporal Graph Modeling: Graph convolution and metric tensors
5. Riemmannian models; time series generation
6. AI for science: mathematical modelling principles
7. <it>Left behind:<\it> data-driven tensor analysis, differential forms, and spinors

State of the Art in 2025

The NeurIPS workshop "Foundational models for science" reflected this theme in 2024:

1. Foundation Models for Science: Progress, Opportunities, and Challenges URL
2. Foundation Models for the Earth system UPL, no paper
3. Foundation Methods for foundation models for scientific machine learning URL, no paper
4. AI-Augmented Climate simulators and emulators URL, no paper
5. Provable in-context learning of linear systems and linear elliptic PDEs with transformers NIPS
6. VSMNO: Solving PDE by Utilizing Spectral Patterns of Different Neural Operators NIPS

Physics problem Simulations

1. The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning ArXiv, Code
2. Polymatic Advancing Science through Multi‑Disciplinary AI blog
3. Long Term Memory: The Foundation of AI Self-Evolution ArXiv
4. Distilling Free-Form Natural Laws from Experimental Data, 2009 Science, comment, medium
5. Deep learning for universal linear embeddings of nonlinear dynamics nature
6. A comparison of data-driven approaches to build low-dimensional ocean models, 2021 by Pavel Berloff ArXiv, talk by Daniil Dorin for S.V. Fortova
7. Applications of Deep Learning to Ocean Data Inference and Subgrid Parameterization by Thomas Bolton and Laure Zanna, 2018 preprint, talk by Nilita Kiselev
8. On energy-aware hybrid models by Shevchenko,2024 doi, talk by Mariya Nikitina
9. Science: NASA satellites and computers have provided us with these mesmerizing swirls that cover our planet—but this isn’t star stuff. Each color represents a different aerosol that was floating in the atmosphere above our heads from 1 August to 14 September 2024 video

Spatial-Temporal Graph Modeling

1. Graph WaveNet for Deep Spatial-Temporal Graph Modeling ArXiv
2. Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting ICLR
3. Time-SSM: Simplifying and Unifying State Space Models for Time Series Forecasting ArXiv SSMTool
4. State Space Reconstruction for Multivariate Time Series Prediction ArXiv](Denis)
5. Longitudinal predictive modeling of tau progression along the structural connectome by Joyita Dutta 2021

Key reviews on AI for Science

1. 2018. Diffusion Convolutional Recurrent Neural Network ICLR
2. 2021. Neural Partial Differential Equations with Functional Convolution ICLR
3. 2018. Graph WaveNet for Deep Spatial-Temporal Graph Modeling ArXiV
4. 2021. Neural Rough Differential Equations for Long Time Series (comparison)
5. 2022. Time Series Forecasting Using Manifold Learning, Radial Basis Function Interpolation and Geometric Harmonics doi (all basic models + superpositions review)

Catch-up on LLM

If you are not familiar with the LLM and GPT:

1. Build an LLM from scratch by Sebastian Raschka, 2025 github
2. Agentic Design Patterns by Antonio Gulli, 2025 docx

For fun, the vibe coding: 1, 2, 3, 4, also Foundations of LLM and Karpathy's project nanoGPT

Table of topics for seminars

In these ten weeks, we will discuss the next five topics:

1. Multimodal data
2. Continous time and space models
3. Physics-informed models
4. Multilinear models
5. Riemannian spaces

These items comprise the stochastic-deterministic decomposition. So the questions include three parts:

1. deterministic model,
2. generative model,
3. stochastic-deterministic decomposition method.

Multimodal data

First series

1. Canonical Correlation Analysis
2. CCA in tensor representation
3. Kernel CCA in Hilbert and L2[a,b] spaces
4. CCA versus Cross-Attention Transformers
5. Generative CCA, diffusion, and flow
6. Comparative analysis of variants of CCA, like PLS and others
7. Functional PCA
8. Canonical Correlation Analysis: forecasting model and loss function with variants-
9. CCA parameter estimation algorithm

Talks

1. Canonical Correlation Analysis in tensor representation Marat
2. Kernel CCA in Hilbert and L2[a,b] spaces Bair
3. CCA versus Cross-Attention Transformers Eduard
4. Generative CCA, diffusion, and flow Galina, Galina
5. Functional PCA Parviz

Continous models

1. Neural ODE
2. Continous state space models
3. Continous normalizing flows
4. Adjoint method and continuous backpropagation
5. Neural Delayed Differential Equations
6. Neural CDE (PID control is welcome)
7. Neural PDE
8. S4 and Hippo models [1], [2] (LSSL, SaShiMi, DSS, HTTYH, S4D, and S4ND)
9. Riemannian continuous models

Talks

1. Continuous state space models Bair
2. Continuous normalizing flows Marat
3. Adjoint method and continuous backpropagation Galina
4. Riemannian continuous models Eduard

SINDy

1. Learning partial differential equations via data discovery and sparse optimization by Hayden Schaeffer, 2017 DOI, PDF
2. Data-driven discovery of partial differential equations by Rudy et al. 2017 Science, []
3. Supporting Information for: Discovering governing equations from data by Steven L. Brunton et al. [pnas.1517384113.sapp.pdf PDF]
4. SINDy-PI: a robust algorithm for parallel implicit sparse identification of nonlinear dynamics by Kadierdan Kaheman et al., 2020 DOI
5. Ensemble-SINDy by Fasel et al. 2021 DOI

Connection to NeurODE

1. Hybrid Models: Combining Neural ODEs with Discrete Layers medium
2. ODE manual. Linearization. Особые точки нелинейных систем на плоскости by Ilya Shurov URL Equilibrium points wiki

Physics-Informed models

Third series

1. PINNs as multimodels
2. Spherical harmonics in p dimensions (an IMU example is welcome)
3. PDF and Physics-Informed learning
4. Integral Transforms in Physics-Informed learning

Talks

1. Geometric Clifford Algebra Networks Eduard
2. Integral Transforms in Physics-Informed learning Galina

Multilinear models and topology

Fourth series

1. Clifford or Geometric algebra in machine learning
2. Tensor models, tensor decomposition, and approximation (tensor PLS or CCA)
3. Machine learning models for tensors: Field Equation (Yang-Mills Equations_
4. Machine learning models for theoretical physics (Maxwell’s Equations, Navier-Stokes)
5. Persistent homology and dimensionality reduction (say, arXiv:2302.03447 with embedding delays)

Talks

1. Tensor models, tensor decomposition, and approximation Eduard
2. Machine learning models for theoretical physics (Maxwell’s Equations, Navier-Stocks) Galina

Generative and Riemannian models

Fifth series

1. Generative Riemannian models. How do we extract and use the distribution?
2. Generative Canonical Correlation Analysis and its connection with the Riemannian spaces in the latent part
3. Scoring-based Riemannian models. How do we extract and use the distribution?
4. Generative convolutional models for tensors. Is there a continuous-time? (A variant is the Riemannian Residual Networks).
5. Riemannian continuous normalizing flows. How do we generate a time series of a given distribution?

Talks

1. Scoring-based Riemannian models Eduard
2. Riemannian continuous normalizing flows Galina

Operator learning

An additional topic to summarise all the above. See the introduction in

1. Neural operators wiki
2. Operator Learning: Convolutional Neural Operators blog
3. Convolutional Neural Operators for robust and accurate learning of PDEs arxiv 2023
4. Representation Equivalent Neural Operators: a Framework for Alias-free Operator Learning arxiv 2023
5. PID: Proportional-Integral-Differential-equation modeling with operator learning

Discussed literature

1. Generative CCA, diffusion, and flow by Galina [3] [4] [5] [6]
2. Kernel CCA in Hilbert and L2[a,b] spaces by Bair [7] [8]
3. CCA versus Cross-Attention Transformers by Eduard [9] [10] [11]
4. Ajoint method and continuous backpropagation by Galina [12]
5. Continuous normalizing flows by Galina [13]
6. Tensor models by Eduard [14] [15] [16]
7. Navier-Stokes [17] [18] [19]
8. Classics versus quantum by Galina
   1. Schroedinger vs. Navier–Stokes 2016
   2. Many-particle quantum hydrodynamics: Exact equations and pressure tensors 2019
   3. Quantum hydrodynamics, Wigner transforms, the classical limit 1995
   4. Geometry of Nonadiabatic Quantum Hydrodynamics 2019
   5. Theory of quantum friction 2014
   6. Minimal quantum viscosity from fundamental physical constants
   7. Fluid Dynamics with Incompressible Schrödinger Flow 2017
   8. Гидродинамика Шрёдингера на пальцах
9. Riemannian continuous normalizing flows by Galina [20] [21] [22]

Practical spatial-time series

Datasets

1. ClimateSet, 2023 ArXiv
2. A guide to state–space modeling of ecological time series, 2021 PDF, (Bayesian Kalman)
3. Kalman Filtering and Smoothing, 2025 ArXiv (Riemannian Kalman)

General

1. Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems arxiv 2023
2. Algebra, Topology, Differential Calculus, and Optimization Theory For Computer Science and Machine Learning upenn 2024
3. The Elements of Differentiable Programming arxiv 2024
4. The list from the previous year 2023.
5. Differential Geometry of Curves and Surfaces: Textbook, 2016 by Kristopher Tapp [23]

Basic literature

1. Understanding Deep Learning by Simon J.D. Prince mit 2023
2. Deep Learning by C.M. and H. Bishops Springer 2024 (online version)
3. A Geometric Approach to Differential Forms by David Bachman arxiv 2013
4. Advanced Calculus: Geometric View by James J. Callahan pdf 2010, collection
5. Geometric Deep Learning by Michael M. Bronstein arxiv 2021

Linear and bilinear models

1. A Tutorial on Independent Component Analysis arxiv, 2014
2. On the Stability of Multilinear Dynamical Systems arxiv 2022
3. Tensor-based Regression Models and Applications by Ming Hou Thèse Uni-Laval 2017
4. Tensor Canonical Correlation Analysis for Multi-view Dimension Reduction [24] (Semkin)
5. Tensor Learning in Multi-view Kernel PCA arxiv 2018
6. Tensor decomposition of EEG signals: A brief review 2015

Spherical Harmonics

1. Spherical Harmonic Transforms: In JAX and PyTorch Medium 2024
2. Spherical Harmonics in p Dimensions arxiv 2012
3. Physics of simple pendulum: a case study of nonlinear dynamics RG 2008
4. Time series forecasting using manifold learning, 2021 arxiv
5. Time-series forecasting using manifold learning, radial basis function interpolation, and geometric harmonics 2022 Chaos AIP

State Space Models

1. Missing Slice Recovery for Tensors Using a Low-rank Model in Embedded Space arxiv 2018
2. Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks by A.R. Voelker et al., 2019 NeurIPS

SSM Generative Models

1. Masked Autoregressive Flow for Density Estimation arxiv 2017

SSM+Riemann+Gaussian process regression

• Time-series forecasting using manifold learning, radial basis function interpolation, and geometric harmonics by Ioannis G. Kevrekidis,3 and Constantinos Siettos, 2022 pdf

Physics-Informed Neural Networks

1. Neural partial differential equations with functional convolution ICLP
2. Solving PDEs by variational physics-informed neural networks: an a posteriori error analysis PDF plus several links to the books on the subject inside
3. Predicting the nonlinear dynamics of spatiotemporal PDEs via physics-informed informer networks PDF
4. Three ways to solve partial differential equations with neural networks — A review arxiv 2021
5. NeuPDE: Neural Network Based Ordinary and Partial Differential Equations for Modeling Time-Dependent Data arxiv 2019
6. Physics-based deep learning code
7. PINN by Steve Burton yt
8. Process Model Inversion in the Data-Driven Engineering Context for Improved Parameter Sensitivities mdpi processes 2022 (nice connection pictures)
9. Physics-based Deep Learning github
10. Integral Transforms in a Physics-Informed (Quantum) Neural Network setting arxiv 2022
11. Lectures ny Stephen Brunton AI/ML+Physics, Part 4, Basic PDEs, PDE Overview

PINN Libraries

1. PINA Gianluigi Rozza at SISSA MathLab www see tutorials and solvers

Riemmanian models

1. Riemannian Continuous Normalizing Flows arxiv 2020
2. Residual Riemannian Networks arxiv 2023

Continous time, Neural ODE

1. Neural Spatio-Temporal Point Processes by Ricky Chen et al. iclr 2021 (likelihood for time and space)
2. Neural Ordinary Differential Equations by Ricky Chen et al. arxiv 2018 torchdiffeq github
3. Neural Controlled Differential Equations for Irregular Time Series 'Patrick Kidger et al. arxiv 2020github
4. On Neural Differential Equations by Patrick Kidger arxiv 2021
5. Diffusion Normalizing Flow arxiv 2021
6. Differentiable Programming for Differential Equations: A Review arxiv 2024
7. (code tutorial) Deep Implicit Layers - Neural ODEs, Deep Equilibrium Models, and Beyond nips 2020
8. (code tutorial) 2021
9. Neural CDE and tensors IEEE, IEEE
10. Latent ODEs for Irregularly-Sampled Time Series 2019
11. Apprentissage et calcul scientifique by Emmanuel Franck www draft of a texbook, chapter 11.4
12. Adjoint State Method, Backpropagation and Neural ODEs by Ilya Schurov www

Graph and PDEs

1. Fourier Neural Operator for Parametric Partial Differential Equations arxiv 2020
2. Masked Attention is All You Need for Graphs arxiv 2024

Neural SDE

1. Approximation of Stochastic Quasi-Periodic Responses of Limit Cycles in Non-Equilibrium Systems under Periodic Excitations and Weak Fluctuations mdpi entropy 2017 (great illustrations on the stochastic nature of a simple phase trajectory)
2. Approximation of Stochastic Quasi-Periodic Responses of Limit Cycles in Non-Equilibrium Systems under Periodic Excitations and Weak Fluctuations mdpi entropy 2017 (great illustrations on the stochastic nature of a simple phase trajectory)
3. Neural SDEs for Conditional Time Series Generation arxiv 2023 code github LSTM - CSig-WGAN
4. Neural SDEs as Infinite-Dimensional GANs 2021
5. Efficient and Accurate Gradients for Neural SDEs by Patrick Kidger arxiv 2021 code diffrax

Chains and homology

1. Operator Learning: Algorithms and Analysis arxiv 2024
2. Hi-res weather: Operator learning arxiv 2022
3. Homotopy theory for beginners by J.M. Moeller ku.dk 2015 (is it a pertinent link?)
4. Explorations in Homeomorphic Variational Auto-Encoding arxiv 2018
5. Special Finite Elements for Dipole Modelling master thesis Bauer 2011
6. Selecting embedding delays: An overview of embedding techniques and a new method using persistent homology arxiv 2023 (Denis)
7. (code) Clifford Algebra for Python https://clifford.readthedocs.io/en/v1.1.0/

Appendix

1. Neural Memory Networks stanford reports 2019
2. An Elementary Introduction to Information Geometry by Frank Nielsen [An Elementary Introduction to Information Geometry Frank Nielsen mdpi entropy
3. The Many Faces of Information Geometry by Frank Nielsen ams 2022 (short version)
4. Geometric Clifford Algebra Networks arxiv 3022
5. Clifford Algebras and Dimensionality Reduction for Signal Separation by M. Guillemard Uni-Hamburg 2010code
6. Special Finite Elements for Dipole Modelling by Martin Bauer Master Thesis Erlangen 2012 diff p-form must read
7. Bayesian model selection for complex dynamic systems 2018
8. Visualizing 3-Dimensional Manifolds by Dugan J. Hammock 2013 umass
9. At the Interface of Algebra and Statistics by T-D. Bradley arxiv 2020
10. Time Series Handbook by Borja, 2021 github
11. Physics-informed machine learning Nature reviews: Physics 2021
12. Integral Transforms in a Physics-Informed (Quantum) Neural Network setting: Applications & Use-Cases arxiv 2022
13. Deep Efficient Continuous Manifold Learning for Time Series Modeling arxiv 2021

Causality

1. Toward Causal Representation Learning 2021
2. See the Sugihara collection

Basics

Collection of wiki-links

Time Series

1. Spectral submanifold (with nonlinear dimensional reduction like som)
2. Lagrangian coherent structure (software below)

Signal Processing

1. Estimation of signal parameters via rotational invariance techniques
2. Reproducing kernel Hilbert space
3. Kernel principal component analysis
4. Gram matrix
5. Generalized pencil-of-function method
6. Wavelet transform

Differential Geometry

1. Pushforward (differential)
2. Ffibers, Bundles, Sheaves
3. Homology
4. Topological data analysis
5. Conditional mutual information
6. Convergent cross mapping
7. Differential form
8. The total derivative as a differential form
9. #Riemannian_metrics Riemannian_metrics
10. Multidimensional Differential and Integral Calculus: A Practical Approach (textbook)

Probabilistical Decompisition

1. Wasserstein metric
2. Mutual information
3. Jacobian
4. Fisher information
5. also dobrushin stratonovich wasserstein
6. also fluid dynamics, transportation theory

Tutorials

1. Connected papers search
2. Operator Learning via Physics-Informed DeepONet: Let’s Implement It From Scratch Medium

Tools

1. icebeem
2. ivae
3. fmri-component
4. analysis/blob/master/VAE_for_fMRI/dataset/train/Bystrova0_y-axis.png
5. Neural ODE in Matlab
6. pyRiemann
7. causality inference peps
8. LMM grok-1 with weights

Turbulence

1. Runko: Modern multiphysics toolbox for plasma simulations GitHub
2. 2d-turb-PINN by Parfenyev GitHub

Physics and Engineering of Turbulence

1. Fundamentals of Fluid_Mechanics, 2013 PDF
2. Introduction ot Fluid Mechanics, 2004 by R. Fox et al. PDF
3. Computational fluid dynamics, 1995 by John D. Anderson, Jr. PDF
4. Fluid-dynamic drag, 1965 by S.F. Hoerner PDF
5. TorchDyn: A Neural Differential Equations Library arXiv github
   1. Parameterized Neural Ordinary Differential Equations: Applications to Computational Physics Problems ArXiv
   2. Turbulence forecasting via Neural ODE ArXiv
   3. (not the same) Hamiltonian Neural Networks ArXiv

State Space Reconstruction

(out of this topic)

1. The false nearest neighbors algorithm by Carl Rhodes doi 1997, Carl's another paper
2. Use of False Nearest Neighbours for Selecting Variables andEmbedding Parameters for State Space Reconstruction by Anna Krakovská et al. doi 2014
3. Estimating a Minimum Embedding Dimension by False Nearest Neighbors Method without an Arbitrary Threshold doi 2022

Author’s Name: Kohki Nakane1,a), Akihiro Sugiura2, Hiroki Takada1

1. ODE. Differential manifolds by Vladimir Arnold (last chapter of the textbook)
2. ODE by Ilya Shchurov www
3. State-space representation wiki
4. Phase space wiki

Collection

1. Time-SSM: Simplifying and Unifying State Space Models for Time Series Forecasting arxiv
2. Time-series forecasting using manifold learning, radial basis function interpolation, and geometric harmonics [arxiv]
3. srush/annotated-s4
4. Modeling Nonlinear Dynamics from Equations and Data by George Haller book youtube