Functional Data Analysis

The chat-link

The statistical analysis of spatial time series requires additional methods of data analysis. First, we suppose time is continuous, put to the state space changes \(\frac{d\mathbf{x}}{dt}\) and use neural ordinary and stochastic differential equations. Second, we analyze a multivariate and multidimensional time series and use the tensor representation and tensor analysis. Third, since the time series have significant cross-correlation we model them in the Riemannian space. Fourth, medical time series are periodic, the base model is the pendulum model, \(\frac{d^2x}{dt^2}=-c\sin{x}\). We use physics-informed neural networks to approximate data. Fifth, the practical experiments involve multiple data sources. We use canonical correlation analysis with latent state space. This space aligns the source and target spaces and generates data in source and target manifolds.

Applications

This field of Machine Learning applies to any field where the measurements have continuous time and space data acquired from multimodal sources: climate modeling, neural interfaces, solid-state physics, electronics, fluid dynamics, and many more. We will carefully collect both the theory and its practice.

Course arrangement

Your profit

Your goal is to enhance your abilities to convey messages to the reader in the language of applied mathematics. The main part of your MS thesis work is the theoretical foundations of Machine Learning, where you present your personal results supported by the necessary theory.

Structure of a seminar

The semester has 10 weeks, and five couple of weeks for homework.

• Odd week: introduction to the topic and handout of a theme for the homework.
• Even week: a discussion of the essay, collecting the list of improvements to each essay.
• Odd week: a discussion of the improved essay, putting the essays into a joint structure.

Scoring

Each essay brings one point, and each improvement brings one point. If an easy is perfect, no improvement is required, it counts as one plus one point. The threshold for binary decision is seven points.

The homework

The course gives two credits, so it requires time. The result is a two-page essay. It delivers an introduction to the designated topic. It could be automatically generated or collected from Wikipedia. The main requirement is that you be responsible for each statement of your essay. Each formula is yours.

The essay carries a comprehensive and strict answer to the topic question, illustrative plots are welcome. The result is ready to compile in a joint manuscript after the Even week. So please use the LaTeX template.

The style is the set theory, algebra, analysis, and Bayesian statistics. Category theory and homotopy theory are welcome.

This course gives you two credits, so it is 76/10 = 5 hours of weekly homework.

Templated and links

• The course Git Hub to download the homework essays
• The overleaf to compile the joint manuscript
• The LaTeX template for an essay
• The course chat to ask questions

Requirements for the text and the discussion

1. Comprehensive explanation of the method or the question we discuss
2. Only the principle, no experiments
3. Two-page text (more or less)
4. The reader is a second or third-year student
5. The picture is obligatory
6. However, a brief reference to some deep learning structure is welcome
7. Talk could be a slide or a text itself
8. The list of references with doi
9. Tell how it was generated
10. Observing a gap, put a note about it (to question later)

Style remarks for the essays

Automatic generation of mediocre-quality texts increased requirements for the quality of the new messages. It makes novelty rare and makes the authorship appreciated. But it simplifies the ways of delivering. So since textbook generation has become simple, we will use generative chats to train our skills of reader persuasion. The reader is our MS-thesis defense committee.

Additional remarks for clarification. Люди уже придумали все необходимое. Когда-то давно человечество развивалось очень бурно – постоянно менялись не только вещи, окружавшие людей, но и слова, которыми они пользовались. В те дни было много разных названий для творческого человека - инженер, поэт, ученый. И все они постоянно изобретали новое. Но это было детство человечества. А потом оно достигло зрелости. Творчество не исчезло - но оно стало сводиться к выбору из уже созданного. Говоря образно, мы больше не выращиваем виноград. Мы посылаем за бутылкой в погреб. Людей, которые занимаются этим, называют "сомелье". (В. Пелевин)

Avoid this style (reserved for the seminar)

1. CCA comprehensive overwiev
2. PCA tutorial

Table of homeworks

These ten weeks we discuss the next five topics:

1. Multimodal data
2. Continous time and space models
3. Physics-informed models
4. Multilinear models
5. Riemannian spaces

Note that all these items enlighten stochastic-deterministic decomposition. So the questions include three parts:

1. deterministic model,
2. generative model,
3. stochastic-deterministic decomposition method.

See the questions below for your reference.

Multimodal data

First series

1. Canonical Correlation Analysis
2. CCA in tensor representation
3. Kernel CCA in Hilbert and L2[a,b] spaces
4. CCA versus Cross-Attention Transformers
5. Generative CCA, diffusion, and flow
6. Comparative analysis of variants of CCA like PLS and others
7. Functional PCA

Continous models

Second series

1. Neural ODE
2. Continous state space models
3. Continous normalizing flows
4. Ajoint method and continuous backpropagation
5. Neural Delayed Differential Equations
6. Neural PDE
7. S4 and Hippo models
8. Rimannian continuous models

Physics-Informed models

Third series

1. PINNs as multimodels
2. Spherical harmonics in p dimensions (an IMU example is welcome)
3. PDF and Physics-Informed learning
4. Integral Transforms in Physics-Informed learning

Multilinear models and topology

Fourth series

1. Cliffort or Geometric algebra in machine learning
2. Tensor models, tensor decomposition, and approximation (tensor PLS pr CCA)
3. Machine learning models for tensors: Field Equation (Yang-Mills Equations_
4. Machine learning models for theoretical physics (Maxwell’s Equations, Navier-Stocks)
5. Persistent homology and dimensionality reduction (say, arXiv:2302.03447 with embedding delays)

Generative and Riemannian models

Fifth series

1. Genertive Riemannian models. How do we extract and use the distribution?
2. Gererative Canonical Correlation Analysis and its connection with the Riemannian spaces in the latent part
3. Scoring-based Riemannian models. How do we extract and use the distribution?
4. Generative convolutional models for tensors. Is there a continuous-time? (A variant is the Riemannian Residual Networks).
5. Riemannian continuous normalizing flows. How do we generate a time series of a given distribution?

References

General

1. Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems arxiv 2023
2. Algebra, Topology, Differential Calculus, and Optimization Theory For Computer Science and Machine Learning upenn 2024
3. The Elements of Differentiable Programming arxiv 2024
4. The list from the previous year 2023.

Prerequisites

1. Understanding Deep Learning by Simon J.D. Prince mit 2023
2. Deep Learning by C.M. and H. Bishops Springer 2024 (online version)
3. A Geometric Approach to Differential Forms by David Bachman arxiv 2013
4. Advanced Calculus: Geometric View by James J. Callahan pdf 2010, collection
5. Geometric Deep Learning by Michael M. Bronstein arxiv 2021

Linear and bilinear models

1. A Tutorial on Independent Component Analysis arxiv, 2014
2. On the Stability of Multilinear Dynamical Systems arxiv 2022
3. Tensor-based Regression Models and Applications by Ming Hou Thèse Uni-Laval 2017
4. Tensor Canonical Correlation Analysis for Multi-view Dimension Reduction [1] (Semkin)

Spherical Harmonics

1. Spherical Harmonics in p Dimensions arxiv 2012
2. Physics of simple pendulum a case study of nonlinear dynamics RG 2008
3. Time series forecasting using manifold learning, 2021 arxiv
4. Time-series forecasting using manifold learning, radial basis function interpolation, and geometric harmonics 2022 Chaos AIP

State Space Models

1. Missing Slice Recovery for Tensors Using a Low-rank Model in Embedded Space arxiv 2018

SSM Generative Models

1. Masked Autoregressive Flow for Density Estimation arxiv 2017

SSM+Riemann+Gaussian process regression

• Time-series forecasting using manifold learning, radial basis function interpolation, and geometric harmonics by Ioannis G. Kevrekidis,3 and Constantinos Siettos, 2022 pdf

Physics-Informed Neural Networks

1. Three ways to solve partial differential equations with neural networks — A review arxiv 2021
2. NeuPDE: Neural Network Based Ordinary and Partial Differential Equations for Modeling Time-Dependent Data arxiv 2019
3. Physics-based deep learning code
4. PINN by Steve Burton yt
5. Process Model Inversion in the Data-Driven Engineering Context for Improved Parameter Sensitivities mdpi processes 2022 (nice connection pictures)
6. Physics-based Deep Learning github
7. Integral Transforms in a Physics-Informed (Quantum) Neural Network setting arxiv 2022

Riemmanian models

1. Riemannian Continuous Normalizing Flows arxiv 2020
2. Residual Riemannian Networks arxiv 2023

Continous time, Neural ODE

1. Neural Spatio-Temporal Point Processes by Ricky Chen et al. iclr 2021 (likelihood for time and space)
2. Neural Ordinary Differential Equations by Ricky Chen et al. arxiv 2018
3. Neural Controlled Differential Equations for Irregular Time Series 'Patrick Kidger et al. arxiv 2020github
4. Diffusion Normalizing Flow arxiv 2021
5. Differentiable Programming for Differential Equations: A Review arxiv 2024
6. (code tutorial) Deep Implicit Layers - Neural ODEs, Deep Equilibirum Models, and Beyond nips 2020
7. (code tutorial) 2021
8. Neural CDE and tensors IEEE, IEEE

Graph and PDEs

1. Fourier Neural Operator for Parametric Partial Differential Equations arxiv 2020
2. Masked Attention is All You Need for Graphs arxiv 2024

Neural SDE

1. Approximation of Stochastic Quasi-Periodic Responses of Limit Cycles in Non-Equilibrium Systems under Periodic Excitations and Weak Fluctuations mdpi entropy 2017 (great illustrations on the stochastic nature of a simple phase trajectory)
2. Approximation of Stochastic Quasi-Periodic Responses of Limit Cycles in Non-Equilibrium Systems under Periodic Excitations and Weak Fluctuations mdpi entropy 2017 (great illustrations on the stochastic nature of a simple phase trajectory)
3. Neural SDEs for Conditional Time Series Generation arxiv 2023 code github LSTM - CSig-WGAN
4. Neural SDEs as Infinite-Dimensional GANs 2021
5. Efficient and Accurate Gradients for Neural SDEs by Patrick Kidger arxiv 2021 code diffrax

Chains and homology

1. Operator Learning: Algorithms and Analysis arxiv 2024
2. Hires weather: Operator realning arxiv 2022
3. Homotopy theory for beginners by J.M. Moeller ku.dk 2015 (is it a pertinent link?)
4. Explorations in Homeomorphic Variational Auto-Encoding arxiv 2018
5. Special Finite Elements for Dipole Modelling master thesis Bauer 2011
6. Selecting embedding delays: An overview of embedding techniques and a new method using persistent homology arxiv 2023 (denis)

Appendix

1. Neural Memory Networks stanford reports 2019
2. An Elementary Introduction to Information Geometry by Frank Nielsen [An Elementary Introduction to Information Geometry Frank Nielsen mdpi entropy
3. The Many Faces of Information Geometry by Frank Nielsen ams 2022 (short version)
4. Clifford Algebras and Dimensionality Reduction for Signal Separation by M. Guillemard Uni-Hamburg 2010code
5. Special Finite Elements for Dipole Modelling by Martin Bauer Master Thesis Erlangen 2012 diff p-form must read
6. Bayesian model selection for complex dynamic systems 2018
7. Visualizing 3-Dimensional Manifolds by Dugan J. Hammock 2013 umass
8. At the Interface of Algebra and Statistics by T-D. Bradley arxiv 2020
9. Time Series Handbook by Borja, 2021 github

Basics

Collection of wiki-links

Signal Processing

1. Estimation of signal parameters via rotational invariance techniques
2. Reproducing kernel Hilbert space
3. Kernel principal component analysis
4. Gram matrix
5. Generalized pencil-of-function method
6. Wavelet transform

Differential Geometry

1. Pushforward (differential)
2. Ffibers, Bundles, Sheaves
3. Homology
4. Topological data analysis
5. Conditional mutual information
6. Convergent cross mapping
7. Differential form
8. The total derivative as a differential form
9. #Riemannian_metrics Riemannian_metrics

Probabilistical Decompisition

1. Wasserstein metric
2. Mutual information
3. Jacobian
4. Fisher information
5. also see wiki dobrushin stratonovich wasserstein

Tools

1. icebeem
2. ivae
3. fmri-component
4. analysis/blob/master/VAE_for_fMRI/dataset/train/Bystrova0_y-axis.png