Functional Data Analysis

Intelligent Data Analysis 2024

The chat-link

The statistical analysis of spatial time series requires additional methods of data analysis. First, we suppose time is continuous, put to the state space changes \(\frac{d\mathbf{x}}{dt}\) and use neural ordinary and stochastic differential equations. Second, we analyze a multivariate and multidimensional time series and use the tensor representation and tensor analysis. Third, since the time series have significant cross-correlation we model them in the Riemannian space. Fourth, medical time series are periodic, the base model is the pendulum model, \(\frac{d^2x}{dt^2}=-c\sin{x}\). We use physics-informed neural networks to approximate data. Fifth, the practical experiments involve multiple data sources. We use canonical correlation analysis with latent state space. This space aligns the source and target spaces and generates data in source and target manifolds.

Applications

This field of Machine Learning applies to any field where the measurements have continuous time and space data acquired from multimodal sources: climate modeling, neural interfaces, solid-state physics, electronics, fluid dynamics, and many more. We will carefully collect both the theory and its practice.

Your profit

Your goal is to enhance your abilities to convey messages to the reader in the language of applied mathematics. The main part of your MS thesis work is the theoretical foundations of Machine Learning, where you present your personal results supported by the necessary theory.

Structure of a seminar

The semester has 10 weeks, and five couple of weeks for homework.

• Odd week: introduction to the topic and handout of a theme for the homework.
• Even week: a discussion of the essay, collecting the list of improvements to each essay.
• Odd week: a discussion of the improved essay, putting the essays into a joint structure.

Scoring

Each essay brings one point, and each improvement brings one point. If an easy is perfect, no improvement is required, it counts as one plus one point. The threshold for binary decision is seven points.

The homework

The course gives two credits, so it requires time. The result is a two-page essay. It delivers an introduction to the designated topic. It could be automatically generated or collected from Wikipedia. The main requirement is that you be responsible for each statement of your essay. Each formula is yours.

The essay carries a comprehensive and strict answer to the topic question, illustrative plots are welcome. The result is ready to compile in a joint manuscript after the Even week. So please use the LaTeX template.

The style is the set theory, algebra, analysis, and Bayesian statistics. Category theory and homotopy theory are welcome.

This course gives you two credits, so it is 76/10 = 5 hours of weekly homework.

Templated and links

• The course Git Hub to download the homework essays
• The overleaf to compile the joint manuscript
• The LaTeX template for an essay
• The course chat to ask questions

Requirements for the text and the discussion

1. Comprehensive explanation of the method or the question we discuss
2. Only the principle, no experiments
3. Two-page text (more or less)
4. The reader is a second or third-year student
5. The picture is obligatory
6. However, a brief reference to some deep learning structure is welcome
7. Talk could be a slide or a text itself
8. The list of references with doi
9. Tell how it was generated
10. Observing a gap, put a note about it (to question later)

Style remarks for the essays

Automatic generation of mediocre-quality texts increased requirements for the quality of the new messages. It makes novelty rare and makes the authorship appreciated. But it simplifies the ways of delivering. So since textbook generation has become simple, we will use generative chats to train our skills of reader persuasion. The reader is our MS-thesis defense committee.

Additional remarks for clarification. Люди уже придумали все необходимое. Когда-то давно человечество развивалось очень бурно – постоянно менялись не только вещи, окружавшие людей, но и слова, которыми они пользовались. В те дни было много разных названий для творческого человека - инженер, поэт, ученый. И все они постоянно изобретали новое. Но это было детство человечества. А потом оно достигло зрелости. Творчество не исчезло - но оно стало сводиться к выбору из уже созданного. Говоря образно, мы больше не выращиваем виноград. Мы посылаем за бутылкой в погреб. Людей, которые занимаются этим, называют "сомелье". (В. Пелевин)

Avoid this style (reserved for the seminar)

1. CCA comprehensive overwiev
2. PCA tutorial

Table of homeworks

These ten weeks we discuss the next five topics:

1. Multimodal data
2. Continous time and space models
3. Physics-informed models
4. Riemannian spaces
5. Multilinear models

Note that all these items enlighten stochastic-deterministic decomposition. So the questions include three parts:

1. deterministic model,
2. generative model,
3. stochastic-deterministic decomposition method.

See the questions below for your reference.

1. Multimodal data

First series of questions

1. Canonical Correlation Analysis
2. Canonical Correlation Analysis in tensor representation
3. Kernel CCA in Hilbert and L2[a,b] spaces
4. CCA versus Cross-Attention Transformers
5. Generative CCA, diffusion, and flow
6. Comparative analysis of variants of CCA like PLS and others
7. Functional PCA

Continous models

Second series of questions <!== # Neural ODE ==>

1. Continous state space models
2. Continous normalizing flows
3. Ajoint method and continuous backpropagation
4. Neural CDE (PID control is welcome)
5. Neural PDE
6. S4 and Hippo models
7. Rimannian continuous models

General

1. Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems arxiv 2023
2. Algebra, Topology, Differential Calculus, and Optimization Theory For Computer Science and Machine Learning upenn 2024
3. The Elements of Differentiable Programming arxiv 2024
4. The list from the previous year 2023.

Prerequisites

1. Understanding Deep Learning by Simon J.D. Prince mit 2023
2. Deep Learning by C.M. and H. Bishops Springer 2024 (online version)
3. A Geometric Approach to Differential Forms by David Bachman arxiv 2013
4. Advanced Calculus: Geometric View by James J. Callahan pdf 2010, collection

Linear and bilinear models

1. A Tutorial on Independent Component Analysis arxiv, 2014
2. On the Stability of Multilinear Dynamical Systems arxiv 2022
3. Tensor-based Regression Models and Applications by Ming Hou Thèse Uni-Laval 2017

Spherical Harmonics

1. Spherical Harmonics in p Dimensions arxiv 2012
2. Physics of simple pendulum a case study of nonlinear dynamics RG 2008

State Space Models

1. Missing Slice Recovery for Tensors Using a Low-rank Model in Embedded Space arxiv 2018

SSM Generative Models

1. Masked Autoregressive Flow for Density Estimation arxiv 2017

SSM+Riemann+Gaussian process regression

• Time-series forecasting using manifold learning, radial basis function interpolation, and geometric harmonics by Ioannis G. Kevrekidis,3 and Constantinos Siettos, 2022 pdf

Physics-Informed Neural Networks

1. Three ways to solve partial differential equations with neural networks — A review arxiv 2021
2. NeuPDE: Neural Network Based Ordinary and Partial Differential Equations for Modeling Time-Dependent Data arxiv 2019
3. Physics-based deep learning code
4. PINN by Steve Burton yt
5. Process Model Inversion in the Data-Driven Engineering Context for Improved Parameter Sensitivities mdpi processes 2022 (nice connection pictures)
6. Physics-based Deep Learning github

Riemmanian models

1. Riemannian Continuous Normalizing Flows arxiv 2020

Continous time, Neural ODE

1. Neural Spatio-Temporal Point Processes by Ricky Chen et al. iclr 2021 (likelihood for time and space)
2. Neural Ordinary Differential Equations by Ricky Chen et al. arxiv 2018
3. Neural Controlled Differential Equations for Irregular Time Series 'Patrick Kidger et al. arxiv 2020github
4. Diffusion Normalizing Flow arxiv 2021
5. Differentiable Programming for Differential Equations: A Review arxiv 2024
6. (code tutorial) Deep Implicit Layers - Neural ODEs, Deep Equilibirum Models, and Beyond nips 2020
7. (code tutorial) 2021
8. Neural CDE and tensors IEEE, IEEE

Graph and PDEs

1. Fourier Neural Operator for Parametric Partial Differential Equations arxiv 2020
2. Masked Attention is All You Need for Graphs arxiv 2024

Neural SDE

1. Approximation of Stochastic Quasi-Periodic Responses of Limit Cycles in Non-Equilibrium Systems under Periodic Excitations and Weak Fluctuations mdpi entropy 2017 (great illustrations on the stochastic nature of a simple phase trajectory)
2. Approximation of Stochastic Quasi-Periodic Responses of Limit Cycles in Non-Equilibrium Systems under Periodic Excitations and Weak Fluctuations mdpi entropy 2017 (great illustrations on the stochastic nature of a simple phase trajectory)
3. Neural SDEs for Conditional Time Series Generation arxiv 2023 code github LSTM - CSig-WGAN
4. Neural SDEs as Infinite-Dimensional GANs 2021
5. Efficient and Accurate Gradients for Neural SDEs by Patrick Kidger arxiv 2021 code diffrax

Chains and homology

1. Operator Learning: Algorithms and Analysis arxiv 2024
2. Homotopy theory for beginners by J.M. Moeller ku.dk 2015 (is it a pertinent link?)
3. Explorations in Homeomorphic Variational Auto-Encoding arxiv 2018
4. Special Finite Elements for Dipole Modelling master thesis Bauer 2011

Appendix

1. Neural Memory Networks stanford reports 2019
2. An Elementary Introduction to Information Geometry by Frank Nielsen [An Elementary Introduction to Information Geometry Frank Nielsen mdpi entropy
3. The Many Faces of Information Geometry by Frank Nielsen ams 2022 (short version)
4. Clifford Algebras and Dimensionality Reduction for Signal Separation by M. Guillemard Uni-Hamburg 2010code
5. Special Finite Elements for Dipole Modelling by Martin Bauer Master Thesis Erlangen 2012 diff p-form must read
6. Bayesian model selection for complex dynamic systems 2018
7. Visualizing 3-Dimensional Manifolds by Dugan J. Hammock 2013 umass
8. At the Interface of Algebra and Statistics by T-D. Bradley arxiv 2020
9. Time Series Handbook by Borja, 2021 github