{{#seo: |title=AI for applied scientific research|titlemode=append|keywords=Machine Learning, Signal processing, Quantum computing, Causal Inference|description=This research management course immerses students in research activities that produce scientific papers with code}}

[[File:Miai logo1.jpeg|class=img-responsive|left|alt=My first scientific paper|link=Course_schedule]]  
{{Box|Title=News|Content={{News}}}}

== My first scientific paper ==
This course produces student research papers. It gathers research teams. Each team consists of a student, a consultant, and an expert. The student is a project driver who wants to plunge into scientific research. The graduate student consultant conducts their research and helps. The expert, a professor, states the problem and enlightens the way to the goal. The projects start in February and end in May, according to the [[Course schedule|schedule]].

*[[Week 0|Week 0: Sign up]]
*[[Week 1|Week 1: Set the toolbox]]
*[[Week 2|Week 2: Tell about your project]]
*[[Week 3|Week 3: State your problem]]
*[[Week 4|Week 4: Plan the experiment]]
*[[Week 5|Week 5: Visualise the principle]]
*[[Week 6|Week 6: Write the theory]]
*[[Week 7|Week 7: Analyse the error]]
*[[Week 8|Week 8: Construct your paper]]
*[[Week 9|Week 9: Review a paper]]
*[[Week 10|Week 10: Select a journal to submit]]
*[[Week 11|Week 11: Prepare your presentation]]
*[https://www.youtube.com/watch?v=uwcbMJamBbM Week 12: Show your results (Youtube)]


===Links===
* 2025 results [https://github.com/intsystems/m1p/tree/main-2025 GitHub]
* 2025 [https://github.com/intsystems/m1p/tree/main-2025 The list of problems for 2025]
* 2024 results [https://github.com/intsystems/m1p/tree/main-2024 GitHub]
* 2024 problems [https://github.com/intsystems/m1p/blob/main-2024/problem_list.md GitHub]
* 2023 problems [https://github.com/intsystems/m1p/blob/main-2023/problem_list.md GitHub]
* 2022 results [https://github.com/Intelligent-Systems-Phystech/m1p_2022 GitHub]
* [http://www.machinelearning.ru/wiki/index.php?title=M1 Main page with old homework]
* [http://bit.ly/m1p_2020 Group 674, 694, spring 2020]
* [http://bit.ly/M1_2019_674 Group 674, spring 2019]
* [http://bit.ly/M1_2019_694 Group 694, spring 2019]

==Causal AI Models for Spatial-Time Series==
'''Foundation AI models''' are universal models to solve a wide set of problems. This project proposes to investigate the theoretical properties of foundation models. The domain to model is a spatial-time series. These data are used in various scientific disciplines and serve to generalise scientific knowledge and make forecasts. The essential problems, formulated as user requests that solve a foundation model, are forecasting and generation of time series; analysis and classification of time series; detection of change point, and causal inference. To solve these problems, the foundation AI models are trained on massive datasets. The main goal of this project is to compare various architectures of foundation models to find an optimal architecture that solves the listed problems for a wide range of spatial time series. See the [[Functional Data Analysis]] page.

==Mathematical forecasting, 2026==
This course delivers methods of model selection in machine learning and forecasting. The modeling data are videos, audio, encephalograms, fMRIs, and other measurements in natural science. The models are linear, tensor, deep neural networks, and neural ODEs. The practical ''examples'' are brain-computer interfaces, weather forecasting, and various spatial-time series forecasting. The ''lab works'' are organized as paper-with-code reports. [[Mathematical forecasting|See the page]]

== The Art of Scientific Research ==

The goal is to select and prepare the research topic of your dreams. We must be sure that the problem statement and project planning lead you to successful delivery according [[The Art of Scientific Research|to the syllabus]]. The repository template [https://github.com/vadim-vic/the-Art-homework helps].
* [[Step 0|Step 0: We start]]
* [[Step 1|Step 1: Highlight your work]]
* [[Step 2|Step 2: Describe an industrial project]]
* [[Step 3|Step 3: Explain the method]]
* [[Step 4|Step 4: Graphical highlights]]
* [[Step 5|Step 5: Deliver your message: slides 2 and 3]]
* [[Step 6|Step 6: Risk management in research planning]]
* [[Step 7|Step 7: Yield the foundation of your research]]
* [[Step 8|Step 8: Descriptive tools for your problem]]
* [[Step 9|Step 9: Launch your project with reasoning and statement]]
* [[Step 10|Step 10: Computational experiment and visualizing]]
* [[Step 11|Step 11: The final talk]]

==Articles==
* [[Fundamental theorems]] of Machine Learning
* [https://m1p.org/jmlda JMLDA archive]

My first scientific paper

2026-02-08T13:10:15Z

2025-11-12T17:14:15Z

Wiki: /* SINDy */

Channels:
* [https://t.me/+XyXmEXRlrXB9dZKD The 2025 course seminar]
* [https://t.me/+XyXmEXRlrXB9dZKD The chat-link FDA group]
When:
* September 4, 11, 18, 25 on Thursdays at 10:30 [https://m1p.org/go_zoom m1p.org/go_zoom]
* October (most likely) on Saturdays at 10:30

===Foundation models for spatial-time series===
Foundation AI models are universal models to solve a wide set of problems. This project proposes to investigate the theoretical properties of foundation models. The domain to model is a spatial-time series. These data are used in various scientific disciplines and serve to generalise scientific knowledge and make forecasts. The essential problems, formulated as user requests that solve a foundation model, are forecasting and generation of time series; analysis and classification of time series; detection of change point, and causal inference. To solve these problems, the foundation AI models are trained on massive datasets. The main goal of this project is to compare various architectures of foundation models to find an optimal architecture that solves the listed problems for a wide range of spatial time series.

===Functional data analysis===
The statistical analysis of spatial time series requires additional methods of data analysis. First, we suppose time is continuous, put the state space changes <math>\frac{d\mathbf{x}}{dt}</math>, and use neural ordinary and stochastic differential equations. Second, we analyze a multivariate and multidimensional time series and use the tensor representation and tensor analysis. Third, since the time series have significant cross-correlation, we model them in the Riemannian space. Fourth, medical time series are periodic; the base model is the pendulum model, <math>\frac{d^2x}{dt^2}=-c\sin{x}</math>. We use physics-informed neural networks to approximate data. Fifth, the practical experiments involve multiple data sources. We use canonical correlation analysis with a latent state space. This space aligns the source and target spaces and generates data in the source and target manifolds.

=== Applications ===
This field of Machine Learning applies to any field where the measurements have continuous time and space data acquired from multimodal sources: climate modeling, neural interfaces, solid-state physics, electronics, fluid dynamics, and many more. We will carefully collect both the theory and its practice.

== Fall 2025: Foundation models for time series ==
=== Topics top discuss===
# State Space Models, Convolution, SSA, SSM (Spectral Submanifolds)
# Neural and Controlled ODE, Neural PDE, Geometric Learning
# Operator Learning, Physics-informed learning, and multimodeling
# Spatial-Temporal Graph Modeling: Graph convolution and metric tensors
# Riemmannian models; time series generation
# AI for science: mathematical modelling principles

Outside the course: data-driven tensor analysis, differential forms, and spinors

=== State of the Art in 2025===
In December 2024, a NeurIPS workshop "Foundational models for science" reflected this theme:
# Foundation Models for Science: Progress, Opportunities, and Challenges [https://neurips.cc/virtual/2024/workshop/84714 URL]
# Foundation Models for the Earth system [https://neurips.cc/virtual/2024/107817 UPL, no paper]
# Foundation Methods for foundation models for scientific machine learning [https://neurips.cc/virtual/2024/107819 URL, no paper]
# AI-Augmented Climate simulators and emulators [https://neurips.cc/virtual/2024/107822 URL, no paper]
# Provable in-context learning of linear systems and linear elliptic PDEs with transformers [https://openreview.net/forum?id=xDstmuxn1D NIPS]
# VSMNO: Solving PDE by Utilizing Spectral Patterns of Different Neural Operators [https://openreview.net/pdf?id=oCT8pYix5e NIPS]

=== March 2025 Physics problem Simulations ===
# The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning [https://arxiv.org/pdf/2412.00568 ArXiv], [https://polymathic-ai.org/the_well/data_format/ Code]
# Polymatic Advancing Science through Multi‑Disciplinary AI [https://polymathic-ai.org/ blog]
# Long Term Memory: The Foundation of AI Self-Evolution [https://arxiv.org/html/2410.15665v1 ArXiv]
# Distilling Free-Form Natural Laws from Experimental Data, 2009 [https://www.science.org/doi/abs/10.1126/science.1165893 Science], [https://arxiv.org/pdf/1210.7273 comment], [https://medium.com/@lotussavy/distilling-free-form-natural-laws-from-experimental-data-f55341ae0fa6 medium]
# Deep learning for universal linear embeddings of nonlinear dynamics [https://www.nature.com/articles/s41467-018-07210-0 nature]
# A comparison of data-driven approaches to build low-dimensional ocean models, 2021 by Pavel Berloff [https://arxiv.org/abs/2108.00818 ArXiv], talk by Daniil Dorin for S.V. Fortova
# Applications of Deep Learning to Ocean Data Inference and Subgrid Parameterization by Thomas Bolton and Laure Zanna, 2018 [https://eartharxiv.org/repository/view/1142/ preprint], talk by Nilita Kiselev
# On energy-aware hybrid models by Shevchenko,2024 [https://doi.org/10.1029/2024MS004306 doi], talk by Mariya Nikitina
# Science: NASA satellites and computers have provided us with these mesmerizing swirls that cover our planet—but this isn’t star stuff. Each color represents a different aerosol that was floating in the atmosphere above our heads from 1 August to 14 September 2024 [https://www.facebook.com/reel/4083421318545496 video]

===Spatial-Temporal Graph Modeling===
# Graph WaveNet for Deep Spatial-Temporal Graph Modeling [https://arxiv.org/abs/1906.00121 ArXiv]
# Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting [https://arxiv.org/abs/1707.01926 ICLR]
# Time-SSM: Simplifying and Unifying State Space Models for Time Series Forecasting [https://arxiv.org/pdf/2405.16312 ArXiv] [https://github.com/haller-group/SSMTool-2.4 SSMTool]
# State Space Reconstruction for Multivariate Time Series Prediction [https://arxiv.org/abs/0809.2220 ArXiv]](Denis)
# Longitudinal predictive modeling of tau progression along the structural connectome by Joyita Dutta [https://www.sciencedirect.com/science/article/pii/S1053811921004031?via%3Dihub 2021]

== Key reviews ==
# 2018. Diffusion Convolutional Recurrent Neural Network [https://arxiv.org/pdf/1707.01926 ICLR]
# 2021. Neural Partial Differential Equations with Functional Convolution [https://openreview.net/pdf?id=D4A-v0kltaX ICLR]
# 2018. Graph WaveNet for Deep Spatial-Temporal Graph Modeling [https://arxiv.org/pdf/1906.00121 ArXiV]
# 2021. Neural Rough Differential Equations for Long Time Series (comparison)
# 2022. Time Series Forecasting Using Manifold Learning, Radial Basis Function Interpolation and Geometric Harmonics [https://doi.org/10.1063/5.0094887 doi] (all basic models + superpositions review)

=== Catch-up ===
If you are not familiar with the LLM and GPT:
# Build an LLM from scratch by Sebastian Raschka, 2025 [https://github.com/rasbt/LLMs-from-scratch/ github]
# Agentic Design Patterns by Antonio Gulli, 2025 [https://docs.google.com/document/d/1rsaK53T3Lg5KoGwvf8ukOUvbELRtH-V0LnOIFDxBryE/preview?tab=t.0 docx]
For fun, the vibe coding: [https://cursor.com/home 1], [https://windsurf.com/ 2], [https://www.augmentcode.com/ 3], [https://www.augmentcode.com/install 4], also [https://arxiv.org/abs/2501.09223 Foundations of LLM] and [https://www.youtube.com/@AndrejKarpathy Karpathy's] project
[https://github.com/karpathy/nanoGPT nanoGPT]

== Work arrangements==
{|class="wikitable"
|-
| ''' Week '''
| ''' Date '''
| ''' Theme '''
| ''' Delivery '''
|-
| 1
| sep 4
| Preliminary discussion [https://github.com/vadim-vic/Foundation-ts/tree/main/doc/Foundation_models_for_time_series_Week_1.pdf pdf]
|
|-
| 2
| sep 11
| Problem statement [https://github.com/vadim-vic/Foundation-ts/tree/main/doc/Foundation_models_for_time_series_Week_2.pdf pdf]
|
|-
| 3
| sep 18
| Preliminary solution
| Group talk and discussion
|-
| 4
| oct 2
| Minimum deployment
| Group report
|-
| 5
| oct 7+
| FDA
| Personal talks

|-
| 13
| nov 29
| Final discussion
| Group talks
|-

|-
|}



===Structure of seminars ===
The semester lasts 12 weeks, and six couple of weeks are for homework.
* Odd week: introduction to the topic and a handout of a theme for the homework.
* Every week: a discussion of the essay, collecting the list of improvements to each essay.
* Odd week: a discussion of the improved essay, putting the essays into a joint structure.

===Scoring===
The group activity is evaluated by cross-ranking with the Kemeni median score. The personal talks give a score.

== Week 3==
Homework 1
# Form a group
# Discuss the goals of the project and a solution ([see the problem statement])
# Make a review of various ways to solve the problem
# Select an LLM-GPT
# Run the code to check if it works
## Store the code in the group repository
## Store the talk slides/report, too
# Make a 10-minute talk about
## Functionality and architecture of the model
## Why did you select this model
## The alternative models to select from






====Requirements for the text and the discussion====
# Comprehensive explanation of the method or the question we discuss
# Only the principle, no experiments
# Two-page text (more or less)
# The reader is a second or third-year student
# The picture is obligatory
# However, a brief reference to some deep learning structure is welcome
# Talk could be a slide or a text itself
# The list of references with doi
# Tell how it was generated
# Observing a gap, put a note about it (to question later)

==== Style remarks for the essays ====
Automatic generation of mediocre-quality texts increased the requirements for the quality of the new messages. It makes novelty rare and makes the authorship appreciated. But it simplifies the way of delivering. So, since textbook generation has become simple, we will use generative chat to train our skills in reader persuasion. The reader is our MS thesis defense committee.



==Table of topics for seminars==
In these ten weeks, we will discuss the next five topics:
# Multimodal data
# Continous time and space models
# Physics-informed models
# Multilinear models
# Riemannian spaces

Note that all these items enlighten the stochastic-deterministic decomposition. So the questions include three parts:
# deterministic model,
# generative model,
# stochastic-deterministic decomposition method.
See the questions below for your reference.

=== Multimodal data ===
First series
# Canonical Correlation Analysis
# CCA in tensor representation
# Kernel CCA in Hilbert and L2[a,b] spaces
# CCA versus Cross-Attention Transformers
# Generative CCA, diffusion, and flow
# Comparative analysis of variants of CCA, like PLS and others
# Functional PCA



Talks
# Canonical Correlation Analysis in tensor representation [https://github.com/intsystems/IDA/blob/main-2024/essay-1-cca/khusainov/IDA_MIPT_week12_2.pdf Marat]
# Kernel CCA in Hilbert and L2[a,b] spaces [https://github.com/intsystems/IDA/blob/main-2024/essay-1-cca/mikhailov/main.pdf Bair]
# CCA versus Cross-Attention Transformers [https://github.com/intsystems/IDA/blob/main-2024/essay-1-cca/vladimirov/main-final.pdf Eduard]
# Generative CCA, diffusion, and flow [https://github.com/intsystems/IDA/blob/main-2024/essay-1-cca/boeva/VCCA_FLOWS.pdf Galina], [https://github.com/intsystems/IDA/blob/main-2024/essay-1-cca/boeva/essay1.pdf Galina]
# Functional PCA [https://github.com/intsystems/IDA/blob/main-2024/essay-1-cca/karimov/paper.pdf Parviz]

=== Continous models ===
Second series
# Neural ODE
# Continous state space models
# Continous normalizing flows
# Adjoint method and continuous backpropagation
# Neural Delayed Differential Equations 
# Neural PDE
# S4 and Hippo models [https://doi.org/10.48550/arXiv.2206.12037], [https://github.com/HazyResearch/state-spaces] (LSSL, SaShiMi, DSS, HTTYH, S4D, and S4ND)
# Riemannian continuous models

Talks
# Continuous state space models [https://github.com/intsystems/IDA/tree/main-2024/essay-2-cont/mikhailov Bair]
# Continuous normalizing flows [https://github.com/intsystems/IDA/blob/main-2024/essay-2-cont/khusainov/IDA_MIPT_week34_2.pdf Marat]
# Adjoint method and continuous backpropagation [https://github.com/intsystems/IDA/blob/main-2024/essay-2-cont/boeva/essay2_final.pdf Galina]
# Riemannian continuous models [https://github.com/intsystems/IDA/blob/main-2024/essay-2-cont/vladimirov/main-final.pdf Eduard]
====SINDy====
# Learning partial differential equations via data discovery and sparse optimization
by Hayden Schaeffer, 2017 [https://doi.org/10.1098/rspa.2016.0446 DOI], [https://robotics.caltech.edu/wiki/images/b/bc/LearningPDEs.pdf PDF]
# Data-driven discovery of partial differential equations by Rudy et al. 2017 [https://www.science.org/doi/full/10.1126/sciadv.1602614 Science], []
# Supporting Information for: Discovering governing equations from data [pnas.1517384113.sapp.pdf PDF]
# SINDy-PI: a robust algorithm for parallel implicit sparse identification of nonlinear dynamics by Kadierdan Kaheman et al., 2020 [https://royalsocietypublishing.org/doi/10.1098/rspa.2020.0279 DOI]
# Ensemble-SINDy by Fasel et al. 2021 [https://royalsocietypublishing.org/doi/epdf/10.1098/rspa.2021.0904 DOI]

===Physics-Informed models===
Third series
# PINNs as multimodels
# Spherical harmonics in p dimensions (an IMU example is welcome)
# PDF and Physics-Informed learning
# Integral Transforms in Physics-Informed learning

Talks
# Geometric Clifford Algebra Networks [https://github.com/intsystems/IDA/blob/main-2024/essay-3-pinn/vladimirov/main.pdf Eduard]
# Integral Transforms in Physics-Informed learning [https://github.com/intsystems/IDA/blob/main-2024/essay-3-pinn/boeva/essay3_boeva.pdf Galina]

===Multilinear models and topology===
Fourth series
# Clifford or Geometric algebra in machine learning
# Tensor models, tensor decomposition, and approximation (tensor PLS or CCA)
# Machine learning models for tensors: Field Equation (Yang-Mills Equations_
# Machine learning models for theoretical physics (Maxwell’s Equations, Navier-Stokes)
# Persistent homology and dimensionality reduction (say, arXiv:2302.03447 with embedding delays)

Talks
# Tensor models, tensor decomposition, and approximation [https://github.com/intsystems/IDA/blob/main-2024/essay-4-multilinear/vladimirov/main-final.pdf Eduard]
# Machine learning models for theoretical physics (Maxwell’s Equations, Navier-Stocks) [https://github.com/intsystems/IDA/blob/main-2024/essay-4-multilinear/boeva/essay4_final.pdf Galina]

===Generative and Riemannian models===
Fifth series
# Generative Riemannian models. How do we extract and use the distribution?
# Generative Canonical Correlation Analysis and its connection with the Riemannian spaces in the latent part
# Scoring-based Riemannian models. How do we extract and use the distribution?
# Generative convolutional models for tensors. Is there a continuous-time? (A variant is the Riemannian Residual Networks).
# Riemannian continuous normalizing flows. How do we generate a time series of a given distribution?

Talks
# Scoring-based Riemannian models [https://github.com/intsystems/IDA/blob/main-2024/essay-5-riemannian-generative/vladimirov/main.pdf Eduard]
# Riemannian continuous normalizing flows [https://github.com/intsystems/IDA/blob/main-2024/essay-5-riemannian-generative/boeva/essay5.pdf Galina]

===Operator learning===
An additional topic to summarise all the above. See the introduction in
# Neural operators [https://en.wikipedia.org/wiki/Neural_operators wiki]
# Operator Learning: Convolutional Neural Operators [https://medium.com/@bogdan.raonke/operator-learning-convolutional-neural-operators-for-robust-and-accurate-learning-of-pdes-ebbc43b57434 blog]
# Convolutional Neural Operators for robust and accurate learning of PDEs [https://arxiv.org/pdf/2302.01178 arxiv 2023]
# Representation Equivalent Neural Operators: a Framework for Alias-free Operator Learning [https://arxiv.org/pdf/2305.19913 arxiv 2023]
# PID: Proportional-Integral-Differential-equation modeling with operator learning

===Discussed literature===
# Generative CCA, diffusion, and flow by Galina [https://arxiv.org/html/2312.13455v1] [https://arxiv.org/pdf/2305.11832] [https://arxiv.org/pdf/1610.03454] [https://era.library.ualberta.ca/items/64d14f0d-eb08-4fba-92aa-c8e0d42af448]
# Kernel CCA in Hilbert and L2[a,b] spaces by Bair [https://proceedings.mlr.press/v28/chang13.pdf] [https://www.jmlr.org/papers/volume3/bach02a/bach02a.pdf]
# CCA versus Cross-Attention Transformers by Eduard [https://arxiv.org/pdf/1911.05544] [https://statisticalsupportandresearch.wordpress.com/wp-content/uploads/2017/06/k-v-mardia-j-t-kent-j-m-bibby-multivariate-analysis-probability-and-mathematical-statistics-academic-press-inc-1979.pdf] [http://vigir.missouri.edu/~gdesouza/Research/Conference_CDs/IEEE_WCCI_2020/IJCNN/Papers/N-20729.pdf]
# Ajoint method and continuous backpropagation by Galina [https://ilya.schurov.com/post/adjoint-method]
# Continuous normalizing flows by Galina [https://arxiv.org/pdf/2106.08462v2]
# Tensor models by Eduard [https://www.kolda.net/publication/TensorReview.pdf] [https://arxiv.org/pdf/1502.02330] [https://sci-hub.se/10.1109/tpami.2008.167]
# Navier-Stokes [https://arc.aiaa.org/doi/10.2514/6.2022-1436] [https://www.sci-hub.ru/10.1007/s00521-014-1762-2] [https://www.physicsbaseddeeplearning.org/references.html]
# Classics versus quantum by Galina
## [https://www.mdpi.com/1099-4300/18/1/34 Schroedinger vs. Navier–Stokes 2016]
## [https://arxiv.org/abs/1707.04474v1 Many-particle quantum hydrodynamics: Exact equations and pressure tensors 2019]
## [https://content.iospress.com/articles/asymptotic-analysis/asy14-2-01 Quantum hydrodynamics, Wigner transforms, the classical limit 1995]
## [https://link.springer.com/article/10.1007/s10440-019-00257-1 Geometry of Nonadiabatic Quantum Hydrodynamics 2019]
## [https://iopscience.iop.org/article/10.1088/1367-2630/16/6/063011 Theory of quantum friction 2014]
## [https://www.science.org/doi/10.1126/sciadv.aba3747 Minimal quantum viscosity from fundamental physical constants]
## [https://thesis.library.caltech.edu/10278/ Fluid Dynamics with Incompressible Schrödinger Flow 2017]
## [https://gamedev.ru/code/articles/shrodinger_hydrodynamics Гидродинамика Шрёдингера на пальцах]
# Riemannian continuous normalizing flows by Galina [http://bayesiandeeplearning.org/2016/papers/BDL_33.pdf] [https://aclanthology.org/N19-1025.pdf] [https://arxiv.org/pdf/2006.10605]

==Practical spatial-time series==
# A guide to state–space modeling of ecological time series, 2021 [https://doi.org/10.1002/ecm.1470 PDF], (Bayesian Kalman)
# Kalman Filtering and Smoothing, 2025 [https://arxiv.org/pdf/2405.08971 ArXiv] (Riemannian Kalman)

==Data collections==
# ClimateSet, 2023 [https://arxiv.org/pdf/2311.03721 ArXiv]

==References==
===General===
# Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems [https://arxiv.org/abs/2307.08423 arxiv 2023]
# Algebra, Topology, Differential Calculus, and Optimization Theory For Computer Science and Machine Learning [https://www.cis.upenn.edu/~jean/math-deep.pdf upenn 2024]
# The Elements of Differentiable Programming [https://arxiv.org/abs/2403.14606 arxiv 2024]
# The list from the previous year [https://github.com/intsystems/IDA/tree/main-2023 2023].
# Differential Geometry of Curves and Surfaces: Textbook, 2016 by Kristopher Tapp [https://link.springer.com/book/10.1007/978-3-319-39799-3]

===Prerequisites===
# Understanding Deep Learning ''by Simon J.D. Prince'' [https://udlbook.github.io/udlbook/ mit 2023]
# Deep Learning by ''C.M. and H. Bishops'' [https://www.bishopbook.com/ Springer 2024] (online version)
# A Geometric Approach to Differential Forms ''by David Bachman'' [https://arxiv.org/abs/math/0306194v1 arxiv 2013]
# Advanced Calculus: Geometric View ''by James J. Callahan'' [https://download.tuxfamily.org/openmathdep/calculus_advanced/Advanced_Calculus-Callahan.pdf pdf 2010], [https://download.tuxfamily.org/openmathdep/calculus_advanced/Advanced_Calculus-Callahan.pdf collection]
# Geometric Deep Learning by Michael M. Bronstein [https://arxiv.org/pdf/2104.13478 arxiv 2021]
=== Linear and bilinear models===
# A Tutorial on Independent Component Analysis [https://arxiv.org/abs/1404.2986 arxiv, 2014]
# On the Stability of Multilinear Dynamical Systems [https://arxiv.org/abs/2105.01041 arxiv 2022]
# Tensor-based Regression Models and Applications ''by Ming Hou'' Thèse [https://core.ac.uk/download/pdf/442636056.pdf Uni-Laval 2017] 
# Tensor Canonical Correlation Analysis for Multi-view Dimension Reduction [https://arxiv.org/pdf/1502.02330] (Semkin)
#Tensor Learning in Multi-view Kernel PCA [https://link.springer.com/chapter/10.1007/978-3-030-01421-6_21 arxiv 2018]
# Tensor decomposition of EEG signals: A brief review [http://dx.doi.org/10.1016/j.jneumeth.2015.03.018 2015]
====Spherical Harmonics====
# Spherical Harmonic Transforms: In JAX and PyTorch [https://medium.com/data-science/differentiable-and-accelerated-spherical-harmonic-transforms-c269393d08f1 Medium 2024]
# Spherical Harmonics in p Dimensions [https://arxiv.org/abs/1205.3548 arxiv 2012]
# Physics of simple pendulum: a case study of nonlinear dynamics [https://www.researchgate.net/publication/332766499_Physics_of_simple_pendulum_a_case_study_of_nonlinear_dynamics RG 2008]
# Time series forecasting using manifold learning, 2021 [https://arxiv.org/pdf/2110.03625 arxiv]
# Time-series forecasting using manifold learning, radial basis function interpolation, and geometric harmonics [https://doi.org/10.1063/5.0094887 2022 Chaos AIP]

====State Space Models====
# Missing Slice Recovery for Tensors Using a Low-rank Model in Embedded Space [https://arxiv.org/abs/1804.01736 arxiv 2018]
#Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks by A.R. Voelker et al., 2019 [https://papers.nips.cc/paper_files/paper/2019/file/952285b9b7e7a1be5aa7849f32ffff05-Paper.pdf NeurIPS]

====SSM Generative Models ====
# Masked Autoregressive Flow for Density Estimation [https://arxiv.org/abs/1705.07057 arxiv 2017]
====SSM+Riemann+Gaussian process regression====
* Time-series forecasting using manifold learning, radial basis function interpolation, and geometric harmonics by Ioannis G. Kevrekidis,3 and Constantinos Siettos, 2022 [https://pubs.aip.org/aip/cha/article-pdf/doi/10.1063/5.0094887/16497596/083113_1_online.pdf pdf]

===Physics-Informed Neural Networks===
# Neural partial differential equations with functional convolution [https://openreview.net/pdf?id=D4A-v0kltaX ICLP]
# Solving PDEs by variational physics-informed neural networks: an a posteriori error analysis [https://link.springer.com/article/10.1007/s11565-022-00441-6?fromPaywallRec=true PDF] plus several links to the books on the subject inside
# Predicting the nonlinear dynamics of spatiotemporal PDEs via physics-informed informer networks [https://link.springer.com/article/10.1007/s11071-024-10655-2?fromPaywallRec=true PDF]
# Three ways to solve partial differential equations with neural networks — A review [https://arxiv.org/abs/2102.11802 arxiv 2021]
# NeuPDE: Neural Network Based Ordinary and Partial Differential Equations for Modeling Time-Dependent Data [https://arxiv.org/abs/1908.03190 arxiv 2019]
# Physics-based deep learning [https://www.physicsbaseddeeplearning.org/intro-teaser.html code]
# PINN by Steve Burton [https://www.youtube.com/watch?v=g-S0m2zcKUg&list=PLMrJAkhIeNNQ0BaKuBKY43k4xMo6NSbBa&index=3 yt]
# Process Model Inversion in the Data-Driven Engineering Context for Improved Parameter Sensitivities [https://www.mdpi.com/2227-9717/10/9/1764 mdpi processes 2022] ('''nice connection pictures''')
# Physics-based Deep Learning [https://www.physicsbaseddeeplearning.org/intro.html github]
# Integral Transforms in a Physics-Informed (Quantum) Neural Network setting [https://arxiv.org/pdf/2206.14184 arxiv 2022]
# Lectures ny Stephen Brunton [https://www.youtube.com/watch?v=fiX8c-4K0-Q&list=PLMrJAkhIeNNQ0BaKuBKY43k4xMo6NSbBa&index=5 AI/ML+Physics], [https://www.youtube.com/watch?v=3SNkQ8jhKXc&list=PLMrJAkhIeNNQ0BaKuBKY43k4xMo6NSbBa&index=7 Part 4], [https://www.youtube.com/watch?v=nmvs0vrBT18 Basic PDEs], [https://www.youtube.com/watch?v=pvrIagjEk4c PDE Overview],

=== Riemmanian models===
# Riemannian Continuous Normalizing Flows [https://arxiv.org/abs/2006.10605 arxiv 2020]
# Residual Riemannian Networks [https://arxiv.org/pdf/2310.10013 arxiv 2023] 
=== Continous time, Neural ODE===
# Neural Spatio-Temporal Point Processes ''by Ricky Chen et al.'' [https://arxiv.org/abs/2011.04583 iclr 2021] (likelihood for time and space)
# Neural Ordinary Differential Equations ''by Ricky Chen et al.'' [https://arxiv.org/abs/1806.07366 arxiv 2018]
# Neural Controlled Differential Equations for Irregular Time Series 'Patrick Kidger et al.'' [https://arxiv.org/abs/2005.08926 arxiv 2020][https://github.com/patrick-kidger/NeuralCDE github]
# Diffusion Normalizing Flow [https://arxiv.org/pdf/2110.07579 arxiv 2021]
# Differentiable Programming for Differential Equations: A Review [https://arxiv.org/abs/2406.09699 arxiv 2024]
# (code tutorial) Deep Implicit Layers - Neural ODEs, Deep Equilibrium Models, and Beyond [https://implicit-layers-tutorial.org/ nips 2020]
# (code tutorial) [https://www.physicsbaseddeeplearning.org/overview-ns-forw.html 2021]
# Neural CDE and tensors [https://ieeexplore.ieee.org/abstract/document/9979806 IEEE], [https://ieeexplore.ieee.org/abstract/document/9533771 IEEE]
# Latent ODEs for Irregularly-Sampled Time Series [https://proceedings.neurips.cc/paper_files/paper/2019/file/42a6845a557bef704ad8ac9cb4461d43-Paper.pdf 2019]

=== Graph and PDEs ===
# Fourier Neural Operator for Parametric Partial Differential Equations [https://arxiv.org/abs/2010.08895 arxiv 2020]
# Masked Attention is All You Need for Graphs [https://arxiv.org/abs/2402.10793 arxiv 2024]
=== Neural SDE===
# Approximation of Stochastic Quasi-Periodic Responses of Limit Cycles in Non-Equilibrium Systems under Periodic Excitations and Weak Fluctuations [https://doi.org/10.3390/e19060280 mdpi entropy 2017] (great illustrations on the stochastic nature of a simple phase trajectory)
# Approximation of Stochastic Quasi-Periodic Responses of Limit Cycles in Non-Equilibrium Systems under Periodic Excitations and Weak Fluctuations [https://doi.org/10.3390/e19060280 mdpi entropy 2017] (great illustrations on the stochastic nature of a simple phase trajectory)
# Neural SDEs for Conditional Time Series Generation [https://arxiv.org/abs/2301.01315 arxiv 2023] code [https://github.com/pere98diaz/Neural-SDEs-for-Conditional-Time-Series-Generation-and-the-Signature-Wasserstein-1-metric github LSTM - CSig-WGAN]
# Neural SDEs as Infinite-Dimensional GANs [https://arxiv.org/pdf/2102.03657 2021]
# Efficient and Accurate Gradients for Neural SDEs ''by Patrick Kidger'' [https://arxiv.org/pdf/2105.13493 arxiv 2021] code [https://docs.kidger.site/diffrax/examples/neural_sde/ diffrax]
=== Chains and homology===
# Operator Learning: Algorithms and Analysis [https://arxiv.org/pdf/2402.15715 arxiv 2024]
# Hi-res weather: Operator learning [https://arxiv.org/pdf/2202.11214 arxiv 2022]
# Homotopy theory for beginners by J.M. Moeller [https://web.math.ku.dk/~moller/e01/algtopI/comments.pdf ku.dk 2015] (is it a pertinent link?)
# Explorations in Homeomorphic Variational Auto-Encoding [https://arxiv.org/abs/1807.04689 arxiv 2018]
# Special Finite Elements for Dipole Modelling ''master thesis Bauer'' [https://www.sci.utah.edu/~wolters/PaperWolters/2012/BauerMaster.pdf 2011]
# Selecting embedding delays: An overview of embedding techniques and a new method using persistent homology [https://arxiv.org/pdf/2302.03447v1 arxiv 2023] (Denis)
# (code) Clifford Algebra for Python https://clifford.readthedocs.io/en/v1.1.0/

===Appendix===
# Neural Memory Networks [https://cs229.stanford.edu/proj2015/367_report.pdf stanford reports 2019]
# An Elementary Introduction to Information Geometry ''by Frank Nielsen'' [An Elementary Introduction to Information Geometry Frank Nielsen [https://doi.org/10.3390/e22101100 mdpi entropy]
# The Many Faces of Information Geometry ''by Frank Nielsen'' [https://www.ams.org/journals/notices/202201/rnoti-p36.pdf ams 2022] (short version)
# Geometric Clifford Algebra Networks [https://arxiv.org/abs/2302.06594 arxiv 3022]
# Clifford Algebras and Dimensionality Reduction for Signal Separation ''by [https://www.math.uni-hamburg.de/home/guillemard/ M. Guillemard]'' [https://www.math.uni-hamburg.de/home/guillemard/papers/clifford7.pdf Uni-Hamburg 2010][https://www.math.uni-hamburg.de/home/guillemard/clifford/ code]
# Special Finite Elements for Dipole Modelling ''by Martin Bauer'' Master Thesis [https://www.sci.utah.edu/~wolters/PaperWolters/2012/BauerMaster.pdf Erlangen 2012] diff p-form must read
# Bayesian model selection for complex dynamic systems [https://www.nature.com/articles/s41467-018-04241-5 2018]
# Visualizing 3-Dimensional Manifolds ''by Dugan J. Hammock'' [https://archive.bridgesmathart.org/2013/bridges2013-551.pdf 2013 umass]
# At the Interface of Algebra and Statistics by ''T-D. Bradley'' [https://arxiv.org/abs/2004.05631 arxiv 2020]
# Time Series Handbook by Borja, 2021 [https://github.com/phdinds-aim/time_series_handbook github]
# Physics-informed machine learning [https://www.nature.com/articles/s42254-021-00314-5 Nature reviews: Physics 2021]
# Integral Transforms in a Physics-Informed (Quantum) Neural Network setting: Applications & Use-Cases [https://arxiv.org/abs/2206.14184 arxiv 2022]
# Deep Efficient Continuous Manifold Learning for Time Series Modeling [https://arxiv.org/abs/2112.03379 arxiv 2021]
====Causality====
# Toward Causal Representation Learning [https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9363924 2021]
# See the Sugihara collection

==Basics==
Collection of wiki-links
===Time Series===
# [https://en.wikipedia.org/wiki/Spectral_submanifold Spectral submanifold] (with nonlinear dimensional reduction like [https://en.wikipedia.org/wiki/Self-organizing_map som])
# [https://en.wikipedia.org/wiki/Lagrangian_coherent_structure Lagrangian coherent structure] (software below)
===Signal Processing===
#[https://en.wikipedia.org/wiki/Estimation_of_signal_parameters_via_rotational_invariance_techniques Estimation of signal parameters via rotational invariance techniques]
#[https://en.wikipedia.org/wiki/Reproducing_kernel_Hilbert_space Reproducing kernel Hilbert space]
#[https://en.wikipedia.org/wiki/Kernel_principal_component_analysis Kernel principal component analysis]
#[https://en.wikipedia.org/wiki/Gram_matrix Gram matrix]
#[https://en.wikipedia.org/wiki/Generalized_pencil-of-function_method Generalized pencil-of-function method]
#[https://en.wikipedia.org/wiki/Wavelet_transform Wavelet transform]

===Differential Geometry===
#[https://en.wikipedia.org/wiki/Pushforward_(differential) Pushforward (differential)]
#[https://en.wikipedia.org/wiki/Pullback_bundle Ffibers, Bundles, Sheaves]
#[https://en.wikipedia.org/wiki/Homology_(mathematics) Homology]
#[https://en.wikipedia.org/wiki/Topological_data_analysis Topological data analysis]
#[https://en.wikipedia.org/wiki/Conditional_mutual_information Conditional mutual information]
#[https://en.wikipedia.org/wiki/Convergent_cross_mapping Convergent cross mapping]
#[https://en.wikipedia.org/wiki/Differential_form Differential form]
#[https://en.wikipedia.org/wiki/Total_derivative The total derivative as a differential form]
#[https://en.wikipedia.org/wiki/Riemannian_manifold #Riemannian_metrics Riemannian_metrics]
# Multidimensional Differential and Integral Calculus: A Practical Approach (textbook)

===Probabilistical Decompisition===
#[https://en.wikipedia.org/wiki/Wasserstein_metric Wasserstein metric]
#[https://en.wikipedia.org/wiki/Mutual_information Mutual information]
#[https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant Jacobian]
#[https://en.wikipedia.org/wiki/Fisher_information Fisher information]
# also dobrushin stratonovich wasserstein
# also fluid dymanics, transportation theory

===Tutorials===
# [https://www.connectedpapers.com/main/d86084808994ac54ef4840ae65295f3c0ec4decd/Physics%20informed-neural-networks%3A-A-deep-learning-framework-for-solving-forward-and-inverse-problems-involving-nonlinear-partial-differential-equations/graph Connected papers search]
# Operator Learning via Physics-Informed DeepONet: Let’s Implement It From Scratch [https://towardsdatascience.com/operator-learning-via-physics-informed-deeponet-lets-implement-it-from-scratch-6659f3179887 Medium]

==Tools==
# [https://github.com/ilkhem/icebeem icebeem]
# [https://github.com/ilkhem/ivae ivae]
# [https://github.com/kondratevakate/fmri-component-# fmri-component]
# analysis/blob/master/VAE_for_fMRI/dataset/train/Bystrova0_y-axis.png
# [https://fr.mathworks.com/help/deeplearning/ug/dynamical-system-modeling-using-neural-ode.html Neural ODE in Matlab]
# [https://github.com/pyRiemann/pyRiemann pyRiemann]
# causality inference [https://peps.python.org/pep-0484/#scoping-rules-for-type-variables peps]
# [https://github.com/xai-org/grok-1 LMM grok-1 with weights]

===Turbulence===
# Runko: Modern multiphysics toolbox for plasma simulations [https://github.com/hel-astro-lab/runko GitHub]
# 2d-turb-PINN by Parfenyev [https://github.com/parfenyev/2d-turb-PINN/tree/main GitHub]

====Physics and Engineering of Turbulence====
# Fundamentals of Fluid_Mechanics, 2013 [https://students.aiu.edu/submissions/profiles/resources/onlineBook/L5g8S6_Fundamentals_of_Fluid_Mechanics-_7.pdf PDF]
# Introduction ot Fluid Mechanics, 2004 by R. Fox et al. [https://ahsheikh.github.io/Courses/NumMod/FM_RobertFox.pdf PDF]
# Computational fluid dynamics, 1995 by John D. Anderson, Jr. [https://www.airloads.net/Downloads/Textbooks/Computational-Fluid-Dynamics-the-Basics-With-Applications-Anderson-J-D.pdf PDF]
# Fluid-dynamic drag, 1965 by S.F. Hoerner [https://ia800606.us.archive.org/17/items/FluidDynamicDragHoerner1965/Fluid-dynamic_drag__Hoerner__1965_text.pdf PDF]
# TorchDyn: A Neural Differential Equations Library [https://arxiv.org/abs/2009.09346 arXiv] [https://github.com/DiffEqML/torchdyn github]
## Parameterized Neural Ordinary Differential Equations: Applications to Computational Physics Problems [https://arxiv.org/abs/2010.14685 ArXiv]
## Turbulence forecasting via Neural ODE [https://arxiv.org/abs/1911.05180v1 ArXiv]
## (not the same) Hamiltonian Neural Networks [https://arxiv.org/abs/1906.01563 ArXiv]

Functional Data Analysis

2025-11-12T13:28:33Z

Wiki: /* Continous models */

Channels:
* [https://t.me/+XyXmEXRlrXB9dZKD The 2025 course seminar]
* [https://t.me/+XyXmEXRlrXB9dZKD The chat-link FDA group]
When:
* September 4, 11, 18, 25 on Thursdays at 10:30 [https://m1p.org/go_zoom m1p.org/go_zoom]
* October (most likely) on Saturdays at 10:30

===Foundation models for spatial-time series===
Foundation AI models are universal models to solve a wide set of problems. This project proposes to investigate the theoretical properties of foundation models. The domain to model is a spatial-time series. These data are used in various scientific disciplines and serve to generalise scientific knowledge and make forecasts. The essential problems, formulated as user requests that solve a foundation model, are forecasting and generation of time series; analysis and classification of time series; detection of change point, and causal inference. To solve these problems, the foundation AI models are trained on massive datasets. The main goal of this project is to compare various architectures of foundation models to find an optimal architecture that solves the listed problems for a wide range of spatial time series.

===Functional data analysis===
The statistical analysis of spatial time series requires additional methods of data analysis. First, we suppose time is continuous, put the state space changes <math>\frac{d\mathbf{x}}{dt}</math>, and use neural ordinary and stochastic differential equations. Second, we analyze a multivariate and multidimensional time series and use the tensor representation and tensor analysis. Third, since the time series have significant cross-correlation, we model them in the Riemannian space. Fourth, medical time series are periodic; the base model is the pendulum model, <math>\frac{d^2x}{dt^2}=-c\sin{x}</math>. We use physics-informed neural networks to approximate data. Fifth, the practical experiments involve multiple data sources. We use canonical correlation analysis with a latent state space. This space aligns the source and target spaces and generates data in the source and target manifolds.

=== Applications ===
This field of Machine Learning applies to any field where the measurements have continuous time and space data acquired from multimodal sources: climate modeling, neural interfaces, solid-state physics, electronics, fluid dynamics, and many more. We will carefully collect both the theory and its practice.

== Fall 2025: Foundation models for time series ==
=== Topics top discuss===
# State Space Models, Convolution, SSA, SSM (Spectral Submanifolds)
# Neural and Controlled ODE, Neural PDE, Geometric Learning
# Operator Learning, Physics-informed learning, and multimodeling
# Spatial-Temporal Graph Modeling: Graph convolution and metric tensors
# Riemmannian models; time series generation
# AI for science: mathematical modelling principles

Outside the course: data-driven tensor analysis, differential forms, and spinors

=== State of the Art in 2025===
In December 2024, a NeurIPS workshop "Foundational models for science" reflected this theme:
# Foundation Models for Science: Progress, Opportunities, and Challenges [https://neurips.cc/virtual/2024/workshop/84714 URL]
# Foundation Models for the Earth system [https://neurips.cc/virtual/2024/107817 UPL, no paper]
# Foundation Methods for foundation models for scientific machine learning [https://neurips.cc/virtual/2024/107819 URL, no paper]
# AI-Augmented Climate simulators and emulators [https://neurips.cc/virtual/2024/107822 URL, no paper]
# Provable in-context learning of linear systems and linear elliptic PDEs with transformers [https://openreview.net/forum?id=xDstmuxn1D NIPS]
# VSMNO: Solving PDE by Utilizing Spectral Patterns of Different Neural Operators [https://openreview.net/pdf?id=oCT8pYix5e NIPS]

=== March 2025 Physics problem Simulations ===
# The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning [https://arxiv.org/pdf/2412.00568 ArXiv], [https://polymathic-ai.org/the_well/data_format/ Code]
# Polymatic Advancing Science through Multi‑Disciplinary AI [https://polymathic-ai.org/ blog]
# Long Term Memory: The Foundation of AI Self-Evolution [https://arxiv.org/html/2410.15665v1 ArXiv]
# Distilling Free-Form Natural Laws from Experimental Data, 2009 [https://www.science.org/doi/abs/10.1126/science.1165893 Science], [https://arxiv.org/pdf/1210.7273 comment], [https://medium.com/@lotussavy/distilling-free-form-natural-laws-from-experimental-data-f55341ae0fa6 medium]
# Deep learning for universal linear embeddings of nonlinear dynamics [https://www.nature.com/articles/s41467-018-07210-0 nature]
# A comparison of data-driven approaches to build low-dimensional ocean models, 2021 by Pavel Berloff [https://arxiv.org/abs/2108.00818 ArXiv], talk by Daniil Dorin for S.V. Fortova
# Applications of Deep Learning to Ocean Data Inference and Subgrid Parameterization by Thomas Bolton and Laure Zanna, 2018 [https://eartharxiv.org/repository/view/1142/ preprint], talk by Nilita Kiselev
# On energy-aware hybrid models by Shevchenko,2024 [https://doi.org/10.1029/2024MS004306 doi], talk by Mariya Nikitina
# Science: NASA satellites and computers have provided us with these mesmerizing swirls that cover our planet—but this isn’t star stuff. Each color represents a different aerosol that was floating in the atmosphere above our heads from 1 August to 14 September 2024 [https://www.facebook.com/reel/4083421318545496 video]

===Spatial-Temporal Graph Modeling===
# Graph WaveNet for Deep Spatial-Temporal Graph Modeling [https://arxiv.org/abs/1906.00121 ArXiv]
# Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting [https://arxiv.org/abs/1707.01926 ICLR]
# Time-SSM: Simplifying and Unifying State Space Models for Time Series Forecasting [https://arxiv.org/pdf/2405.16312 ArXiv] [https://github.com/haller-group/SSMTool-2.4 SSMTool]
# State Space Reconstruction for Multivariate Time Series Prediction [https://arxiv.org/abs/0809.2220 ArXiv]](Denis)
# Longitudinal predictive modeling of tau progression along the structural connectome by Joyita Dutta [https://www.sciencedirect.com/science/article/pii/S1053811921004031?via%3Dihub 2021]

== Key reviews ==
# 2018. Diffusion Convolutional Recurrent Neural Network [https://arxiv.org/pdf/1707.01926 ICLR]
# 2021. Neural Partial Differential Equations with Functional Convolution [https://openreview.net/pdf?id=D4A-v0kltaX ICLR]
# 2018. Graph WaveNet for Deep Spatial-Temporal Graph Modeling [https://arxiv.org/pdf/1906.00121 ArXiV]
# 2021. Neural Rough Differential Equations for Long Time Series (comparison)
# 2022. Time Series Forecasting Using Manifold Learning, Radial Basis Function Interpolation and Geometric Harmonics [https://doi.org/10.1063/5.0094887 doi] (all basic models + superpositions review)

=== Catch-up ===
If you are not familiar with the LLM and GPT:
# Build an LLM from scratch by Sebastian Raschka, 2025 [https://github.com/rasbt/LLMs-from-scratch/ github]
# Agentic Design Patterns by Antonio Gulli, 2025 [https://docs.google.com/document/d/1rsaK53T3Lg5KoGwvf8ukOUvbELRtH-V0LnOIFDxBryE/preview?tab=t.0 docx]
For fun, the vibe coding: [https://cursor.com/home 1], [https://windsurf.com/ 2], [https://www.augmentcode.com/ 3], [https://www.augmentcode.com/install 4], also [https://arxiv.org/abs/2501.09223 Foundations of LLM] and [https://www.youtube.com/@AndrejKarpathy Karpathy's] project
[https://github.com/karpathy/nanoGPT nanoGPT]

== Work arrangements==
{|class="wikitable"
|-
| ''' Week '''
| ''' Date '''
| ''' Theme '''
| ''' Delivery '''
|-
| 1
| sep 4
| Preliminary discussion [https://github.com/vadim-vic/Foundation-ts/tree/main/doc/Foundation_models_for_time_series_Week_1.pdf pdf]
|
|-
| 2
| sep 11
| Problem statement [https://github.com/vadim-vic/Foundation-ts/tree/main/doc/Foundation_models_for_time_series_Week_2.pdf pdf]
|
|-
| 3
| sep 18
| Preliminary solution
| Group talk and discussion
|-
| 4
| oct 2
| Minimum deployment
| Group report
|-
| 5
| oct 7+
| FDA
| Personal talks

|-
| 13
| nov 29
| Final discussion
| Group talks
|-

|-
|}



===Structure of seminars ===
The semester lasts 12 weeks, and six couple of weeks are for homework.
* Odd week: introduction to the topic and a handout of a theme for the homework.
* Every week: a discussion of the essay, collecting the list of improvements to each essay.
* Odd week: a discussion of the improved essay, putting the essays into a joint structure.

===Scoring===
The group activity is evaluated by cross-ranking with the Kemeni median score. The personal talks give a score.

== Week 3==
Homework 1
# Form a group
# Discuss the goals of the project and a solution ([see the problem statement])
# Make a review of various ways to solve the problem
# Select an LLM-GPT
# Run the code to check if it works
## Store the code in the group repository
## Store the talk slides/report, too
# Make a 10-minute talk about
## Functionality and architecture of the model
## Why did you select this model
## The alternative models to select from






====Requirements for the text and the discussion====
# Comprehensive explanation of the method or the question we discuss
# Only the principle, no experiments
# Two-page text (more or less)
# The reader is a second or third-year student
# The picture is obligatory
# However, a brief reference to some deep learning structure is welcome
# Talk could be a slide or a text itself
# The list of references with doi
# Tell how it was generated
# Observing a gap, put a note about it (to question later)

==== Style remarks for the essays ====
Automatic generation of mediocre-quality texts increased the requirements for the quality of the new messages. It makes novelty rare and makes the authorship appreciated. But it simplifies the way of delivering. So, since textbook generation has become simple, we will use generative chat to train our skills in reader persuasion. The reader is our MS thesis defense committee.



==Table of topics for seminars==
In these ten weeks, we will discuss the next five topics:
# Multimodal data
# Continous time and space models
# Physics-informed models
# Multilinear models
# Riemannian spaces

Note that all these items enlighten the stochastic-deterministic decomposition. So the questions include three parts:
# deterministic model,
# generative model,
# stochastic-deterministic decomposition method.
See the questions below for your reference.

=== Multimodal data ===
First series
# Canonical Correlation Analysis
# CCA in tensor representation
# Kernel CCA in Hilbert and L2[a,b] spaces
# CCA versus Cross-Attention Transformers
# Generative CCA, diffusion, and flow
# Comparative analysis of variants of CCA, like PLS and others
# Functional PCA



Talks
# Canonical Correlation Analysis in tensor representation [https://github.com/intsystems/IDA/blob/main-2024/essay-1-cca/khusainov/IDA_MIPT_week12_2.pdf Marat]
# Kernel CCA in Hilbert and L2[a,b] spaces [https://github.com/intsystems/IDA/blob/main-2024/essay-1-cca/mikhailov/main.pdf Bair]
# CCA versus Cross-Attention Transformers [https://github.com/intsystems/IDA/blob/main-2024/essay-1-cca/vladimirov/main-final.pdf Eduard]
# Generative CCA, diffusion, and flow [https://github.com/intsystems/IDA/blob/main-2024/essay-1-cca/boeva/VCCA_FLOWS.pdf Galina], [https://github.com/intsystems/IDA/blob/main-2024/essay-1-cca/boeva/essay1.pdf Galina]
# Functional PCA [https://github.com/intsystems/IDA/blob/main-2024/essay-1-cca/karimov/paper.pdf Parviz]

=== Continous models ===
Second series
# Neural ODE
# Continous state space models
# Continous normalizing flows
# Adjoint method and continuous backpropagation
# Neural Delayed Differential Equations 
# Neural PDE
# S4 and Hippo models [https://doi.org/10.48550/arXiv.2206.12037], [https://github.com/HazyResearch/state-spaces] (LSSL, SaShiMi, DSS, HTTYH, S4D, and S4ND)
# Riemannian continuous models

Talks
# Continuous state space models [https://github.com/intsystems/IDA/tree/main-2024/essay-2-cont/mikhailov Bair]
# Continuous normalizing flows [https://github.com/intsystems/IDA/blob/main-2024/essay-2-cont/khusainov/IDA_MIPT_week34_2.pdf Marat]
# Adjoint method and continuous backpropagation [https://github.com/intsystems/IDA/blob/main-2024/essay-2-cont/boeva/essay2_final.pdf Galina]
# Riemannian continuous models [https://github.com/intsystems/IDA/blob/main-2024/essay-2-cont/vladimirov/main-final.pdf Eduard]
====SINDy====
# Data-driven discovery of partial differential equations by Rudy et al. 2017 [https://www.science.org/doi/full/10.1126/sciadv.1602614 Science]
# Ensemble-SINDy by Fasel et al. 2021 [https://royalsocietypublishing.org/doi/epdf/10.1098/rspa.2021.0904 DOI]

===Physics-Informed models===
Third series
# PINNs as multimodels
# Spherical harmonics in p dimensions (an IMU example is welcome)
# PDF and Physics-Informed learning
# Integral Transforms in Physics-Informed learning

Talks
# Geometric Clifford Algebra Networks [https://github.com/intsystems/IDA/blob/main-2024/essay-3-pinn/vladimirov/main.pdf Eduard]
# Integral Transforms in Physics-Informed learning [https://github.com/intsystems/IDA/blob/main-2024/essay-3-pinn/boeva/essay3_boeva.pdf Galina]

===Multilinear models and topology===
Fourth series
# Clifford or Geometric algebra in machine learning
# Tensor models, tensor decomposition, and approximation (tensor PLS or CCA)
# Machine learning models for tensors: Field Equation (Yang-Mills Equations_
# Machine learning models for theoretical physics (Maxwell’s Equations, Navier-Stokes)
# Persistent homology and dimensionality reduction (say, arXiv:2302.03447 with embedding delays)

Talks
# Tensor models, tensor decomposition, and approximation [https://github.com/intsystems/IDA/blob/main-2024/essay-4-multilinear/vladimirov/main-final.pdf Eduard]
# Machine learning models for theoretical physics (Maxwell’s Equations, Navier-Stocks) [https://github.com/intsystems/IDA/blob/main-2024/essay-4-multilinear/boeva/essay4_final.pdf Galina]

===Generative and Riemannian models===
Fifth series
# Generative Riemannian models. How do we extract and use the distribution?
# Generative Canonical Correlation Analysis and its connection with the Riemannian spaces in the latent part
# Scoring-based Riemannian models. How do we extract and use the distribution?
# Generative convolutional models for tensors. Is there a continuous-time? (A variant is the Riemannian Residual Networks).
# Riemannian continuous normalizing flows. How do we generate a time series of a given distribution?

Talks
# Scoring-based Riemannian models [https://github.com/intsystems/IDA/blob/main-2024/essay-5-riemannian-generative/vladimirov/main.pdf Eduard]
# Riemannian continuous normalizing flows [https://github.com/intsystems/IDA/blob/main-2024/essay-5-riemannian-generative/boeva/essay5.pdf Galina]

===Operator learning===
An additional topic to summarise all the above. See the introduction in
# Neural operators [https://en.wikipedia.org/wiki/Neural_operators wiki]
# Operator Learning: Convolutional Neural Operators [https://medium.com/@bogdan.raonke/operator-learning-convolutional-neural-operators-for-robust-and-accurate-learning-of-pdes-ebbc43b57434 blog]
# Convolutional Neural Operators for robust and accurate learning of PDEs [https://arxiv.org/pdf/2302.01178 arxiv 2023]
# Representation Equivalent Neural Operators: a Framework for Alias-free Operator Learning [https://arxiv.org/pdf/2305.19913 arxiv 2023]
# PID: Proportional-Integral-Differential-equation modeling with operator learning

===Discussed literature===
# Generative CCA, diffusion, and flow by Galina [https://arxiv.org/html/2312.13455v1] [https://arxiv.org/pdf/2305.11832] [https://arxiv.org/pdf/1610.03454] [https://era.library.ualberta.ca/items/64d14f0d-eb08-4fba-92aa-c8e0d42af448]
# Kernel CCA in Hilbert and L2[a,b] spaces by Bair [https://proceedings.mlr.press/v28/chang13.pdf] [https://www.jmlr.org/papers/volume3/bach02a/bach02a.pdf]
# CCA versus Cross-Attention Transformers by Eduard [https://arxiv.org/pdf/1911.05544] [https://statisticalsupportandresearch.wordpress.com/wp-content/uploads/2017/06/k-v-mardia-j-t-kent-j-m-bibby-multivariate-analysis-probability-and-mathematical-statistics-academic-press-inc-1979.pdf] [http://vigir.missouri.edu/~gdesouza/Research/Conference_CDs/IEEE_WCCI_2020/IJCNN/Papers/N-20729.pdf]
# Ajoint method and continuous backpropagation by Galina [https://ilya.schurov.com/post/adjoint-method]
# Continuous normalizing flows by Galina [https://arxiv.org/pdf/2106.08462v2]
# Tensor models by Eduard [https://www.kolda.net/publication/TensorReview.pdf] [https://arxiv.org/pdf/1502.02330] [https://sci-hub.se/10.1109/tpami.2008.167]
# Navier-Stokes [https://arc.aiaa.org/doi/10.2514/6.2022-1436] [https://www.sci-hub.ru/10.1007/s00521-014-1762-2] [https://www.physicsbaseddeeplearning.org/references.html]
# Classics versus quantum by Galina
## [https://www.mdpi.com/1099-4300/18/1/34 Schroedinger vs. Navier–Stokes 2016]
## [https://arxiv.org/abs/1707.04474v1 Many-particle quantum hydrodynamics: Exact equations and pressure tensors 2019]
## [https://content.iospress.com/articles/asymptotic-analysis/asy14-2-01 Quantum hydrodynamics, Wigner transforms, the classical limit 1995]
## [https://link.springer.com/article/10.1007/s10440-019-00257-1 Geometry of Nonadiabatic Quantum Hydrodynamics 2019]
## [https://iopscience.iop.org/article/10.1088/1367-2630/16/6/063011 Theory of quantum friction 2014]
## [https://www.science.org/doi/10.1126/sciadv.aba3747 Minimal quantum viscosity from fundamental physical constants]
## [https://thesis.library.caltech.edu/10278/ Fluid Dynamics with Incompressible Schrödinger Flow 2017]
## [https://gamedev.ru/code/articles/shrodinger_hydrodynamics Гидродинамика Шрёдингера на пальцах]
# Riemannian continuous normalizing flows by Galina [http://bayesiandeeplearning.org/2016/papers/BDL_33.pdf] [https://aclanthology.org/N19-1025.pdf] [https://arxiv.org/pdf/2006.10605]

==Practical spatial-time series==
# A guide to state–space modeling of ecological time series, 2021 [https://doi.org/10.1002/ecm.1470 PDF], (Bayesian Kalman)
# Kalman Filtering and Smoothing, 2025 [https://arxiv.org/pdf/2405.08971 ArXiv] (Riemannian Kalman)

==Data collections==
# ClimateSet, 2023 [https://arxiv.org/pdf/2311.03721 ArXiv]

==References==
===General===
# Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems [https://arxiv.org/abs/2307.08423 arxiv 2023]
# Algebra, Topology, Differential Calculus, and Optimization Theory For Computer Science and Machine Learning [https://www.cis.upenn.edu/~jean/math-deep.pdf upenn 2024]
# The Elements of Differentiable Programming [https://arxiv.org/abs/2403.14606 arxiv 2024]
# The list from the previous year [https://github.com/intsystems/IDA/tree/main-2023 2023].
# Differential Geometry of Curves and Surfaces: Textbook, 2016 by Kristopher Tapp [https://link.springer.com/book/10.1007/978-3-319-39799-3]

===Prerequisites===
# Understanding Deep Learning ''by Simon J.D. Prince'' [https://udlbook.github.io/udlbook/ mit 2023]
# Deep Learning by ''C.M. and H. Bishops'' [https://www.bishopbook.com/ Springer 2024] (online version)
# A Geometric Approach to Differential Forms ''by David Bachman'' [https://arxiv.org/abs/math/0306194v1 arxiv 2013]
# Advanced Calculus: Geometric View ''by James J. Callahan'' [https://download.tuxfamily.org/openmathdep/calculus_advanced/Advanced_Calculus-Callahan.pdf pdf 2010], [https://download.tuxfamily.org/openmathdep/calculus_advanced/Advanced_Calculus-Callahan.pdf collection]
# Geometric Deep Learning by Michael M. Bronstein [https://arxiv.org/pdf/2104.13478 arxiv 2021]
=== Linear and bilinear models===
# A Tutorial on Independent Component Analysis [https://arxiv.org/abs/1404.2986 arxiv, 2014]
# On the Stability of Multilinear Dynamical Systems [https://arxiv.org/abs/2105.01041 arxiv 2022]
# Tensor-based Regression Models and Applications ''by Ming Hou'' Thèse [https://core.ac.uk/download/pdf/442636056.pdf Uni-Laval 2017] 
# Tensor Canonical Correlation Analysis for Multi-view Dimension Reduction [https://arxiv.org/pdf/1502.02330] (Semkin)
#Tensor Learning in Multi-view Kernel PCA [https://link.springer.com/chapter/10.1007/978-3-030-01421-6_21 arxiv 2018]
# Tensor decomposition of EEG signals: A brief review [http://dx.doi.org/10.1016/j.jneumeth.2015.03.018 2015]
====Spherical Harmonics====
# Spherical Harmonic Transforms: In JAX and PyTorch [https://medium.com/data-science/differentiable-and-accelerated-spherical-harmonic-transforms-c269393d08f1 Medium 2024]
# Spherical Harmonics in p Dimensions [https://arxiv.org/abs/1205.3548 arxiv 2012]
# Physics of simple pendulum: a case study of nonlinear dynamics [https://www.researchgate.net/publication/332766499_Physics_of_simple_pendulum_a_case_study_of_nonlinear_dynamics RG 2008]
# Time series forecasting using manifold learning, 2021 [https://arxiv.org/pdf/2110.03625 arxiv]
# Time-series forecasting using manifold learning, radial basis function interpolation, and geometric harmonics [https://doi.org/10.1063/5.0094887 2022 Chaos AIP]

====State Space Models====
# Missing Slice Recovery for Tensors Using a Low-rank Model in Embedded Space [https://arxiv.org/abs/1804.01736 arxiv 2018]
#Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks by A.R. Voelker et al., 2019 [https://papers.nips.cc/paper_files/paper/2019/file/952285b9b7e7a1be5aa7849f32ffff05-Paper.pdf NeurIPS]

====SSM Generative Models ====
# Masked Autoregressive Flow for Density Estimation [https://arxiv.org/abs/1705.07057 arxiv 2017]
====SSM+Riemann+Gaussian process regression====
* Time-series forecasting using manifold learning, radial basis function interpolation, and geometric harmonics by Ioannis G. Kevrekidis,3 and Constantinos Siettos, 2022 [https://pubs.aip.org/aip/cha/article-pdf/doi/10.1063/5.0094887/16497596/083113_1_online.pdf pdf]

===Physics-Informed Neural Networks===
# Neural partial differential equations with functional convolution [https://openreview.net/pdf?id=D4A-v0kltaX ICLP]
# Solving PDEs by variational physics-informed neural networks: an a posteriori error analysis [https://link.springer.com/article/10.1007/s11565-022-00441-6?fromPaywallRec=true PDF] plus several links to the books on the subject inside
# Predicting the nonlinear dynamics of spatiotemporal PDEs via physics-informed informer networks [https://link.springer.com/article/10.1007/s11071-024-10655-2?fromPaywallRec=true PDF]
# Three ways to solve partial differential equations with neural networks — A review [https://arxiv.org/abs/2102.11802 arxiv 2021]
# NeuPDE: Neural Network Based Ordinary and Partial Differential Equations for Modeling Time-Dependent Data [https://arxiv.org/abs/1908.03190 arxiv 2019]
# Physics-based deep learning [https://www.physicsbaseddeeplearning.org/intro-teaser.html code]
# PINN by Steve Burton [https://www.youtube.com/watch?v=g-S0m2zcKUg&list=PLMrJAkhIeNNQ0BaKuBKY43k4xMo6NSbBa&index=3 yt]
# Process Model Inversion in the Data-Driven Engineering Context for Improved Parameter Sensitivities [https://www.mdpi.com/2227-9717/10/9/1764 mdpi processes 2022] ('''nice connection pictures''')
# Physics-based Deep Learning [https://www.physicsbaseddeeplearning.org/intro.html github]
# Integral Transforms in a Physics-Informed (Quantum) Neural Network setting [https://arxiv.org/pdf/2206.14184 arxiv 2022]
# Lectures ny Stephen Brunton [https://www.youtube.com/watch?v=fiX8c-4K0-Q&list=PLMrJAkhIeNNQ0BaKuBKY43k4xMo6NSbBa&index=5 AI/ML+Physics], [https://www.youtube.com/watch?v=3SNkQ8jhKXc&list=PLMrJAkhIeNNQ0BaKuBKY43k4xMo6NSbBa&index=7 Part 4], [https://www.youtube.com/watch?v=nmvs0vrBT18 Basic PDEs], [https://www.youtube.com/watch?v=pvrIagjEk4c PDE Overview],

=== Riemmanian models===
# Riemannian Continuous Normalizing Flows [https://arxiv.org/abs/2006.10605 arxiv 2020]
# Residual Riemannian Networks [https://arxiv.org/pdf/2310.10013 arxiv 2023] 
=== Continous time, Neural ODE===
# Neural Spatio-Temporal Point Processes ''by Ricky Chen et al.'' [https://arxiv.org/abs/2011.04583 iclr 2021] (likelihood for time and space)
# Neural Ordinary Differential Equations ''by Ricky Chen et al.'' [https://arxiv.org/abs/1806.07366 arxiv 2018]
# Neural Controlled Differential Equations for Irregular Time Series 'Patrick Kidger et al.'' [https://arxiv.org/abs/2005.08926 arxiv 2020][https://github.com/patrick-kidger/NeuralCDE github]
# Diffusion Normalizing Flow [https://arxiv.org/pdf/2110.07579 arxiv 2021]
# Differentiable Programming for Differential Equations: A Review [https://arxiv.org/abs/2406.09699 arxiv 2024]
# (code tutorial) Deep Implicit Layers - Neural ODEs, Deep Equilibrium Models, and Beyond [https://implicit-layers-tutorial.org/ nips 2020]
# (code tutorial) [https://www.physicsbaseddeeplearning.org/overview-ns-forw.html 2021]
# Neural CDE and tensors [https://ieeexplore.ieee.org/abstract/document/9979806 IEEE], [https://ieeexplore.ieee.org/abstract/document/9533771 IEEE]
# Latent ODEs for Irregularly-Sampled Time Series [https://proceedings.neurips.cc/paper_files/paper/2019/file/42a6845a557bef704ad8ac9cb4461d43-Paper.pdf 2019]

=== Graph and PDEs ===
# Fourier Neural Operator for Parametric Partial Differential Equations [https://arxiv.org/abs/2010.08895 arxiv 2020]
# Masked Attention is All You Need for Graphs [https://arxiv.org/abs/2402.10793 arxiv 2024]
=== Neural SDE===
# Approximation of Stochastic Quasi-Periodic Responses of Limit Cycles in Non-Equilibrium Systems under Periodic Excitations and Weak Fluctuations [https://doi.org/10.3390/e19060280 mdpi entropy 2017] (great illustrations on the stochastic nature of a simple phase trajectory)
# Approximation of Stochastic Quasi-Periodic Responses of Limit Cycles in Non-Equilibrium Systems under Periodic Excitations and Weak Fluctuations [https://doi.org/10.3390/e19060280 mdpi entropy 2017] (great illustrations on the stochastic nature of a simple phase trajectory)
# Neural SDEs for Conditional Time Series Generation [https://arxiv.org/abs/2301.01315 arxiv 2023] code [https://github.com/pere98diaz/Neural-SDEs-for-Conditional-Time-Series-Generation-and-the-Signature-Wasserstein-1-metric github LSTM - CSig-WGAN]
# Neural SDEs as Infinite-Dimensional GANs [https://arxiv.org/pdf/2102.03657 2021]
# Efficient and Accurate Gradients for Neural SDEs ''by Patrick Kidger'' [https://arxiv.org/pdf/2105.13493 arxiv 2021] code [https://docs.kidger.site/diffrax/examples/neural_sde/ diffrax]
=== Chains and homology===
# Operator Learning: Algorithms and Analysis [https://arxiv.org/pdf/2402.15715 arxiv 2024]
# Hi-res weather: Operator learning [https://arxiv.org/pdf/2202.11214 arxiv 2022]
# Homotopy theory for beginners by J.M. Moeller [https://web.math.ku.dk/~moller/e01/algtopI/comments.pdf ku.dk 2015] (is it a pertinent link?)
# Explorations in Homeomorphic Variational Auto-Encoding [https://arxiv.org/abs/1807.04689 arxiv 2018]
# Special Finite Elements for Dipole Modelling ''master thesis Bauer'' [https://www.sci.utah.edu/~wolters/PaperWolters/2012/BauerMaster.pdf 2011]
# Selecting embedding delays: An overview of embedding techniques and a new method using persistent homology [https://arxiv.org/pdf/2302.03447v1 arxiv 2023] (Denis)
# (code) Clifford Algebra for Python https://clifford.readthedocs.io/en/v1.1.0/

===Appendix===
# Neural Memory Networks [https://cs229.stanford.edu/proj2015/367_report.pdf stanford reports 2019]
# An Elementary Introduction to Information Geometry ''by Frank Nielsen'' [An Elementary Introduction to Information Geometry Frank Nielsen [https://doi.org/10.3390/e22101100 mdpi entropy]
# The Many Faces of Information Geometry ''by Frank Nielsen'' [https://www.ams.org/journals/notices/202201/rnoti-p36.pdf ams 2022] (short version)
# Geometric Clifford Algebra Networks [https://arxiv.org/abs/2302.06594 arxiv 3022]
# Clifford Algebras and Dimensionality Reduction for Signal Separation ''by [https://www.math.uni-hamburg.de/home/guillemard/ M. Guillemard]'' [https://www.math.uni-hamburg.de/home/guillemard/papers/clifford7.pdf Uni-Hamburg 2010][https://www.math.uni-hamburg.de/home/guillemard/clifford/ code]
# Special Finite Elements for Dipole Modelling ''by Martin Bauer'' Master Thesis [https://www.sci.utah.edu/~wolters/PaperWolters/2012/BauerMaster.pdf Erlangen 2012] diff p-form must read
# Bayesian model selection for complex dynamic systems [https://www.nature.com/articles/s41467-018-04241-5 2018]
# Visualizing 3-Dimensional Manifolds ''by Dugan J. Hammock'' [https://archive.bridgesmathart.org/2013/bridges2013-551.pdf 2013 umass]
# At the Interface of Algebra and Statistics by ''T-D. Bradley'' [https://arxiv.org/abs/2004.05631 arxiv 2020]
# Time Series Handbook by Borja, 2021 [https://github.com/phdinds-aim/time_series_handbook github]
# Physics-informed machine learning [https://www.nature.com/articles/s42254-021-00314-5 Nature reviews: Physics 2021]
# Integral Transforms in a Physics-Informed (Quantum) Neural Network setting: Applications & Use-Cases [https://arxiv.org/abs/2206.14184 arxiv 2022]
# Deep Efficient Continuous Manifold Learning for Time Series Modeling [https://arxiv.org/abs/2112.03379 arxiv 2021]
====Causality====
# Toward Causal Representation Learning [https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9363924 2021]
# See the Sugihara collection

==Basics==
Collection of wiki-links
===Time Series===
# [https://en.wikipedia.org/wiki/Spectral_submanifold Spectral submanifold] (with nonlinear dimensional reduction like [https://en.wikipedia.org/wiki/Self-organizing_map som])
# [https://en.wikipedia.org/wiki/Lagrangian_coherent_structure Lagrangian coherent structure] (software below)
===Signal Processing===
#[https://en.wikipedia.org/wiki/Estimation_of_signal_parameters_via_rotational_invariance_techniques Estimation of signal parameters via rotational invariance techniques]
#[https://en.wikipedia.org/wiki/Reproducing_kernel_Hilbert_space Reproducing kernel Hilbert space]
#[https://en.wikipedia.org/wiki/Kernel_principal_component_analysis Kernel principal component analysis]
#[https://en.wikipedia.org/wiki/Gram_matrix Gram matrix]
#[https://en.wikipedia.org/wiki/Generalized_pencil-of-function_method Generalized pencil-of-function method]
#[https://en.wikipedia.org/wiki/Wavelet_transform Wavelet transform]

===Differential Geometry===
#[https://en.wikipedia.org/wiki/Pushforward_(differential) Pushforward (differential)]
#[https://en.wikipedia.org/wiki/Pullback_bundle Ffibers, Bundles, Sheaves]
#[https://en.wikipedia.org/wiki/Homology_(mathematics) Homology]
#[https://en.wikipedia.org/wiki/Topological_data_analysis Topological data analysis]
#[https://en.wikipedia.org/wiki/Conditional_mutual_information Conditional mutual information]
#[https://en.wikipedia.org/wiki/Convergent_cross_mapping Convergent cross mapping]
#[https://en.wikipedia.org/wiki/Differential_form Differential form]
#[https://en.wikipedia.org/wiki/Total_derivative The total derivative as a differential form]
#[https://en.wikipedia.org/wiki/Riemannian_manifold #Riemannian_metrics Riemannian_metrics]
# Multidimensional Differential and Integral Calculus: A Practical Approach (textbook)

===Probabilistical Decompisition===
#[https://en.wikipedia.org/wiki/Wasserstein_metric Wasserstein metric]
#[https://en.wikipedia.org/wiki/Mutual_information Mutual information]
#[https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant Jacobian]
#[https://en.wikipedia.org/wiki/Fisher_information Fisher information]
# also dobrushin stratonovich wasserstein
# also fluid dymanics, transportation theory

===Tutorials===
# [https://www.connectedpapers.com/main/d86084808994ac54ef4840ae65295f3c0ec4decd/Physics%20informed-neural-networks%3A-A-deep-learning-framework-for-solving-forward-and-inverse-problems-involving-nonlinear-partial-differential-equations/graph Connected papers search]
# Operator Learning via Physics-Informed DeepONet: Let’s Implement It From Scratch [https://towardsdatascience.com/operator-learning-via-physics-informed-deeponet-lets-implement-it-from-scratch-6659f3179887 Medium]

==Tools==
# [https://github.com/ilkhem/icebeem icebeem]
# [https://github.com/ilkhem/ivae ivae]
# [https://github.com/kondratevakate/fmri-component-# fmri-component]
# analysis/blob/master/VAE_for_fMRI/dataset/train/Bystrova0_y-axis.png
# [https://fr.mathworks.com/help/deeplearning/ug/dynamical-system-modeling-using-neural-ode.html Neural ODE in Matlab]
# [https://github.com/pyRiemann/pyRiemann pyRiemann]
# causality inference [https://peps.python.org/pep-0484/#scoping-rules-for-type-variables peps]
# [https://github.com/xai-org/grok-1 LMM grok-1 with weights]

===Turbulence===
# Runko: Modern multiphysics toolbox for plasma simulations [https://github.com/hel-astro-lab/runko GitHub]
# 2d-turb-PINN by Parfenyev [https://github.com/parfenyev/2d-turb-PINN/tree/main GitHub]

====Physics and Engineering of Turbulence====
# Fundamentals of Fluid_Mechanics, 2013 [https://students.aiu.edu/submissions/profiles/resources/onlineBook/L5g8S6_Fundamentals_of_Fluid_Mechanics-_7.pdf PDF]
# Introduction ot Fluid Mechanics, 2004 by R. Fox et al. [https://ahsheikh.github.io/Courses/NumMod/FM_RobertFox.pdf PDF]
# Computational fluid dynamics, 1995 by John D. Anderson, Jr. [https://www.airloads.net/Downloads/Textbooks/Computational-Fluid-Dynamics-the-Basics-With-Applications-Anderson-J-D.pdf PDF]
# Fluid-dynamic drag, 1965 by S.F. Hoerner [https://ia800606.us.archive.org/17/items/FluidDynamicDragHoerner1965/Fluid-dynamic_drag__Hoerner__1965_text.pdf PDF]
# TorchDyn: A Neural Differential Equations Library [https://arxiv.org/abs/2009.09346 arXiv] [https://github.com/DiffEqML/torchdyn github]
## Parameterized Neural Ordinary Differential Equations: Applications to Computational Physics Problems [https://arxiv.org/abs/2010.14685 ArXiv]
## Turbulence forecasting via Neural ODE [https://arxiv.org/abs/1911.05180v1 ArXiv]
## (not the same) Hamiltonian Neural Networks [https://arxiv.org/abs/1906.01563 ArXiv]

2025-10-01T15:34:32Z

Wiki: /* Key reviews */

Channels:
* [https://t.me/+XyXmEXRlrXB9dZKD The 2025 course seminar]
* [https://t.me/+XyXmEXRlrXB9dZKD The chat-link FDA group]
When:
* September 4, 11, 18, 25 on Thursdays at 10:30 [https://m1p.org/go_zoom m1p.org/go_zoom]
* October (most likely) on Saturdays at 10:30

===Foundation models for spatial-time series===
Foundation AI models are universal models to solve a wide set of problems. This project proposes to investigate the theoretical properties of foundation models. The domain to model is a spatial-time series. These data are used in various scientific disciplines and serve to generalise scientific knowledge and make forecasts. The essential problems, formulated as user requests that solve a foundation model, are forecasting and generation of time series; analysis and classification of time series; detection of change point, and causal inference. To solve these problems, the foundation AI models are trained on massive datasets. The main goal of this project is to compare various architectures of foundation models to find an optimal architecture that solves the listed problems for a wide range of spatial time series.

===Functional data analysis===
The statistical analysis of spatial time series requires additional methods of data analysis. First, we suppose time is continuous, put the state space changes <math>\frac{d\mathbf{x}}{dt}</math>, and use neural ordinary and stochastic differential equations. Second, we analyze a multivariate and multidimensional time series and use the tensor representation and tensor analysis. Third, since the time series have significant cross-correlation, we model them in the Riemannian space. Fourth, medical time series are periodic; the base model is the pendulum model, <math>\frac{d^2x}{dt^2}=-c\sin{x}</math>. We use physics-informed neural networks to approximate data. Fifth, the practical experiments involve multiple data sources. We use canonical correlation analysis with a latent state space. This space aligns the source and target spaces and generates data in the source and target manifolds.

=== Applications ===
This field of Machine Learning applies to any field where the measurements have continuous time and space data acquired from multimodal sources: climate modeling, neural interfaces, solid-state physics, electronics, fluid dynamics, and many more. We will carefully collect both the theory and its practice.

== Fall 2025: Foundation models for time series ==
=== Topics top discuss===
# State Space Models, Convolution, SSA, SSM (Spectral Submanifolds)
# Neural and Controlled ODE, Neural PDE, Geometric Learning
# Operator Learning, Physics-informed learning, and multimodeling
# Spatial-Temporal Graph Modeling: Graph convolution and metric tensors
# Riemmannian models; time series generation
# AI for science: mathematical modelling principles

Outside the course: data-driven tensor analysis, differential forms, and spinors

=== State of the Art in 2025===
In December 2024, a NeurIPS workshop "Foundational models for science" reflected this theme:
# Foundation Models for Science: Progress, Opportunities, and Challenges [https://neurips.cc/virtual/2024/workshop/84714 URL]
# Foundation Models for the Earth system [https://neurips.cc/virtual/2024/107817 UPL, no paper]
# Foundation Methods for foundation models for scientific machine learning [https://neurips.cc/virtual/2024/107819 URL, no paper]
# AI-Augmented Climate simulators and emulators [https://neurips.cc/virtual/2024/107822 URL, no paper]
# Provable in-context learning of linear systems and linear elliptic PDEs with transformers [https://openreview.net/forum?id=xDstmuxn1D NIPS]
# VSMNO: Solving PDE by Utilizing Spectral Patterns of Different Neural Operators [https://openreview.net/pdf?id=oCT8pYix5e NIPS]

=== March 2025 Physics problem Simulations ===
# The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning [https://arxiv.org/pdf/2412.00568 ArXiv], [https://polymathic-ai.org/the_well/data_format/ Code]
# Polymatic Advancing Science through Multi‑Disciplinary AI [https://polymathic-ai.org/ blog]
# Long Term Memory: The Foundation of AI Self-Evolution [https://arxiv.org/html/2410.15665v1 ArXiv]
# Distilling Free-Form Natural Laws from Experimental Data, 2009 [https://www.science.org/doi/abs/10.1126/science.1165893 Science], [https://arxiv.org/pdf/1210.7273 comment], [https://medium.com/@lotussavy/distilling-free-form-natural-laws-from-experimental-data-f55341ae0fa6 medium]
# Deep learning for universal linear embeddings of nonlinear dynamics [https://www.nature.com/articles/s41467-018-07210-0 nature]
# A comparison of data-driven approaches to build low-dimensional ocean models, 2021 by Pavel Berloff [https://arxiv.org/abs/2108.00818 ArXiv], talk by Daniil Dorin for S.V. Fortova
# Applications of Deep Learning to Ocean Data Inference and Subgrid Parameterization by Thomas Bolton and Laure Zanna, 2018 [https://eartharxiv.org/repository/view/1142/ preprint], talk by Nilita Kiselev
# On energy-aware hybrid models by Shevchenko,2024 [https://doi.org/10.1029/2024MS004306 doi], talk by Mariya Nikitina
# Science: NASA satellites and computers have provided us with these mesmerizing swirls that cover our planet—but this isn’t star stuff. Each color represents a different aerosol that was floating in the atmosphere above our heads from 1 August to 14 September 2024 [https://www.facebook.com/reel/4083421318545496 video]

===Spatial-Temporal Graph Modeling===
# Graph WaveNet for Deep Spatial-Temporal Graph Modeling [https://arxiv.org/abs/1906.00121 ArXiv]
# Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting [https://arxiv.org/abs/1707.01926 ICLR]
# Time-SSM: Simplifying and Unifying State Space Models for Time Series Forecasting [https://arxiv.org/pdf/2405.16312 ArXiv] [https://github.com/haller-group/SSMTool-2.4 SSMTool]
# State Space Reconstruction for Multivariate Time Series Prediction [https://arxiv.org/abs/0809.2220 ArXiv]](Denis)
# Longitudinal predictive modeling of tau progression along the structural connectome by Joyita Dutta [https://www.sciencedirect.com/science/article/pii/S1053811921004031?via%3Dihub 2021]

== Key reviews ==
# 2018. Diffusion Convolutional Recurrent Neural Network [https://arxiv.org/pdf/1707.01926 ICLR]
# 2021. Neural Partial Differential Equations with Functional Convolution [https://openreview.net/pdf?id=D4A-v0kltaX ICLR]
# 2018. Graph WaveNet for Deep Spatial-Temporal Graph Modeling [https://arxiv.org/pdf/1906.00121 ArXiV]
# 2021. Neural Rough Differential Equations for Long Time Series (comparison)
# 2022. Time Series Forecasting Using Manifold Learning, Radial Basis Function Interpolation and Geometric Harmonics [https://doi.org/10.1063/5.0094887 doi] (all basic models + superpositions review)

=== Catch-up ===
If you are not familiar with the LLM and GPT:
# Build an LLM from scratch by Sebastian Raschka, 2025 [https://github.com/rasbt/LLMs-from-scratch/ github]
# Agentic Design Patterns by Antonio Gulli, 2025 [https://docs.google.com/document/d/1rsaK53T3Lg5KoGwvf8ukOUvbELRtH-V0LnOIFDxBryE/preview?tab=t.0 docx]
For fun, the vibe coding: [https://cursor.com/home 1], [https://windsurf.com/ 2], [https://www.augmentcode.com/ 3], [https://www.augmentcode.com/install 4], also [https://arxiv.org/abs/2501.09223 Foundations of LLM] and [https://www.youtube.com/@AndrejKarpathy Karpathy's] project
[https://github.com/karpathy/nanoGPT nanoGPT]

== Work arrangements==
{|class="wikitable"
|-
| ''' Week '''
| ''' Date '''
| ''' Theme '''
| ''' Delivery '''
|-
| 1
| sep 4
| Preliminary discussion [https://github.com/vadim-vic/Foundation-ts/tree/main/doc/Foundation_models_for_time_series_Week_1.pdf pdf]
|
|-
| 2
| sep 11
| Problem statement [https://github.com/vadim-vic/Foundation-ts/tree/main/doc/Foundation_models_for_time_series_Week_2.pdf pdf]
|
|-
| 3
| sep 18
| Preliminary solution
| Group talk and discussion
|-
| 4
| oct 2
| Minimum deployment
| Group report
|-
| 5
| oct 7+
| FDA
| Personal talks

|-
| 13
| nov 29
| Final discussion
| Group talks
|-

|-
|}



===Structure of seminars ===
The semester lasts 12 weeks, and six couple of weeks are for homework.
* Odd week: introduction to the topic and a handout of a theme for the homework.
* Every week: a discussion of the essay, collecting the list of improvements to each essay.
* Odd week: a discussion of the improved essay, putting the essays into a joint structure.

===Scoring===
The group activity is evaluated by cross-ranking with the Kemeni median score. The personal talks give a score.

== Week 3==
Homework 1
# Form a group
# Discuss the goals of the project and a solution ([see the problem statement])
# Make a review of various ways to solve the problem
# Select an LLM-GPT
# Run the code to check if it works
## Store the code in the group repository
## Store the talk slides/report, too
# Make a 10-minute talk about
## Functionality and architecture of the model
## Why did you select this model
## The alternative models to select from






====Requirements for the text and the discussion====
# Comprehensive explanation of the method or the question we discuss
# Only the principle, no experiments
# Two-page text (more or less)
# The reader is a second or third-year student
# The picture is obligatory
# However, a brief reference to some deep learning structure is welcome
# Talk could be a slide or a text itself
# The list of references with doi
# Tell how it was generated
# Observing a gap, put a note about it (to question later)

==== Style remarks for the essays ====
Automatic generation of mediocre-quality texts increased the requirements for the quality of the new messages. It makes novelty rare and makes the authorship appreciated. But it simplifies the way of delivering. So, since textbook generation has become simple, we will use generative chat to train our skills in reader persuasion. The reader is our MS thesis defense committee.



==Table of topics for seminars==
In these ten weeks, we will discuss the next five topics:
# Multimodal data
# Continous time and space models
# Physics-informed models
# Multilinear models
# Riemannian spaces

Note that all these items enlighten the stochastic-deterministic decomposition. So the questions include three parts:
# deterministic model,
# generative model,
# stochastic-deterministic decomposition method.
See the questions below for your reference.

=== Multimodal data ===
First series
# Canonical Correlation Analysis
# CCA in tensor representation
# Kernel CCA in Hilbert and L2[a,b] spaces
# CCA versus Cross-Attention Transformers
# Generative CCA, diffusion, and flow
# Comparative analysis of variants of CCA, like PLS and others
# Functional PCA



Talks
# Canonical Correlation Analysis in tensor representation [https://github.com/intsystems/IDA/blob/main-2024/essay-1-cca/khusainov/IDA_MIPT_week12_2.pdf Marat]
# Kernel CCA in Hilbert and L2[a,b] spaces [https://github.com/intsystems/IDA/blob/main-2024/essay-1-cca/mikhailov/main.pdf Bair]
# CCA versus Cross-Attention Transformers [https://github.com/intsystems/IDA/blob/main-2024/essay-1-cca/vladimirov/main-final.pdf Eduard]
# Generative CCA, diffusion, and flow [https://github.com/intsystems/IDA/blob/main-2024/essay-1-cca/boeva/VCCA_FLOWS.pdf Galina], [https://github.com/intsystems/IDA/blob/main-2024/essay-1-cca/boeva/essay1.pdf Galina]
# Functional PCA [https://github.com/intsystems/IDA/blob/main-2024/essay-1-cca/karimov/paper.pdf Parviz]

=== Continous models ===
Second series
# Neural ODE
# Continous state space models
# Continous normalizing flows
# Adjoint method and continuous backpropagation
# Neural Delayed Differential Equations 
# Neural PDE
# S4 and Hippo models [https://doi.org/10.48550/arXiv.2206.12037], [https://github.com/HazyResearch/state-spaces] (LSSL, SaShiMi, DSS, HTTYH, S4D, and S4ND)
# Riemannian continuous models

Talks
# Continuous state space models [https://github.com/intsystems/IDA/tree/main-2024/essay-2-cont/mikhailov Bair]
# Continuous normalizing flows [https://github.com/intsystems/IDA/blob/main-2024/essay-2-cont/khusainov/IDA_MIPT_week34_2.pdf Marat]
# Adjoint method and continuous backpropagation [https://github.com/intsystems/IDA/blob/main-2024/essay-2-cont/boeva/essay2_final.pdf Galina]
# Riemannian continuous models [https://github.com/intsystems/IDA/blob/main-2024/essay-2-cont/vladimirov/main-final.pdf Eduard]

===Physics-Informed models===
Third series
# PINNs as multimodels
# Spherical harmonics in p dimensions (an IMU example is welcome)
# PDF and Physics-Informed learning
# Integral Transforms in Physics-Informed learning

Talks
# Geometric Clifford Algebra Networks [https://github.com/intsystems/IDA/blob/main-2024/essay-3-pinn/vladimirov/main.pdf Eduard]
# Integral Transforms in Physics-Informed learning [https://github.com/intsystems/IDA/blob/main-2024/essay-3-pinn/boeva/essay3_boeva.pdf Galina]

===Multilinear models and topology===
Fourth series
# Clifford or Geometric algebra in machine learning
# Tensor models, tensor decomposition, and approximation (tensor PLS or CCA)
# Machine learning models for tensors: Field Equation (Yang-Mills Equations_
# Machine learning models for theoretical physics (Maxwell’s Equations, Navier-Stokes)
# Persistent homology and dimensionality reduction (say, arXiv:2302.03447 with embedding delays)

Talks
# Tensor models, tensor decomposition, and approximation [https://github.com/intsystems/IDA/blob/main-2024/essay-4-multilinear/vladimirov/main-final.pdf Eduard]
# Machine learning models for theoretical physics (Maxwell’s Equations, Navier-Stocks) [https://github.com/intsystems/IDA/blob/main-2024/essay-4-multilinear/boeva/essay4_final.pdf Galina]

===Generative and Riemannian models===
Fifth series
# Generative Riemannian models. How do we extract and use the distribution?
# Generative Canonical Correlation Analysis and its connection with the Riemannian spaces in the latent part
# Scoring-based Riemannian models. How do we extract and use the distribution?
# Generative convolutional models for tensors. Is there a continuous-time? (A variant is the Riemannian Residual Networks).
# Riemannian continuous normalizing flows. How do we generate a time series of a given distribution?

Talks
# Scoring-based Riemannian models [https://github.com/intsystems/IDA/blob/main-2024/essay-5-riemannian-generative/vladimirov/main.pdf Eduard]
# Riemannian continuous normalizing flows [https://github.com/intsystems/IDA/blob/main-2024/essay-5-riemannian-generative/boeva/essay5.pdf Galina]

===Operator learning===
An additional topic to summarise all the above. See the introduction in
# Neural operators [https://en.wikipedia.org/wiki/Neural_operators wiki]
# Operator Learning: Convolutional Neural Operators [https://medium.com/@bogdan.raonke/operator-learning-convolutional-neural-operators-for-robust-and-accurate-learning-of-pdes-ebbc43b57434 blog]
# Convolutional Neural Operators for robust and accurate learning of PDEs [https://arxiv.org/pdf/2302.01178 arxiv 2023]
# Representation Equivalent Neural Operators: a Framework for Alias-free Operator Learning [https://arxiv.org/pdf/2305.19913 arxiv 2023]
# PID: Proportional-Integral-Differential-equation modeling with operator learning

===Discussed literature===
# Generative CCA, diffusion, and flow by Galina [https://arxiv.org/html/2312.13455v1] [https://arxiv.org/pdf/2305.11832] [https://arxiv.org/pdf/1610.03454] [https://era.library.ualberta.ca/items/64d14f0d-eb08-4fba-92aa-c8e0d42af448]
# Kernel CCA in Hilbert and L2[a,b] spaces by Bair [https://proceedings.mlr.press/v28/chang13.pdf] [https://www.jmlr.org/papers/volume3/bach02a/bach02a.pdf]
# CCA versus Cross-Attention Transformers by Eduard [https://arxiv.org/pdf/1911.05544] [https://statisticalsupportandresearch.wordpress.com/wp-content/uploads/2017/06/k-v-mardia-j-t-kent-j-m-bibby-multivariate-analysis-probability-and-mathematical-statistics-academic-press-inc-1979.pdf] [http://vigir.missouri.edu/~gdesouza/Research/Conference_CDs/IEEE_WCCI_2020/IJCNN/Papers/N-20729.pdf]
# Ajoint method and continuous backpropagation by Galina [https://ilya.schurov.com/post/adjoint-method]
# Continuous normalizing flows by Galina [https://arxiv.org/pdf/2106.08462v2]
# Tensor models by Eduard [https://www.kolda.net/publication/TensorReview.pdf] [https://arxiv.org/pdf/1502.02330] [https://sci-hub.se/10.1109/tpami.2008.167]
# Navier-Stokes [https://arc.aiaa.org/doi/10.2514/6.2022-1436] [https://www.sci-hub.ru/10.1007/s00521-014-1762-2] [https://www.physicsbaseddeeplearning.org/references.html]
# Classics versus quantum by Galina
## [https://www.mdpi.com/1099-4300/18/1/34 Schroedinger vs. Navier–Stokes 2016]
## [https://arxiv.org/abs/1707.04474v1 Many-particle quantum hydrodynamics: Exact equations and pressure tensors 2019]
## [https://content.iospress.com/articles/asymptotic-analysis/asy14-2-01 Quantum hydrodynamics, Wigner transforms, the classical limit 1995]
## [https://link.springer.com/article/10.1007/s10440-019-00257-1 Geometry of Nonadiabatic Quantum Hydrodynamics 2019]
## [https://iopscience.iop.org/article/10.1088/1367-2630/16/6/063011 Theory of quantum friction 2014]
## [https://www.science.org/doi/10.1126/sciadv.aba3747 Minimal quantum viscosity from fundamental physical constants]
## [https://thesis.library.caltech.edu/10278/ Fluid Dynamics with Incompressible Schrödinger Flow 2017]
## [https://gamedev.ru/code/articles/shrodinger_hydrodynamics Гидродинамика Шрёдингера на пальцах]
# Riemannian continuous normalizing flows by Galina [http://bayesiandeeplearning.org/2016/papers/BDL_33.pdf] [https://aclanthology.org/N19-1025.pdf] [https://arxiv.org/pdf/2006.10605]

==Practical spatial-time series==
# A guide to state–space modeling of ecological time series, 2021 [https://doi.org/10.1002/ecm.1470 PDF], (Bayesian Kalman)
# Kalman Filtering and Smoothing, 2025 [https://arxiv.org/pdf/2405.08971 ArXiv] (Riemannian Kalman)

==Data collections==
# ClimateSet, 2023 [https://arxiv.org/pdf/2311.03721 ArXiv]

==References==
===General===
# Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems [https://arxiv.org/abs/2307.08423 arxiv 2023]
# Algebra, Topology, Differential Calculus, and Optimization Theory For Computer Science and Machine Learning [https://www.cis.upenn.edu/~jean/math-deep.pdf upenn 2024]
# The Elements of Differentiable Programming [https://arxiv.org/abs/2403.14606 arxiv 2024]
# The list from the previous year [https://github.com/intsystems/IDA/tree/main-2023 2023].
# Differential Geometry of Curves and Surfaces: Textbook, 2016 by Kristopher Tapp [https://link.springer.com/book/10.1007/978-3-319-39799-3]

===Prerequisites===
# Understanding Deep Learning ''by Simon J.D. Prince'' [https://udlbook.github.io/udlbook/ mit 2023]
# Deep Learning by ''C.M. and H. Bishops'' [https://www.bishopbook.com/ Springer 2024] (online version)
# A Geometric Approach to Differential Forms ''by David Bachman'' [https://arxiv.org/abs/math/0306194v1 arxiv 2013]
# Advanced Calculus: Geometric View ''by James J. Callahan'' [https://download.tuxfamily.org/openmathdep/calculus_advanced/Advanced_Calculus-Callahan.pdf pdf 2010], [https://download.tuxfamily.org/openmathdep/calculus_advanced/Advanced_Calculus-Callahan.pdf collection]
# Geometric Deep Learning by Michael M. Bronstein [https://arxiv.org/pdf/2104.13478 arxiv 2021]
=== Linear and bilinear models===
# A Tutorial on Independent Component Analysis [https://arxiv.org/abs/1404.2986 arxiv, 2014]
# On the Stability of Multilinear Dynamical Systems [https://arxiv.org/abs/2105.01041 arxiv 2022]
# Tensor-based Regression Models and Applications ''by Ming Hou'' Thèse [https://core.ac.uk/download/pdf/442636056.pdf Uni-Laval 2017] 
# Tensor Canonical Correlation Analysis for Multi-view Dimension Reduction [https://arxiv.org/pdf/1502.02330] (Semkin)
#Tensor Learning in Multi-view Kernel PCA [https://link.springer.com/chapter/10.1007/978-3-030-01421-6_21 arxiv 2018]
# Tensor decomposition of EEG signals: A brief review [http://dx.doi.org/10.1016/j.jneumeth.2015.03.018 2015]
====Spherical Harmonics====
# Spherical Harmonic Transforms: In JAX and PyTorch [https://medium.com/data-science/differentiable-and-accelerated-spherical-harmonic-transforms-c269393d08f1 Medium 2024]
# Spherical Harmonics in p Dimensions [https://arxiv.org/abs/1205.3548 arxiv 2012]
# Physics of simple pendulum: a case study of nonlinear dynamics [https://www.researchgate.net/publication/332766499_Physics_of_simple_pendulum_a_case_study_of_nonlinear_dynamics RG 2008]
# Time series forecasting using manifold learning, 2021 [https://arxiv.org/pdf/2110.03625 arxiv]
# Time-series forecasting using manifold learning, radial basis function interpolation, and geometric harmonics [https://doi.org/10.1063/5.0094887 2022 Chaos AIP]

====State Space Models====
# Missing Slice Recovery for Tensors Using a Low-rank Model in Embedded Space [https://arxiv.org/abs/1804.01736 arxiv 2018]
#Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks by A.R. Voelker et al., 2019 [https://papers.nips.cc/paper_files/paper/2019/file/952285b9b7e7a1be5aa7849f32ffff05-Paper.pdf NeurIPS]

====SSM Generative Models ====
# Masked Autoregressive Flow for Density Estimation [https://arxiv.org/abs/1705.07057 arxiv 2017]
====SSM+Riemann+Gaussian process regression====
* Time-series forecasting using manifold learning, radial basis function interpolation, and geometric harmonics by Ioannis G. Kevrekidis,3 and Constantinos Siettos, 2022 [https://pubs.aip.org/aip/cha/article-pdf/doi/10.1063/5.0094887/16497596/083113_1_online.pdf pdf]

===Physics-Informed Neural Networks===
# Neural partial differential equations with functional convolution [https://openreview.net/pdf?id=D4A-v0kltaX ICLP]
# Solving PDEs by variational physics-informed neural networks: an a posteriori error analysis [https://link.springer.com/article/10.1007/s11565-022-00441-6?fromPaywallRec=true PDF] plus several links to the books on the subject inside
# Predicting the nonlinear dynamics of spatiotemporal PDEs via physics-informed informer networks [https://link.springer.com/article/10.1007/s11071-024-10655-2?fromPaywallRec=true PDF]
# Three ways to solve partial differential equations with neural networks — A review [https://arxiv.org/abs/2102.11802 arxiv 2021]
# NeuPDE: Neural Network Based Ordinary and Partial Differential Equations for Modeling Time-Dependent Data [https://arxiv.org/abs/1908.03190 arxiv 2019]
# Physics-based deep learning [https://www.physicsbaseddeeplearning.org/intro-teaser.html code]
# PINN by Steve Burton [https://www.youtube.com/watch?v=g-S0m2zcKUg&list=PLMrJAkhIeNNQ0BaKuBKY43k4xMo6NSbBa&index=3 yt]
# Process Model Inversion in the Data-Driven Engineering Context for Improved Parameter Sensitivities [https://www.mdpi.com/2227-9717/10/9/1764 mdpi processes 2022] ('''nice connection pictures''')
# Physics-based Deep Learning [https://www.physicsbaseddeeplearning.org/intro.html github]
# Integral Transforms in a Physics-Informed (Quantum) Neural Network setting [https://arxiv.org/pdf/2206.14184 arxiv 2022]

=== Riemmanian models===
# Riemannian Continuous Normalizing Flows [https://arxiv.org/abs/2006.10605 arxiv 2020]
# Residual Riemannian Networks [https://arxiv.org/pdf/2310.10013 arxiv 2023] 
=== Continous time, Neural ODE===
# Neural Spatio-Temporal Point Processes ''by Ricky Chen et al.'' [https://arxiv.org/abs/2011.04583 iclr 2021] (likelihood for time and space)
# Neural Ordinary Differential Equations ''by Ricky Chen et al.'' [https://arxiv.org/abs/1806.07366 arxiv 2018]
# Neural Controlled Differential Equations for Irregular Time Series 'Patrick Kidger et al.'' [https://arxiv.org/abs/2005.08926 arxiv 2020][https://github.com/patrick-kidger/NeuralCDE github]
# Diffusion Normalizing Flow [https://arxiv.org/pdf/2110.07579 arxiv 2021]
# Differentiable Programming for Differential Equations: A Review [https://arxiv.org/abs/2406.09699 arxiv 2024]
# (code tutorial) Deep Implicit Layers - Neural ODEs, Deep Equilibrium Models, and Beyond [https://implicit-layers-tutorial.org/ nips 2020]
# (code tutorial) [https://www.physicsbaseddeeplearning.org/overview-ns-forw.html 2021]
# Neural CDE and tensors [https://ieeexplore.ieee.org/abstract/document/9979806 IEEE], [https://ieeexplore.ieee.org/abstract/document/9533771 IEEE]
# Latent ODEs for Irregularly-Sampled Time Series [https://proceedings.neurips.cc/paper_files/paper/2019/file/42a6845a557bef704ad8ac9cb4461d43-Paper.pdf 2019]

=== Graph and PDEs ===
# Fourier Neural Operator for Parametric Partial Differential Equations [https://arxiv.org/abs/2010.08895 arxiv 2020]
# Masked Attention is All You Need for Graphs [https://arxiv.org/abs/2402.10793 arxiv 2024]
=== Neural SDE===
# Approximation of Stochastic Quasi-Periodic Responses of Limit Cycles in Non-Equilibrium Systems under Periodic Excitations and Weak Fluctuations [https://doi.org/10.3390/e19060280 mdpi entropy 2017] (great illustrations on the stochastic nature of a simple phase trajectory)
# Approximation of Stochastic Quasi-Periodic Responses of Limit Cycles in Non-Equilibrium Systems under Periodic Excitations and Weak Fluctuations [https://doi.org/10.3390/e19060280 mdpi entropy 2017] (great illustrations on the stochastic nature of a simple phase trajectory)
# Neural SDEs for Conditional Time Series Generation [https://arxiv.org/abs/2301.01315 arxiv 2023] code [https://github.com/pere98diaz/Neural-SDEs-for-Conditional-Time-Series-Generation-and-the-Signature-Wasserstein-1-metric github LSTM - CSig-WGAN]
# Neural SDEs as Infinite-Dimensional GANs [https://arxiv.org/pdf/2102.03657 2021]
# Efficient and Accurate Gradients for Neural SDEs ''by Patrick Kidger'' [https://arxiv.org/pdf/2105.13493 arxiv 2021] code [https://docs.kidger.site/diffrax/examples/neural_sde/ diffrax]
=== Chains and homology===
# Operator Learning: Algorithms and Analysis [https://arxiv.org/pdf/2402.15715 arxiv 2024]
# Hi-res weather: Operator learning [https://arxiv.org/pdf/2202.11214 arxiv 2022]
# Homotopy theory for beginners by J.M. Moeller [https://web.math.ku.dk/~moller/e01/algtopI/comments.pdf ku.dk 2015] (is it a pertinent link?)
# Explorations in Homeomorphic Variational Auto-Encoding [https://arxiv.org/abs/1807.04689 arxiv 2018]
# Special Finite Elements for Dipole Modelling ''master thesis Bauer'' [https://www.sci.utah.edu/~wolters/PaperWolters/2012/BauerMaster.pdf 2011]
# Selecting embedding delays: An overview of embedding techniques and a new method using persistent homology [https://arxiv.org/pdf/2302.03447v1 arxiv 2023] (Denis)
# (code) Clifford Algebra for Python https://clifford.readthedocs.io/en/v1.1.0/

===Appendix===
# Neural Memory Networks [https://cs229.stanford.edu/proj2015/367_report.pdf stanford reports 2019]
# An Elementary Introduction to Information Geometry ''by Frank Nielsen'' [An Elementary Introduction to Information Geometry Frank Nielsen [https://doi.org/10.3390/e22101100 mdpi entropy]
# The Many Faces of Information Geometry ''by Frank Nielsen'' [https://www.ams.org/journals/notices/202201/rnoti-p36.pdf ams 2022] (short version)
# Geometric Clifford Algebra Networks [https://arxiv.org/abs/2302.06594 arxiv 3022]
# Clifford Algebras and Dimensionality Reduction for Signal Separation ''by [https://www.math.uni-hamburg.de/home/guillemard/ M. Guillemard]'' [https://www.math.uni-hamburg.de/home/guillemard/papers/clifford7.pdf Uni-Hamburg 2010][https://www.math.uni-hamburg.de/home/guillemard/clifford/ code]
# Special Finite Elements for Dipole Modelling ''by Martin Bauer'' Master Thesis [https://www.sci.utah.edu/~wolters/PaperWolters/2012/BauerMaster.pdf Erlangen 2012] diff p-form must read
# Bayesian model selection for complex dynamic systems [https://www.nature.com/articles/s41467-018-04241-5 2018]
# Visualizing 3-Dimensional Manifolds ''by Dugan J. Hammock'' [https://archive.bridgesmathart.org/2013/bridges2013-551.pdf 2013 umass]
# At the Interface of Algebra and Statistics by ''T-D. Bradley'' [https://arxiv.org/abs/2004.05631 arxiv 2020]
# Time Series Handbook by Borja, 2021 [https://github.com/phdinds-aim/time_series_handbook github]
# Physics-informed machine learning [https://www.nature.com/articles/s42254-021-00314-5 Nature reviews: Physics 2021]
# Integral Transforms in a Physics-Informed (Quantum) Neural Network setting: Applications & Use-Cases [https://arxiv.org/abs/2206.14184 arxiv 2022]
# Deep Efficient Continuous Manifold Learning for Time Series Modeling [https://arxiv.org/abs/2112.03379 arxiv 2021]
====Causality====
# Toward Causal Representation Learning [https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9363924 2021]
# See the Sugihara collection

==Basics==
Collection of wiki-links
===Time Series===
# [https://en.wikipedia.org/wiki/Spectral_submanifold Spectral submanifold] (with nonlinear dimensional reduction like [https://en.wikipedia.org/wiki/Self-organizing_map som])
# [https://en.wikipedia.org/wiki/Lagrangian_coherent_structure Lagrangian coherent structure] (software below)
===Signal Processing===
#[https://en.wikipedia.org/wiki/Estimation_of_signal_parameters_via_rotational_invariance_techniques Estimation of signal parameters via rotational invariance techniques]
#[https://en.wikipedia.org/wiki/Reproducing_kernel_Hilbert_space Reproducing kernel Hilbert space]
#[https://en.wikipedia.org/wiki/Kernel_principal_component_analysis Kernel principal component analysis]
#[https://en.wikipedia.org/wiki/Gram_matrix Gram matrix]
#[https://en.wikipedia.org/wiki/Generalized_pencil-of-function_method Generalized pencil-of-function method]
#[https://en.wikipedia.org/wiki/Wavelet_transform Wavelet transform]

===Differential Geometry===
#[https://en.wikipedia.org/wiki/Pushforward_(differential) Pushforward (differential)]
#[https://en.wikipedia.org/wiki/Pullback_bundle Ffibers, Bundles, Sheaves]
#[https://en.wikipedia.org/wiki/Homology_(mathematics) Homology]
#[https://en.wikipedia.org/wiki/Topological_data_analysis Topological data analysis]
#[https://en.wikipedia.org/wiki/Conditional_mutual_information Conditional mutual information]
#[https://en.wikipedia.org/wiki/Convergent_cross_mapping Convergent cross mapping]
#[https://en.wikipedia.org/wiki/Differential_form Differential form]
#[https://en.wikipedia.org/wiki/Total_derivative The total derivative as a differential form]
#[https://en.wikipedia.org/wiki/Riemannian_manifold #Riemannian_metrics Riemannian_metrics]
# Multidimensional Differential and Integral Calculus: A Practical Approach (textbook)

===Probabilistical Decompisition===
#[https://en.wikipedia.org/wiki/Wasserstein_metric Wasserstein metric]
#[https://en.wikipedia.org/wiki/Mutual_information Mutual information]
#[https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant Jacobian]
#[https://en.wikipedia.org/wiki/Fisher_information Fisher information]
# also dobrushin stratonovich wasserstein
# also fluid dymanics, transportation theory

===Tutorials===
# [https://www.connectedpapers.com/main/d86084808994ac54ef4840ae65295f3c0ec4decd/Physics%20informed-neural-networks%3A-A-deep-learning-framework-for-solving-forward-and-inverse-problems-involving-nonlinear-partial-differential-equations/graph Connected papers search]
# Operator Learning via Physics-Informed DeepONet: Let’s Implement It From Scratch [https://towardsdatascience.com/operator-learning-via-physics-informed-deeponet-lets-implement-it-from-scratch-6659f3179887 Medium]

==Tools==
# [https://github.com/ilkhem/icebeem icebeem]
# [https://github.com/ilkhem/ivae ivae]
# [https://github.com/kondratevakate/fmri-component-# fmri-component]
# analysis/blob/master/VAE_for_fMRI/dataset/train/Bystrova0_y-axis.png
# [https://fr.mathworks.com/help/deeplearning/ug/dynamical-system-modeling-using-neural-ode.html Neural ODE in Matlab]
# [https://github.com/pyRiemann/pyRiemann pyRiemann]
# causality inference [https://peps.python.org/pep-0484/#scoping-rules-for-type-variables peps]

===Turbulence===
# Runko: Modern multiphysics toolbox for plasma simulations [https://github.com/hel-astro-lab/runko GitHub]
# 2d-turb-PINN by Parfenyev [https://github.com/parfenyev/2d-turb-PINN/tree/main GitHub]

====Physics and Engineering of Turbulence====
# Fundamentals of Fluid_Mechanics, 2013 [https://students.aiu.edu/submissions/profiles/resources/onlineBook/L5g8S6_Fundamentals_of_Fluid_Mechanics-_7.pdf PDF]
# Introduction ot Fluid Mechanics, 2004 by R. Fox et al. [https://ahsheikh.github.io/Courses/NumMod/FM_RobertFox.pdf PDF]
# Computational fluid dynamics, 1995 by John D. Anderson, Jr. [https://www.airloads.net/Downloads/Textbooks/Computational-Fluid-Dynamics-the-Basics-With-Applications-Anderson-J-D.pdf PDF]
# Fluid-dynamic drag, 1965 by S.F. Hoerner [https://ia800606.us.archive.org/17/items/FluidDynamicDragHoerner1965/Fluid-dynamic_drag__Hoerner__1965_text.pdf PDF]
# TorchDyn: A Neural Differential Equations Library [https://arxiv.org/abs/2009.09346 arXiv] [https://github.com/DiffEqML/torchdyn github]
## Parameterized Neural Ordinary Differential Equations: Applications to Computational Physics Problems [https://arxiv.org/abs/2010.14685 ArXiv]
## Turbulence forecasting via Neural ODE [https://arxiv.org/abs/1911.05180v1 ArXiv]
## (not the same) Hamiltonian Neural Networks [https://arxiv.org/abs/1906.01563 ArXiv]