Research Statement - Revision history

Vs at 18:59, 11 February 2024

2024-02-11T18:59:52Z

Wiki: /* Multimodeling and knowledge transfer */

2023-10-23T13:38:40Z

Multimodeling and knowledge transfer

Wiki: /* Spatio-time series and manifold learning */

2022-12-11T17:19:10Z

Spatio-time series and manifold learning

Wiki: /* References */

2022-11-06T01:41:11Z

References

Wiki: /* References */

2022-11-06T01:39:12Z

References

Wiki: /* Spatio-time series and manifold learning */

2022-11-06T01:38:42Z

Spatio-time series and manifold learning

Wiki: /* Multimodeling and knowledge transfer */

2022-11-06T01:35:14Z

Multimodeling and knowledge transfer

Wiki: /* Multimodeling and knowledge transfer */

2022-11-06T01:34:55Z

Multimodeling and knowledge transfer

Wiki: /* Bayesian model selection */

2022-11-06T01:34:02Z

Bayesian model selection

Wiki: /* Tensor decomposition for model selection */

2022-11-06T01:30:08Z

Tensor decomposition for model selection

← Older revision		Revision as of 18:59, 11 February 2024
Line 1:		Line 1:
		+	{{#seo:
		+	\|title=Model selection in Machine Learning
		+	\|titlemode=append
		+	\|keywords=Machine Learning
		+	\|description=My research focuses on the problems of model selection in Machine Learning. It explores methods of Applied Mathematics and Computer Science.
		+	}}
	''[[Vadim]]'', 2023		''[[Vadim]]'', 2023

← Older revision		Revision as of 13:38, 23 October 2023
Line 18:		Line 18:

	==Multimodeling and knowledge transfer==		==Multimodeling and knowledge transfer==
−
−	To reduce the complexity of deep learning models, we transfer information about the structure, parameters, and distribution from the teacher to the student model. This method is called distillation or privileged learning.
−	We assume the student has fewer parameters than the teacher. Our method of Bayesian model selection treats the prior distribution of the student parameters as the posterior distribution of the teacher parameters. We align the model structures by omitting non-informative parts of the teacher model for specific data.

	To reduce the complexity of deep learning models, we transfer information about the structure, parameters, and distribution from the teacher to the student model [14]. This method is called distillation or privileged learning. We assume the student has fewer parameters than the teacher [15]. Our method of Bayesian model selection treats the prior distribution of the student parameters as the posterior distribution of the teacher parameters [16]. We align the model structures by omitting non-informative parts of the teacher model for specific data.		To reduce the complexity of deep learning models, we transfer information about the structure, parameters, and distribution from the teacher to the student model [14]. This method is called distillation or privileged learning. We assume the student has fewer parameters than the teacher [15]. Our method of Bayesian model selection treats the prior distribution of the student parameters as the posterior distribution of the teacher parameters [16]. We align the model structures by omitting non-informative parts of the teacher model for specific data.

@@ Line 25: / Line 25: @@
 ==Spatio-time series and manifold learning==
-he multiple and heterogeneous data sources require sophisticated forecasting models. These models come from the canonical correlation analysis&nbsp;[17]. To obtain a forecast, we make several models. First, we reconstruct embedding manifolds of all sources to reduce the dimensionality of all data spaces. Second, to align these sources in the latent space. Third, to construct the forecasting model as a superposition of particular models for each source&nbsp;[9]. We use convergent cross-mapping to guarantee that all these sources refer to the same dynamic system we model &nbsp;[18]. It brings us the Taken's theorem on delay embedding.
+The multiple and heterogeneous data sources require sophisticated forecasting models. These models come from the canonical correlation analysis&nbsp;[17]. To obtain a forecast, we make several models. First, we reconstruct embedding manifolds of all sources to reduce the dimensionality of all data spaces. Second, to align these sources in the latent space. Third, to construct the forecasting model as a superposition of particular models for each source&nbsp;[9]. We use convergent cross-mapping to guarantee that all these sources refer to the same dynamic system we model &nbsp;[18]. It brings us the Taken's theorem on delay embedding.
 To reconstruct embedding manifolds, we use discrete and continuous time and space representation. In the discrete case, we use tensor decomposition&nbsp;[8]. Namely, tensor-train decomposition brings decent reconstruction quality and easily extends toward deep neural networks. For the continuous case, we use controlled differential equations. They reflect the continuous topology of neural network structure involving automatic differentiation methods. The future combination of these two cases significantly impacts the modeling of heterogeneous signals.
@@ Line 35: / Line 35: @@
 <!-- ==Future directions of research== -->
 <!-- ==Physis-informed machine learning== -->
 ==References==
 [1] Robert Tibshirani. Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society, 58(1):267–288, 1996.<br>

@@ Line 36: / Line 36: @@
 <!-- ==Physis-informed machine learning== -->
 ==References==
-[1] Robert Tibshirani. Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society, 58(1):267–288, 1996.<p>
+[1] Robert Tibshirani. Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society, 58(1):267–288, 1996.<br>
-[2] Bradley Efron et al. Least angle regression. The Annals of Statistics, 32(2), 2004.
+[2] Bradley Efron et al. Least angle regression. The Annals of Statistics, 32(2), 2004.<br>
-[3] Alexandr Katrutsa and Vadim Strijov. Comprehensive study of feature selection methods to solve multicollinearity problem according to evalu- ation criteria. Expert Systems with Applications, 76:1–11, 2017.
+[3] Alexandr Katrutsa and Vadim Strijov. Comprehensive study of feature selection methods to solve multicollinearity problem according to evaluation criteria. Expert Systems with Applications, 76:1–11, 2017.<br>
-[4] Alexandr Katrutsa and Vadim Strijov. Stress test procedure for feature selection algorithms. Chemometrics and Intelligent Laboratory Systems, 142:172–183, 2015.
+[4] Alexandr Katrutsa and Vadim Strijov. Stress test procedure for feature selection algorithms. Chemometrics and Intelligent Laboratory Systems, 142:172–183, 2015.<br>
-[5] Roman Isachenko and Vadim Strijov. Quadratic programming optimiza- tion with feature selection for non-linear models. Lobachevskii Journal of Mathematics, 39(9):1179–1187, 2018.
+[5] Roman Isachenko and Vadim Strijov. Quadratic programming optimization with feature selection for non-linear models. Lobachevskii Journal of Mathematics, 39(9):1179–1187, 2018.<br>
-[6] Tamara Kolda and Brett Bader. Tensor decompositions and applications. SIAM Review, 51(3):455–500, 2009.
+[6] Tamara Kolda and Brett Bader. Tensor decompositions and applications. SIAM Review, 51(3):455–500, 2009.<br>
-[7] Julia Berezutskaya et al. Open multimodal iEEG-fMRI dataset from naturalistic stimulation with a short audiovisual film. Scientific Data, 9(1), 2022.
+[7] Julia Berezutskaya et al. Open multimodal iEEG-fMRI dataset from naturalistic stimulation with a short audiovisual film. Scientific Data, 9(1), 2022.<br>
-[8] Anastasia Motrenko and Vadim Strijov. Multi-way feature selection for ecog-based brain-computer interface. Expert Systems with Applications, 114(30):402–413, 2018.
+[8] Anastasia Motrenko and Vadim Strijov. Multi-way feature selection for ecog-based brain-computer interface. Expert Systems with Applications, 114(30):402–413, 2018.<br>
-[9] Roman Isachenko and Vadim Strijov. Quadratic programming feature selection for multicorrelated signal decoding with partial least squares. Expert Systems with Applications, 207:117967, 2022.
+[9] Roman Isachenko and Vadim Strijov. Quadratic programming feature selection for multi-correlated signal decoding with partial least squares. Expert Systems with Applications, 207:117967, 2022.<br>
-[10] Christoph Mark et al. Bayesian model selection for complex dynamic systems. Nature Communications, 9(1), 2018.
+[10] Christoph Mark et al. Bayesian model selection for complex dynamic systems. Nature Communications, 9(1), 2018.<br>
-[11] Oleg Bakhteev and Vadim Strijov. Deep learning model selection of suboptimal complexity. Automation Remote Control, 79(8):1474–1488, 2018.
+[11] Oleg Bakhteev and Vadim Strijov. Deep learning model selection of suboptimal complexity. Automation Remote Control, 79(8):1474–1488, 2018.<br>
-[12] Oleg Bakhteev and Vadim Strijov. Comprehensive analysis of gradient- based hyperparameter optimization algorithmss. Annals of Operations Research, 1–15, 2020.
+[12] Oleg Bakhteev and Vadim Strijov. Comprehensive analysis of gradient-based hyperparameter optimization algorithms. Annals of Operations Research, 1–15, 2020.<br>
-[13] Kuznetsov Tokmakova and Vadim Strijov. Analytic and stochastic meth- ods of structure parameter estimation. Informatica, 27(3):607–624, 2016.
+[13] Kuznetsov Tokmakova and Vadim Strijov. Analytic and stochastic methods of structure parameter estimation. Informatica, 27(3):607–624, 2016.<br>
-[14] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, ArXiv, 2015.
+[14] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, ArXiv, 2015.<br>
-[15] Andiy Grabovoy and Vadim Strijov. Probabilistic interpretation of the distillation problem. Automation Remote Control, 83(1):123–137, 2022.
+[15] Andiy Grabovoy and Vadim Strijov. Probabilistic interpretation of the distillation problem. Automation Remote Control, 83(1):123–137, 2022.<br>
-[16] Andiy Grabovoy and Vadim Strijov. Prior distribution selection for a mixture of experts. Computational Mathematics and Mathematical Physics, 61(7):1149–1161, 2021.
+[16] Andiy Grabovoy and Vadim Strijov. Prior distribution selection for a mixture of experts. Computational Mathematics and Mathematical Physics, 61(7):1149–1161, 2021.<br>
-[17] Wolfgang H ̈ardle and L ́eopold Simar. Canonical correlation analysis. Applied Multivariate Statistical Analysis, 2007.
+[17] Wolfgang Hardle and Leopold Simar. Canonical correlation analysis. Applied Multivariate Statistical Analysis, 2007.<br>
 [18] George Sugihara et al. Fogarty, and Stephan Munch. Detecting causality in complex ecosystems. Science, 338(6106):496–500, 2012.

@@ Line 36: / Line 36: @@
 <!-- ==Physis-informed machine learning== -->
 ==References==
-[1] Robert Tibshirani. Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society, 58(1):267–288, 1996.<br>
+[1] Robert Tibshirani. Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society, 58(1):267–288, 1996.<p>
 [2] Bradley Efron et al. Least angle regression. The Annals of Statistics, 32(2), 2004.
 [3] Alexandr Katrutsa and Vadim Strijov. Comprehensive study of feature selection methods to solve multicollinearity problem according to evalu- ation criteria. Expert Systems with Applications, 76:1–11, 2017.

@@ Line 22: / Line 22: @@
 We assume the student has fewer parameters than the teacher. Our method of  Bayesian model selection treats the prior distribution of the student parameters as the posterior distribution of the teacher parameters. We align the model structures by omitting non-informative parts of the teacher model for specific data.
-To reduce the complexity of deep learning models, we transfer information about the structure, parameters, and distribution from the teacher to the student model&nbsp;[14]. This method is called distillation or privileged learning. We assume the student has fewer parameters than the teacher&nbsp;[15]. Our method of Bayesian model selection treats the prior distribution of the student pa-rameters as the posterior distribution of the teacher parameters&nbsp;[16]. We align the model structures by omitting non-informative parts of the teacher model for specific data.
+To reduce the complexity of deep learning models, we transfer information about the structure, parameters, and distribution from the teacher to the student model&nbsp;[14]. This method is called distillation or privileged learning. We assume the student has fewer parameters than the teacher&nbsp;[15]. Our method of Bayesian model selection treats the prior distribution of the student parameters as the posterior distribution of the teacher parameters&nbsp;[16]. We align the model structures by omitting non-informative parts of the teacher model for specific data.
 ==Spatio-time series and manifold learning==

@@ Line 12: / Line 12: @@
 ==Bayesian model selection==
-Bayesian model selection relies on the analysis of the model parameters. To select models, we shall use several types of parameters, including structure parameters and hyperparameters. The first one defines the structure of the model, the computational graph. The second one represents the distribution of model parameters and infers the criterion to select the optimal model. The proper hyperparameter values prevent the model from overfitting. Optimization of neural networks with large amounts of hyperparameters is computationally expensive. We proposed modifications of various gradient-based methods based on evidence lower bound.
+==Hyperparameter optimization==
-<!-- \cite{Bakhteev2017HypergradEn4} -->
+A large part of my research is devoted to the methods of parameter covariance matrix estimations.These estimations form a criterion to select a particular parameter of the model or subset of parameters like a neuron or layer of a neural network. The hyperparameters here are the expectation and covariance matrix. For some types of models, they are estimated directly, but the following three ways dive decent results. The likelihood function optimization. Let us treat the model evidence or the evidence lower bound as an example. The bootstrapping methods tackle empirical estimations for some given models&nbsp;[13]. As the most productive way to get the hyperparameters, I treat stochastic processes, starting from Metropolis-Hastings Bayesian sampling.
 ==Multimodeling and knowledge transfer==

@@ Line 9: / Line 9: @@
 ==Tensor decomposition for model selection==
-We expand this approach for tensor decomposition [6] to reduce the dimen- sionality of brain signals [7]. We use a multi-way structure to predict hand trajectories from the spatial time series of cortical activity [8]. It resides in the spatial-spectra-temporal domains. Since these data highly correlate in the domains, redundancy of the feature space and its dimensionality be- come a major obstacle for a robust solution of the regression problem both in multi-way and flat cases. The proposed method extends the quadratic programming feature selection approach [9]. We do not have to optimize the model’s parameters to get an optimal model structure.
+We expand this approach for tensor decomposition&nbsp;[6] to reduce the dimen- sionality of brain signals&nbsp;[7]. We use a multi-way structure to predict hand trajectories from the spatial time series of cortical activity&nbsp;[8]. It resides in the spatial-spectra-temporal domains. Since these data highly correlate in the domains, redundancy of the feature space and its dimensionality become a major obstacle for a robust solution of the regression problem both in multi-way and flat cases. The proposed method extends the quadratic programming feature selection approach&nbsp;[9]. We do not have to optimize the model’s parameters to get an optimal model structure.
 ==Bayesian model selection==