Week 4

The goal is to get the simplest possible solution to your problem: it is models and its parameters. So make the model fit data with the minimum of your efforts.

X: Experiment planning

Plan your computational experiment.

1. Discuss the experiment goal with your adviser
   • and put this goal in the section Computational experiment
2. Describe your basic data set, a synthetic, or a simple real one:
   • put in the text the title, source, and set up of measurements (it is the technical description, the theoretical one is in the problem statement section),
   • write down the number of objects, and features, describe general statistics,
   • for a synthetic data set describe the generation model, and its parameters (for example, uniform random independent sampling at some given interval).
3. Describe the configuration of the algorithm run.
4. Plan the whole experimental part.
5. List expected tables and figures:
   • make short and long list, for each
   • describe axes,
   • make a draft with a pencil.

R: Preliminary report

1. Make sure that the obtained results do not logically contradict the goals of the computational experiment.
2. Illustrate the obtained results with the preliminary plot. Optimally this plot is hand-made. Just draw it with a pencil on a piece of paper. See for an example. For the final version use this format.
3. Write a mini-report on the results with
   1. a short description of the figure: what the reader could see, what are the consequences,
   2. the results in numbers and comments on it,
   3. put the report to the section computational experiment.

B: Run basic code

Select the basic algorithm and run it using a simple data set.

1. Run your basic algorithm: select the simplest algorithm to get the fastest draft solution of the problem you set.
2. Collect a synthetic data set or download a simple real-world data set of small size.
3. Upload your data to the repository. If the data size exceeds 5MB or the data set consists of numerous files, please discuss with your adviser and team how to keep and share these data.
4. Do not use custom or client's data. Use only open-access data that are easy to download and use.
5. Run the basic algorithm on the synthetic data set, and estimate the error.
6. Describe the basic algorithm, analyze its features, and list competitive models. Here the examples of the description style.
   1. Description refers to the name of some black box model. It is advisable to indicate the source, where the contents of the black box model are described in detail. The description specifies the structural parameters of the black box.
   2. Description defines a model as a map from the design space of features to the space of target variables. Since the model has its parameters the description may refer to the algorithm for optimizing the model parameters in the form of a black box.
   3. Description of the model and algorithm for optimizing its parameters in the form of pseudocode.

Resources to read

See examples inside the reports.

1. Системы и средства глубокого обучения, Бахтеев О.Ю.
2. Повышение качества классификации, Мотренко А.П.
3. Снижение размерности в задаче декодирования, Исаченко Р.В.

Mimic the goals of computational experiments.

1. А. Грабовой,
2. В. Алексеев,
3. А. Рогозина,
4. И. Игашов,
5. Н. Уваров

An example of the measurement description is Old Faithful by Bishop C.P. Pattern recognition and machine learning, 2006. Pp. 677-683.

Dataset sources

1. Top 8 Sources For Machine Learning Datasets
2. List of data-sets for Machine Learning projects
3. Google dataset search
4. Datasets released by Google Research

Homework

1. Write the goal of your computational experiment. A couple of sentences help you focus your efforts.
2. Find the code that works to run the preliminary experiment.
3. Generate or find the simplest dataset. Avoid struggling with data.
4. Run the code on the simplest dataset.
5. Write a draft of your desired report and draw a plot for the error analysis.
6. Collect the letters:
   1. X put the goal of your experiment into the section Computational Experiment.
   2. R put the plan of the hypothesis testing and illustration after.
   3. B upload the notebook or code with a basic experiment into your repository.

Prepare your project to Check 1, which starts this week and goes the next 2-3 weeks.

References

1. Research Methodology: Methods and Techniques by C.R. Kothari, 2004 pdf or pdf
2. Experiment planning: Questionable practices in machine learning by Gavin Leech et al., 2024

Site and docs generators

1. MkDocs is an easy one
2. Sphinx is more complex
3. An example is RelaxIt