Statistical physics insights into optimal training strategies

Training machine learning models relies on various optimization strategies to enhance performance. These include optimization algorithms with adaptive hyper-parameters, time-dependent selection of training examples, and model refinement through dynamic architectures. While these strategies aim to accelerate training and steer models toward solutions with good generalization properties, they often rely on trial-and-error heuristics and lack a solid theoretical foundation. Furthermore, machine learning problems are inherently high-dimensional—in terms of dataset size, input dimensions, and model parameters—challenging meta-optimization techniques that can suffer from the curse of dimensionality. Recent advances in the statistical physics of neural networks have provided powerful tools to capture high-dimensional training dynamics through low-dimensional effective equations that track the evolution of key order parameters. In this talk, I will present how to integrate dimensionality-reduction techniques from statistical physics with control-theoretic methods to identify optimal training strategies for prototypical learning problems.