How do neural networks learn simple functions?

To illustrate how concepts and methods at the intersection of high-dimensional probability and statistical physics offer fresh insights into neural networks, I will explore how progress can be made by studying simple models and questions. This talk will cover the mechanisms by which two- (and more) layer neural networks learn simple high-dimensional functions from data over time. I will emphasize the complex interplay between algorithms, iterations, and task complexity, demonstrating how gradient descent and stochastic gradient descent can capture specific features of functions and enhance generalization beyond random initialization and kernel methods.