Memorizing without overfitting: Over-parameterization in machine learning, physics and biology

Over the last decade, advances in Machine Learning, and in particular Deep Learning, have resulted in incredible progress in the ability to learn statistical relationships from large data sets and make accurate predictions. In contrast to models from classical statistics, Deep Learning models almost always have many more fit parameters than data points, a setting in which classical statistical intuitions such as the bias-variance tradeoff no longer apply. In this presentation, we analyze the generalization properties of two-layer neural networks to showcase some of the new, unaccounted-for behaviors that arise in these "over-parameterized" models that are not present in classic statistics. We also provide additional intuition by proposing a new geometric picture of generalization in over-parameterized models. Finally, we discuss how over-parameterization in Deep Learning Models may reveal a deeper, more general understanding of a wide range of physical systems, including allosteric proteins, physics-based learning machines, and even eco-evolutionary models.