Statistical Mechanics of Kernel Regression and Wide Neural Networks

A theoretical understanding of generalization remains an open problem for many machine learning models, including deep networks where overparameterization leads to better performance. Here, we study this problem for kernel regression, which, besides being a popular machine learning method, also describes infinitely overparameterized neural networks. We develop an analytical theory of generalization in kernel regression using replica theory of statistical mechanics. This theory is applicable to any kernel and data distribution. Experiments with practical kernels including those arising from wide neural networks show perfect agreement with our theory. Further, our theory accurately predicts generalization performance of neural networks with modest widths. We provide an in-depth analysis of our analytical expression for kernel generalization. We show that kernel machines employ an inductive bias towards simple functions, preventing them to overfit the data. We characterize whether a kernel is compatible with a learning task in terms of sample efficiency. We identify a first order phase transition in our theory where more data may impair generalization when the task is noisy or not expressible by the kernel. Finally, we extend these results to out-of-distribution generalization.