Statistical Mechanics of Infinitely-Wide Convolutional Networks

Understanding how convolutional neural networks (CNNs) can efficiently learn high-dimensional tasks remains a challenge. A popular belief is that these models harness the translational invariant, local, and hierarchical structure of natural data such as images. Yet, we lack a quantitative understanding of how this structure affects performance. To study this problem, we consider wide CNNs in the kernel limit, where generalisation can be characterised using statistical mechanics methods. We introduce a stylised teacher-student framework where a CNN is trained on the output of another CNN with random weights. In this framework, we control the structure of the target function by adding weight sharing and by tuning the size of the neuron receptive fields and the depth of the teacher network. First, we find that translational invariance does not change the scaling of learning curves, that measure the decay of the generalisation error with the number of training examples, and therefore is not enough to beat the curse of dimensionality. Then, we show that if the target function has a local structure, i.e., it depends only on low-dimensional subsets of adjacent input variables, CNNs beat the curse of dimensionality. In fact, the learning curve scaling is controlled by the dimension of these subsets and not by the full input dimension. Finally, we show that the hierarchical structure of CNNs is too rich to be efficiently learnable in high dimensions and discuss further classes of hierarchical target functions.