On Criticality of the Internal States in a Learning Machine

To understand how an artificial neural networks evolve in response to the training process that gives the networks desired functionalities, the method of statistical modeling has been applied to the state distribution of their internal neurons allowing the characterization the networks with the thermodynamic properties of the mapped models. It has been shown in an ensemble of machines trained for pattern recognition, the mapped models generally tend towards a critical state over the training process. With expanded conditions of the statistical modeling using various binarization thresholds and optimization methods, we find notable influence on the proximity to the critical states in contrast with the robustness for similar modeling in biological neurons of mouse brains. In our study of a large ensemble of simple four layer machines, the impact of binarization threshold is more significant for well-trained machines compared to lightly trained ones as functional performance is more affected by the binarization. Conversely, optimization methods are more relevant before the networks are sufficiently trained, likely due to the random initialization of the network leading to a more rugged likelihood landscape for model fitting. For a much deeper network such as ResNet-50, we find robustness of criticality similar to that of animal brains.