Long range memory in deep neural networks' neural activations

The biological brain neurons exhibit various critical phase transition patterns, among them is the long-range memory phenomenon. One common hypothesis is that the healthy brain is operating at some critical point, leading to such long range effect. In the artificial neural networks, in particular deep learning models, it has been recently found that they operate optimally at the critical state between periodic cycle phase and chaotic phase in various benchmark tests. Hence, it remains to be seen that if such critical state leads to the long range memory effect similar to that of the biological brain at a quantitatively level. Here, we investigate several widely adopted deep learning models of different architectures, and look for the evidence of such long range memory effect in the neuron's activations, when the model achieves the highest accuracy on benchmark datasets. In some of the models, we found signatures of long range memory in the high frequency region, while the low frequency region is governed by short memory effects. The robustness of the phenomenon is also investigated across different types of training datasets to test the dependence of the long range memory on the long memory of input data.