1/f Phenomena in Deep Neural Networks

The occurrence of 1/f noise, a known characteristic within the neural networks of living organisms, is significant for how the brain processes information. Our research indicates that this type of noise isn't exclusive to biological systems; it's also present in artificial deep neural networks, particularly those trained in natural language processing. We focused on Long Short-Term Memory (LSTM) networks, which we trained using the IMDb dataset. Upon analyzing the neuron activity within the LSTM layer using detrended fluctuation analysis (DFA), we observed distinct 1/f noise patterns. These patterns were not present in the input data fed into the network. Interestingly, when we increased the network's capacity beyond what was necessary for the task (overcapacity), the noise pattern shifted from 1/f to something closer to white noise. This shift occurs because many neurons become underutilized, exhibiting minimal activity when processing inputs. Additionally, we studied the exponents associated with 1/f noise in both the internal workings and external outputs of LSTM cells, noting similarities with the exponent variations observed in fMRI studies of human brains. This study strengthens the theory that 1/f noise might be indicative of a network's optimal learning conditions.