Programming Memories and Computations in Recurrent Neural Networks Without Training

Recurrent neural networks (RNNs) perform unique and powerful computations through the internal dynamics of their hidden states. Such dynamics are crucial for cognitive processes that rely on the formation and control of mental representations such as working memory and spatial navigation. While substantial progress has been made in training RNNs, we still lack a fundamental understanding of the native language of recurrent and distributed representations. Here we provide such an understanding and language to the extent of programming the RNN connections to store and modify internal memories without training. Specifically, we use tools from dynamical systems and control to track the dynamical variables of inputs as they propagate through the RNN. We then use the distributed representation of variables in the neurons to program the RNN connection weights to store memories and precise modifications to these memories without actually simulating or training the RNN. Equipped with this language of distributed representation, we answer fundamental questions about the learnability of memories and storage capacity of RNNs as a function of the number of neurons and initial connectivity architecture, thereby enabling the principled design of RNNs for specific applications.