Learning a biologically plausible linear controller for nonlinear systems

Understanding how an animal's brain learns to execute proper movements is a major problem of interest in neuroscience. A prominent framework to solve this problem is Optimal Feedback Control (OFC). However, solving the OFC problem requires knowledge of the underlying dynamics as well as iterating a matrix Riccati equation. A biologically plausible learning mechanism is expected to 1) be able to perform online learning on the fly, and 2) have local synaptic plasticity rules. This work presents a model-free control approach for nonlinear dynamic systems that satisfies both these requirements. Specifically, the proposed approach employs policy gradient to learn a linear controller for the nonlinear dynamic system in kernel space and using local learning rules, without requiring any knowledge about the underlying dynamic of the system.

As an example, we learn the proper nonlinear control input for pendulum swing-up. After evaluating the optimal controller in the state space using different basis functions including one-hot encoding, radial basis functions (RBF), and Fourier modes, it was observed that one can represent the nonlinear system in terms of fewer Fourier modes compared to RBF and one hot encoding. Moreover, one-hot encoding basis does not provide a good approximation of the control input, due to its inability to represent smooth, continuous functions on the boundaries. From another perspective, RBFs can be mapped onto place cells and therefore, are biologically implementable. Accordingly, since the nonlinear optimal control signal is smooth in the state space and the desired learning rule should be biologically plausible, the kernel space is selected to be the space expanded by RBFs. Then, the parameters of the RBFs (including the centroids and the widths) are updated iteratively using REINFORCE technique.