Deep Reinforcement Learning Using Data-Driven Reduced-Order Models Discovers and Stabilizes Low Dissipation Equilibria

Deep reinforcement learning (RL) is a data-driven method capable of discovering complex control strategies for high-dimensional systems, making it promising for flow control. However, a major challenge of RL is that substantial training data must be generated by interacting with the target system, making it costly when the flow system is computationally or experimentally expensive. We mitigate this challenge in a data-driven manner by combining dimension reduction via an autoencoder with a neural ODE framework to obtain a low-dimensional dynamical model from just a limited data set. We substitute this ROM in place of the true system during RL training to efficiently estimate the optimal policy, which is then deployed to the true system. We apply our method to the Kuramoto-Sivashinsky equation (KSE), a proxy system for turbulence that displays spatiotemporal chaos, equipped with equidistant actuators and demonstrate that we can learn a sufficient ROM of the actuated dynamics. With this ROM and a goal of minimizing dissipation and power cost, we extract control policies from it using RL. We show that the ROM-based strategies translate well to the KSE and highlight that the RL agent discovers and stabilizes a forced equilibrium solution.