Harnessing rotational symmetry in reinforcement-learning-based control of the flow around a four-roll mill

The four-roll mill is designed to study the morphology of a drop at the centre of an extensional flow formed by four rotating cylinders with the same speed but alternate signs. However, the drop is prone to escaping due to the unstable nature. Thus it is necessary to implement flow control to correct the drop's trajectory. Traditional model-based controllers face challenges when attempting this task, as an accurate flow model is hard to obtain in real-world conditions. To address this issue, we apply a model-free deep-reinforcement-learning-based (DRL) control scheme to adjust the speeds of the four cylinders in real-time, aiming to drive the drop back to the centre when it drifts away. The drop is modeled as a rigid particle transported by the flow and the flow itself is initially modeled by a linear superposition of four rotlets and then extended to the realistic 2-D direct numerical simulations. In both cases, it is found that with a proper choice of state representations and reward functions, the learnt policy is effective in driving the drop back to the centre from any random initial position. One key to the training lies in embedding the rotational symmetry inherent in the geometry into the DRL framework. This way, the control policy learnt in one quadrant can be directly applied to control drops located in the other three quadrants. Compared to the naive training, the symmetry-aware agent has a much smaller state space, resulting in higher sampling efficiency and faster learning speed.