Deep reinforcement learning for fish fin ray control

For ray-finned fishes, the ray-fin structure is a highly sophisticated control system enabling versatile locomotion in complex fluid environments. Although the kinematics and hydrodynamics of fish fin locomotion have been extensively studied, the complex control strategy is still poorly understood. In this work, we develop a deep reinforcement learning (DeepRL) solution coupled with multi-fidelity fluid-structure interaction (FSI) models to decipher the control strategy and understand the underlying mechanism of ray-fin locomotion. In particular, we will leverage state-of-the-art off-policy RL structures, including Twin Delayed Deep Deterministic Policy Gradient (TD3) and Soft Actor Critic (SAC), to learn the complex ray-fin control strategies for different swimming needs. To accelerate the training process, the DeepRL agent interacts with virtual environments built upon the FSI models of different fidelities, where a transfer learning strategy is adopted for efficient learning. We also combine both the model-based DeepRL and model-free fine-tuning methods to improve the sample efficiency and learning performance.