Towards Reliable and Sample Efficient Deep Reinforcement Learning for Aerodynamics : a Benchmark Application

Deep Reinforcement Learning (DRL) recently led to new control solutions for dynamic systems across various domains but its application to Computational Fluid Dynamics (CFD) remains a challenge. DRL algorithms usually require a large number of samples, whereas the high computing cost associated to CFD limits the amount of data which can be produced. Therefore, it is crucial to establish (i) which algorithms are both sample efficient and reliable (ii) how control laws generalize from simple to more sophisticated environments on the same problem. This study targets specifically continuous actions algorithms with a replay buffer: DDPG, TD3 and SAC. The algorithms are trained and evaluated on an aerodynamic benchmark which consists in controlling the trajectory of a 2D airfoil. Trainings performed with a low-order model show that optimal control can be achieved with all three algorithms. Besides, SAC is found to be effective and reliable whereas DDPG and TD3 are prone to instabilities and half of the trainings do not converge to a proper solution. Finally, solutions obtained with the low-order model and with CFD on the same problem are compared and transfer of control solutions between the two modeling methods is discussed.