Path planning of swimmers in complex flows: Comparing reinforcement learning vs optimizing a discrete loss (ODIL)

Path planning for swimmers in complex flow fields is fundamental in domains ranging from targeted drug delivery to underwater navigation. Reinforcement Learning (RL) is often used in such problems where a swimmer repeatedly interacts with an environment to find an optimal path control policy. RL treats the environment as a black box which may result in poor sampling efficiency. We propose a method for closed-loop optimal control based on the ODIL (Optimizing a DIscrete Loss) framework where we combine both the dynamics and control objective into the same optimization problem. We compare this method to RL on a variety of path planning problems involving swimmers in fluid flow. Our results suggest that ODIL is more robust and requires 10–100 times fewer policy evaluations during training, especially in a high-dimensional action space. The implementation of the method is straightforward as it takes advantage of standard machine learning tools for automatic differentiation and gradient-based optimization. Overall, we find that ODIL is a fast and easy to adopt computational tool for solving path planning and control problems in fluid mechanics.