Offline reinforcement learning for closed-loop control of the VENUS ion source

While deep reinforcement learning (RL) for control offers the promise of optimal steering and improved resource utilization for electron cyclotron resonance (ECR) ion sources, there are risks associated with opening a critical user facility equipment with complex phase space to the free exploration of online RL. At the same time, accurate modeling of an ECR ion source remained challenging. We discuss the development of an automated, closed-loop control system using RL for VENUS (Versatile ECR ion source for NUclear Science), the third-generation superconducting electron cyclotron resonance ion source for the 88-Inch Cyclotron at the Lawrence Berkeley National Laboratory. Risks are minimized by utilizing offline RL on historical recordings of human operators interacting with the ion source without involving any simulation or surrogate model. We discuss our first result using behavior cloning (BC) and conservative Q-learning (CQL). We further provide lessons and an outlook on offline RL for secure, AI-based closed-loop control at DOE user facilities.