Performance Analysis and Improvement of an Unstructured Mesh Gyrokinetic Particle-in-cell Code for Exascale Fusion Plasma Simulations

We report on efforts to improve the performance of a recently developed unstructured mesh gyrokinetic particle-in-cell (PIC) code, XGCm, for modeling fusion plasma (Zhang et. al, APS DPP Meeting 2021, PP11.00053). XGCm executes all steps on GPU accelerators, and exhibits good performance and portability. It builds on the Omega and PUMIPic libraries that perform mesh and particle related operations. This provided us with advantages of supporting distributed meshes and better memory usage and locality, while it does pose performance challenges in operations involving interprocess communications and the interaction between particles and mesh fields. To improve the performance, we analyzed the time costs of major components within the code using realistic test problems. This process identified that the key routines to improve were the particle search operation, which finds the mesh element each particle belongs to, and the particle migration operation, where each particle is put onto the correct mesh element and its information transferred to the correct MPI rank. With the improvements made, XCGm is as performant as XGC for delta-f PIC calculations while also able to support distributed meshes.