Optimization of GEM using OpenMP GPU Offloading

GEM is a particle-in-cell gyrokinetic code for investigation of low-frequency phenomena such as micro-turbulence and energetic particle driven Alfven waves in tokamak plasma. In the GEM code, the particle arrays should be moved from CPU memory to GPU memory with all particle loops performed in GPUs. Moreover, the particle shift consists of two steps: the initialization step and actual data movement, in which the first step should be modified to enable more loops to run on GPU while the actual transferred particles in the second one should be updated between CPU and GPU. In order to minimize the data transfer and offload the calculation processes from CPU to GPU, the porting of GEM code from OpenACC to OpenMP GPU offload is implemented. The details of conversion from OpenACC to OpenMP GPU offloading are illustrated. Furthermore, we make the comparison of acceleration performance between CPU and GPU as well as OpenACC and OpenMP. Additionally, the results about comparison of profiling data between weak scaling (fixed grid size and increased particle number) and strong scaling (increased both grid and particle number in proportion) for different machines (e.g., Summit and Cori) are presented and discussed.