Toward exascale whole-device modeling of fusion devices: Porting the GENE gyrokinetic microturbulence code to GPU

The GENE code solves the five-dimensional gyrokinetic equations to simulate the development and evolution of plasma microturbulence in magnetic fusion devices. In order to use the emerging computational capabilities to gain new physics insights through whole device modeling, several new numerical and computational developments are required. Here, we focus on the fact that it is crucial to efficiently utilize GPUs that provide the vast majority of the computational power on such systems. We introduce a novel library called gtensor that was developed along the way to support the GENE GPU port in a performance portable and maintainable fashion. Performance results are presented for the ported code, which on a single node of the Summit supercomputer achieves a speed-up of almost 15x compared to running on CPU only. Typical GPU kernels used in GENE are memory-bound, achieving about 90% of peak. Our analysis shows that there may still be room for improvement if we can refactor/fuse kernels to achieve higher arithmetic intensity. We also performed a weak parallel scalability study, which shows that the code runs well on a massively parallel system, but communication costs start becoming a significant bottleneck when using 1000s of GPUs.