IMEXLBM: A portable lattice Boltzmann solver based on the Kokkos library

An open-source, performance-portable, lattice Boltzmann-based solver, IMEXLBM, is developed for heterogeneous high performance computing (HPC) platforms. Written in C++, this solver is enabled with the Kokkos and message-passing interface (MPI) libraries whereby, we achieve strong scalability on multi-node, GPU-based HPC systems. The implementation and performance metrics for this solver will be assessed across GPU architectures, such as NVIDIA, Intel, and AMD GPUs. Different memory modes (e.g. unified shared memory, device memory) are explored to understand the trade-off between programmability and performance. Our solver is validated by investigating fundamental CFD benchmark problems. A comparison of GPU vs. CPU results are consistent. The 3D Taylor-Green vortex problem is proposed to study the scaling work of this program. The performance of two local functions without MPI data transfer is quite close to ideal scaling. For the non-local function, an even- located MPI partition method is suggested to obtain better performance and scaling. We also present large-scale results for single- and multi-phase applications on the ThetaGPU and Polaris supercomputers at the Argonne Leadership Computing Facility (ALCF).