Accelerating Large-Scale GW Calculations on Hybrid GPU-CPU Systems

Large-scale GW calculations are state-of-the-art to accurately describe excited state phenomena in materials, which is critical for the design of novel new devices based on complex materials with applications in many fields. Application of the GW method to complex systems is often perceived as being limited due to high computational cost. Reduced time to solution can be achieved through novel methods, algorithms and optimal implementations on modern HPC systems. In particular accelerators such as GPU’s can speed-up by order of magnitudes conventional CPU-only implementations and additionally reduce the energy per flop consumption. However, porting a large scale HPC code to hybrid GPU-CPU systems and achieve best performance is non trivial and faces several challenges. This talk showcases the various techniques used to accelerate the Material Science code BerkeleyGW on hybrid architectures targeting to accelerate large scale simulations with thousands of atoms. These techniques include the efficient use of accelerated libraries, pinned host memory, asynchronous memory transfer, streams, batched operations, shared memory, and the overlapping of MPI communication and GPU computation. We achieve good strong- and weak-scaling on thousands of GPUs, and a 16x improvement is obtained on FLOPs/Watt efficiency compared to the CPU-only implementation. We show in this way that GW calculations on systems made of thousands of atoms can be performed with excellent time to solutions, of the order of minutes, even running on a moderate number of hybrid nodes.