Driving exascale computational science with AiiDA

The ever-growing availability of computational power and sustained development of advanced computational methods have contributed much to recent scientific progress.
This progress presents new challenges regarding the sheer amount of calculations and data that has to be managed.
Next-generation exascale computing infrastructures will harden these challenges and we believe will require automated and scalable solutions.
Therefore, we have developed a comprehensive, robust, open source, high-throughput infrastructure AiiDA (http://aiida.net) dedicated to address the challenges in automated workflow management and data provenance storage.
We discuss how AiiDA's engine can now sustain throughputs of ~100'000 processes/hour, while automatically storing full data provenance.
The provenance is stored in relational database that makes the data queryable, traversable, and directly enables high-performance data analytics.
Any simulation software can be interfaced to AiiDA via its plugin system, and AiiDA's workflow language provides advanced automation and error handling features, as we demonstrate through one of our high-throughput projects.
We highlight how the resulting data can be disseminated by and to the wider community in a FAIR way on the Materials Cloud (http://materialscloud.org).