FAIR and Reproducible High-Throughput Workflows with AiiDA

The ever-growing availability of computational power and sustained development of computational methods have contributed much to recent scientific progress. This progress presents new challenges regarding the sheer amount of calculations and data to be managed. Next-generation exascale computing infrastructures will harden these challenges and require automated and scalable solutions. We have thus developed a comprehensive, robust, open source, high-throughput infrastructure AiiDA (http://aiida.net) dedicated to address the challenges in automated workflow management and data provenance storage. We discuss how AiiDA’s engine can now sustain throughputs of ~100'000 processes/hour, while automatically storing full data provenance graphs. These are stored in a database making data queryable, traversable, and directly enabling high-performance data analytics. Any simulation software can be interfaced to AiiDA via its open plugin repository. We demonstrate how AiiDA's workflow engine provides advanced automation and error handling features, allowing users to write modular workflows, interoperable between many different quantum or classical codes. We highlight how the resulting data can be disseminated on the Materials Cloud (http://materialscloud.org) in fully FAIR-compliant mode.