The Zap Energy data pipeline architecture for simulation data and synthetic diagnostics

Synthetic diagnostics generated from full-device modeling are important tools for understanding experimental data as they allow careful examination of how key measurement features relate to important physical processes. However, generating them over a large array of simulations can be cumbersome and technically challenging. An obvious solution to this problem is a data processing pipeline. Unfortunately, many industry-standard tools for data engineering are not well suited for physics data sets, necessitating the creation of in-house pipelines for many large scientific research programs. At Zap Energy we have largely solved this problem for our data ecosystem by learning the necessary modifications for standard tools such as Airflow1, React2, and Mongo3 to enable the automatic processing of thousands of 2D simulations and hundreds of 3D simulations in an efficient and physically meaningful manner. In this poster we examine the pipeline architecture from simulation runs and data generation to data visualization. We will also highlight how certain choices in the engineering design can improve our comparison to experimental data.

1 https://airflow.apache.org/

2 https://react.dev/

3 https://www.mongodb.com/