From Turbulence Simulations to Petascale Interactive Numerical Laboratories

Scientists in many disciplines would like to compare the results of their experiments or theoretical hypotheses to data emerging from numerical simulations based on first principles. This requires not only that we can run sophisticated simulations and models, but that at least a selected subset of the results of these simulations are available publicly, through an easy-to-use portal. We have to turn our simulations into open numerical laboratories in which anyone can perform their own experiments.  For a scalable analysis we must have an inherently scalable data access. Flat files violate this principle: the user cannot do anything until a very large file has been physically transferred.

The JHU Turbulence databases provide an immersive environment, where users can insert their virtual sensors into the simulation, sending a data stream back to the user. The sensors can be pinned to Eulerian locations or they can move with the flow. They can feed back data on multiple channels, have a variety of operators, e.g.  Laplacian, or various filters. This model also enables users to run time backwards, impossible in a direct numerical simulation involving dissipation. The snapshots are saved frequently enough that one can smoothly interpolate velocities. This simple interface has provided a very flexible, yet powerful way to do science with large data sets from anywhere in the world – we have served over 12 trillion measurements to the community.

Soon we will have Exascale systems, with memory footprints of several petabytes. As a result, only a small fraction of the complete output can ever be saved for later reuse and much of the analysis will have to be done in-situ.  This will make it increasingly harder for scientists outside the core simulation team to reuse this data. The talk will explore ideas how this challenge can be potentially resolved in a satisfactory fashion.