The Freight-Train method for production on the NOvA experiment and cosmic filtering efforts at ALCF

A significant challenge for many experiments is how to store, move, and process large amounts of data in an effective manner. The NuMI Off-axis ν_e Appearance (NOvA) experiment is one such experiment. The freight train model aims to alleviate this strain by improving upon the way data files are processed. Data from NOvA is stored on physical tapes at Fermilab, and in order to access the data stored, the tape must be loaded into the network, this is known as prestaging. In the past the basic production principle was to decide on a type of event to get the information of, and then pull all tapes that had these types of events. This led to prestaging the same tape multiple times, since a single tape contains more than one type of event, and since this process is slow it caused a bottleneck in production. The freight train model has changed this by prestaging tapes in sequential order and pulling every file of interest off of the tape and distributing them to corresponding processing pipelines, this effective "parallelization" of the production workflow significantly eases the prestaging bottleneck. Another way that NOvA capitalizes on the efficiency of parallel processing is by sending files to an Argonne Leadership Computing Facilities GPU farm to undergo cosmic filtering. The filtering process is necessary since the far detector of NOvA sits on the surface of the earth, and in order to prepare the data for event reconstruction, the more obvious cosmic signatures are filtered out. This is a computationally expensive task, and the power of GPUs are leveraged in order to increase the throughput of the cosmic filtering process.