Inferring nature at scale: innovative software tools for big data analysis in HEP

We are now in an era where large general-purpose instruments collect hundreds of petabytes of data, from which a wide variety of hypotheses can be tested through distilling the data down to high-level observables. This analysis task requires dedicated software tooling to handle the unprecedented data volume and also the increasing complexity of hypotheses to be tested. Productivity is constrained as much by human time as processing time, hence we must focus on effective software design, as well as scalability and performance, to maintain physics output in this era of big analysis. I will discuss current solutions and a vision for future analysis systems that address these challenges.