Evolution of Analysis Techniques and Statistical Treatment

Experimental physicists benefit from an ever-increasing volume of data and a broad set of analysis techniques and statistical modeling approaches that have been developed over decades. With demands on analysis scale and complexity changing, so are the software tools that are being developed. I will describe how modern tools, frequently based on task graphs and employing columnar analysis techniques, affect the way in which physicists approach their analyses. I will illustrate the role of machine learning in this context with two examples. Simulation-based inference techniques take advantage of the typical HEP problem structure to allow for powerful statistical inference via machine learning. The possibility to use automatic differentiation to differentiate through an analysis pipeline enables gradient-based optimization while also introducing practical challenges.