The strange case of Dr Jekyll and Mr Hyde: The two faces of singular models.

Why does systems biology work in spite of a blizzard of poorly-defined parameters and yet the detection of the Higgs boson requires five-sigma? Both these statistical analyses involve singular models, defined by structural unidentifiability (i.e. the absence of a one-to-one map between parameters and distribution functions). This singular structure leads to profound changes in the phenomenology of inference. In this talk, we will explore the phenomenology of learning from two physical perspectives: First, we explore the correspondence between statistical physics and statistics and demonstrate that there is equivalence between predictive performance and heat capacity, which gives new physical insight into why learning has universal scaling as well as explaining how and why these universal rules fail in the context of singular models. Finally, we explore insights from the Riemannian geometry of the model parameter space to determine what face a singular model will show: anomalously high or low learning performance.