Beyond Explainability: Towards Interpretable Machine Learning for Physics

The importance of machine learning (ML) in modern physics has been rising, especially in applications to detecting quantum phases or simulating complex physical systems. However, with increased use, some critical questions arise regarding its interpretability and extraction of meaningful physical insights. Moreover, physical data poses a unique challenge since its inherent disorder can act like adversarial noise, misleading explanation methods designed to interpret model behavior. This issue, though significant in physics, is rarely addressed in the broader ML community. This talk will explore the differences between explaining a trained model's behavior (post-hoc explainability) and designing machine learning models with interpretable parts from the ground up. Using two case studies from our research - neural networks applied to the Su-Schrieffer-Heeger (SSH) model and the TetrisCNN model tailored to detecting phase transitions and their order parameters in spin systems - we will show the limitations of post-hoc explainability and the advantages of interpretable architectures. We argue that the most insightful interpretable models for non-tabular data are largely task-dependent and we share our recipe for their design.