Evaluating the limits of the physics learned by a machine learning model

In machine learning for materials science, a model should learn relevant physics rather than arbitrary data representations. This has implications for the robustness, generalizability, and overall trustworthiness of model predictions. In this talk, the loss landscape of the model is used to probe the limits of the learned representation: we compare the geometry of the loss landscape between in-distribution and out-of-distribution prediction tasks. Discontinuities in the loss landscape may indicate where a model fails to generalize, and the perturbation scale of the discontinuities can quantify the robustness of the feature representations in the model weights. This method is demonstrated using graph neural networks trained for several out-of-distribution (OOD) generalization tasks from DFT materials datasets. These models, trained by omitting one or more elements, did not generalize to OOD samples containing the omitted chemistry. The reason why the models do not generalize is unclear, but the model robustness and trustworthiness can still be explored using the loss landscape of the model.