Soft Matter Physics for Machine Learning: Dynamical loss functions

The neural network architectures, loss functions or optimizing protocols that are used in deep learning often stem from laborious trial-and-error design. This has triggered great interest in improving theoretical understanding of the connection between the structure of the loss function landscape and the performance of the optimizing protocol, or algorithm. So far, most effort has focused on improving the algorithm (e.g. stochastic gradient descent). We take a different approach by exploring new loss functions. In particular, we explore the effect of dynamical loss functions that change during training. Preliminary results show that this new approach can outperform results obtained with static loss functions for particular cases. We use the Hessian and Neural Tangent Kernel spectrums to understand how topographical changes of the loss function landscape can improve learning.