Tackling the Spectral Bias of Neural Networks for Multiscale Flows

Deep learning is increasingly being applied to weather and climate systems that exhibit multiscale features. Despite being deemed as universal function approximators, neural networks, in practice, struggle to capture multiscale data even with large network sizes and extended training iterations. To address this issue, we developed multi-stage neural networks that divide the training process into different stages, with each stage utilizing a new network optimized to fit the residue from the previous stage. We demonstrate that the prediction error from multi-stage training can nearly reach the machine precision of double-floating points within a finite number of iterations. This advancement mitigates the longstanding accuracy limitations of neural network training and can be used to address the spectral bias in multiscale problems.