Local Extreme Learning Machines: A Neural Network Based Spectral Element-Like Method

Existing deep neural network-based methods for solving boundary/initial-value problems suffer from several drawbacks (e.g. lack of convergence with a certain convergence rate, limited accuracy, extremely high computational cost) that make them numerically less than satisfactory and computationally uncompetitive. Here we present a neural network-based method that has largely overcome these drawbacks. This method, termed local extreme learning machines (locELM), combines three ideas: extreme learning machines, domain decomposition, and local neural networks. The field solution on each sub-domain is represented by a local feed-forward neural network, and C^k continuity conditions are imposed on the sub-domain boundaries. The hidden-layer coefficients of the local neural networks are pre-set to random values and fixed, and only the weight coefficients in the output layers are trainable parameters. The overall neural network is trained by a linear or nonlinear least squares computation, not by the back-propagation (or gradient descent) type algorithms. The current method exhibits a clear sense of convergence with respect to the degrees of freedom in the neural network. Its numerical errors typically decrease exponentially or nearly exponentially as the number of degrees of freedom (e.g. the number of trainable parameters, number of training data points) increases, which is reminiscent of the traditional spectral or spectral element-type methods. LocELM far outperforms the deep Galerkin method (DGM) and the physics informed neural network (PINN) method in terms of the accuracy and computational cost (network training time). Its computational performance (accuracy/cost) is on par with the traditional finite element method (FEM), and outperforms FEM when the problem size becomes larger. These characteristics will be demonstrated for a number of problems.