SineKAN: A Flexible Machine Learning Model for Modelling $L^2$ Functions and Beyond

The multi-layer perceptron (MLP) is near-ubiquitous in modern neural network architectures. Recently, an alternative to the MLP, Kolmogorov-Arnold Networks (KANs), was proposed, motivated by the Kolmogorov-Arnold representation theorem. The representation theorem states that over a finite domain it is possible to model any continuous multi-variate function as a superposition of many continuous functions of a single variable. However, the theorem doesn't predict what the ideal functional form should be for such a representation leaving much room for investigation. Here, we present the SineKAN model which uses a functional form of sinusoidal functions with learnable amplitudes, frequencies, and biases. We find that these models can successfully model $L^2$ functions (which includes normalized wave functions), and functions bounded over a finite domain. We further find that these functions can achieve competitive results on several tasks when compared with MLP and basis-spline KAN models at competitive sizes and speeds and can be integrated into existing transformer architectures to improve performance.