Does the Efficiency Compute Frontier Represent New Physics?

The so-called "Efficiency Compute Frontier" (ECF) refers to an apparent hard constraint on the achieveable error reduction as a function of the amount of computational work incured when processing pre-training data for LLMs (large language models). The Artificial Intelligence (AI) community has questioned if this unexpected but ubiquitous constraint represents some kind of fundamental law of nature.

We present a model of LLM neural-network dynamics exhibiting power-law behavior [1,2] that matches the ECF constraint, F(x) = Ax^-B. The prefactor A = 0.00000001 sets the scale of the neural-network connections, viz., on the order of billions, while the exponent B = 0.05 is indicative of subnetwork correlations that are much stronger than Zipf's law [1]. In this way, we are able to answer the original question in the negative.

Our result notwithstanding and noting the 2024 Physics Nobel Prize[3], this burgeoning area of Generative AI (and possibly related areas) would seem to offer fertile ground for interdisciplinary physics [1,4-6].

References

[1] M. Newman, "Power Laws, Pareto Distributions and Zipf’s Law," Contemporary Physics, 46(5), 323–351 (2005)

[2] N. Maroney and N.J. Gunther, "Power Law Analysis of MagCloud Publications," HP Labs Internal Technical Report CW237032 (2011)

[3] The Nobel Prize in Physics, www.nobelprize.org/prizes/physics/2024/summary (2024)

[4] J.J. Hopfield, "Neural Networks and Physical Systems with Emergent Collective Computational Abilities," Proc. Natl Acad. Sci. USA 79, 2554–2558 (1982)

[5] S.E., Fahlman, G.E.Hinton, and T.J. Sejnowski, "Massively Parallel Architectures for Al: Metl, Thistle, and Boltzmann Machines," in Proc. AAAI Conf. Artif. Intell. 3, 109–113 (1983)

[6] N.J. Gunther, "Path Integral Methods for Computer Performance Analysis," Information Processing Letters, 32, 7-13 (1989)