Understanding the punctuated dynamics of scientific and technological frontiers

Despite its widespread importance for human society, our quantitative understanding of the mechanisms governing scientific and technological frontiers remains limited. Here we collect novel data tracking 5.9M solutions of 5.8K problems across 7 diverse areas in science and technology. We find that, in contrast with predictions of canonical models, the emergence of knowledge frontiers follow a bursty pattern, highlighting an intriguing co-existence between rapid progression (unexpectedly rapid growth of frontiers) and long stasis (power-law waiting time distribution between frontiers). Building on rich literature on innovation, record statistics and cultural evolution, we build a simple yet general mathematical framework integrating incremental and radical innovations, where new solutions can be created through either minor refinements on current frontiers or novel explorations in a random manner. The model not only captures the growth pattern and punctuated nature of frontiers progress, but also offers further rich predictions that are supported across all datasets. Together, these results not only offer a new framework to understand frontiers dynamics in science and technology but also have important implications for a wide range of complex social systems.