Stability and Breakdown of Collective Milling in Large Fish Schools

Swimming fish often self-organize into striking and dynamic collective patterns. Milling “vortex-like” patterns have been observed empirically and in mathematical models. However, less is known about the stability of these milling patterns, especially with increasing number of swimmers. Here, we conducted simulations of schools composed of up to 50,000 swimmers, using a self-propelled particle model with all-to-all flow interactions and behavioral rules inferred from fish trajectories in shallow-water tanks. We found an abrupt transition with increasing school size, where the milling state became unstable, giving rise to turbulent-like behavior. In contrast to polarized schools—where flow interactions are essential for the loss of global order—here, flow plays no role in destabilizing the milling pattern as school size increases. To analyze this phenomenon, we mapped the individual particle data to continuum density and polarization fields, and analyzed the interplay between density waves and polarization waves in schools exhibiting stable milling at moderate group size and in dynamically changing schools after loss of stability. These findings provide a quantitative bridge between individual behavioral rules and emergent instabilities in large schools and offer new insights into collective order in biological systems.