Integrating pose estimation with tag-based tracking to capture dense social networks

The drive to study the behavior of animals living in groups has motivated a vast expansion of behavioral tracking technology. Much of this technology relies on pose estimation tools, such as SLEAP [1], which accurately track the location of individual body parts of the animal. Moreover, they enable the analysis of complex behaviors which cannot be studied by looking at centroids alone, such as communication via specialized sensory systems or aggressive behaviors like mounting, biting, or stinging. While SLEAP and similar frameworks are able to track and maintain the identity of small numbers of animals, they perform poorly when the number of animals becomes very large, animals are allowed to exit and enter the frame, or when the identity of each animal must be maintained across experiments. Past solutions for monitoring many animals use externally applied tags, such as bar codes or paint dots, to uniquely identify individuals but these techniques do not record pose. We present NAPS (NAPS is ArUco Plus SLEAP), a hybrid behavioral tracking framework that combines state-of-the-art, deep learning-based methods for pose tracking (SLEAP) with unique markers for identity persistence (ArUco) and show that we are able to capture pose and identity within dense social networks in complex environments. We analyze the dynamics of colonies of approximately 50 bumblebees and show that NAPS outperforms SLEAP or ArUco alone.

1. Pereira, T.D., et al., 2022. SLEAP: A deep learning system for multi-animal pose tracking. Nature.