3-D pose estimation of larval zebrafish using an artificial neural network and a physical model

Quantitative ethology requires an accurate estimation of an organism’s postural dynamics in three dimensions plus time. Technical advances over the last few decades have made animal posture estimation in challenging scenarios possible with unprecedented detail. Here, we present (i) a physical model-based method to generate realistic, annotated larval images in novel behavioral contexts, useful to train machine-learning algorithms, among other applications; (ii) an automated method to record and track the posture of individual larval zebrafish in a 3-D environment, applicable when accurate human labeling is too time-consuming; and (iii) a rich annotated dataset of 3-D larval poses for ethologists and the general zebrafish and machine learning community. Using three cameras calibrated with refraction correction, we record 3-D larval swims under free swimming conditions and in response to acoustic and optical stimuli. We then employ a convolutional neural network to estimate 3-D larval postures from these swims. The network was trained by a set of synthetic larval images rendered using a 3-D physical model of larvae. The physical model samples from a distribution of realistic larval postures estimated a priori by classical pattern recognition of larval swimming recordings, which, by itself would not be as accurate as the final neural network. Our final neural network model, trained without any human annotation, performs with a higher accuracy and speed than the classical approach, capturing detailed kinematics of 3-D larval swims.