Semantic Segmentation for Analysis of Melting of Nanoscale Ice via Fully Convolutional Neural Networks

Phase segmentation from electron microscopy datasets has emerged as a major subclass of computer vision problems for materials characterization. In this work, we show the application of semantic segmentation to image analysis of the melting of ice in the nanoscale regime, via in-situ observation by Transmission Electron Microscopy (TEM). Nano water confined into a carbon film based liquid cell were transferred into the ultra-high vacuum TEM chamber by a cryo-TEM holder, which provided the required control over temperature and phase. A high-speed K2 direct detection camera was used to record the melting process under low electron dose (<0.1 e^-/nm²), at a rate of 400 frames per second. An ensemble of pre-trained U-Nets were trained to segment ice from the image frames. These neural networks were fine-tuned on a dataset consisting of image frames from the melting experiments. During inference, image frames from a given experiment are passed as input to the trained networks to generate time profiles of the area fraction occupied by ice. The demonstrated approach allows rapid feedback, with high segmentation accuracy and a measure to quantify the uncertainty which can be used to isolate low-quality images, and autonomous control of TEM experiments with high speed cameras.