Automating microscopy with machine learning: from object identification to hypothesis learning

Machine learning and artificial intelligence are becoming increasingly important components in physics research, with applications ranging from high-energy physics to materials sciences. Recently, there has been a growing interest in using AI models that interact with physical systems, rather than being pre-trained on large datasets, for tasks such as materials discovery, chemical synthesis, and physical property measurements. Microscopy is particularly well-suited for these active learning tasks as it combines aspects of materials discovery, physics learning, and synthesis through atomic fabrication. In this presentation, I will discuss advances in using machine learning for automated experiments in electron and scanning probe microscopies, including on-the-fly object detection, atomic defects engineering in quantum materials, and physics discovery through active learning. I will also address challenges such as out-of-distribution drift in traditional deep learning methods and the limitations of simple Gaussian processes-based approaches for active learning in complex systems. I will propose solutions such as ensemble learning and iterative training (ELIT), deep kernel learning, and structured Gaussian processes allowing for exploring complex systems and discovering structure-property relationships in an autonomous fashion. Finally, I will discuss the edge computing infrastructure needed for turning modern-day microscopes into autonomous platforms for scientific discovery.