Machine learning methods are progressively gaining acceptance in the electron microscopy community for de-noising, semantic segmentation, and dimensionality reduction of data post-acquisition. The introduction of the APIs by major instrument manufacturers now allows the deployment of ML workflows in microscopes, not only for data analytics but also for real-time decision-making and feedback for microscope operation. However, the number of use cases for real-time ML remains remarkably small. Here, we discuss some considerations in designing ML-based active experiments and pose that the likely strategy for the next several years will be human-in-the-loop automated experiments (hAE). In this paradigm, the ML learning agent directly controls beam position and image and spectroscopy acquisition functions, and human operator monitors experiment progression in real- and feature space of the system and tunes the policies of the ML agent to steer the experiment towards specific objectives.
@article{arxiv.2310.05018,
title = {Human-in-the-loop: The future of Machine Learning in Automated Electron Microscopy},
author = {Sergei V. Kalinin and Yongtao Liu and Arpan Biswas and Gerd Duscher and Utkarsh Pratiush and Kevin Roccapriore and Maxim Ziatdinov and Rama Vasudevan},
journal= {arXiv preprint arXiv:2310.05018},
year = {2023}
}