The electron microscope predates the transistor and the charge-coupled device (CCD). The opportunity to integrate these advancements into the electron microscope was seized and revolutionized in the modern (scanning) transmission electron microscope (STEM). Real-time analysis became possible and is now routine. Consider the efforts one would have to undertake to perform a Fourier transform in the days when images were acquired by exposing photographic plates. Indeed, it was real-time data analysis that enabled the measurement and correction of aberrations in modern instrumentation.^{Reference Pennycook and Nellist1} What are the next opportunities on the horizon for the STEM?

With the impressive progress made in the fields of deep learning, computer vision, and automation, we posit that the next revolution in microscopy will stem from the integration of these tool sets: a self-driving microscope. Such a machine will “understand” what it is looking at and automatically document features of interest. The microscopist will have high-level tools to tell the microscope to “look for distortions at that interface” or “obtain a tomographic reconstruction of this structure.” The microscope will know what various features look like by referencing databases, or it can be shown examples on the fly. Consider the trainable Weka segmentation plugin for ImageJ.^{Reference Arganda-Carreras, Kaynig, Rueden, Eliceiri, Schindelin, Cardona and Seung2,Reference Schneider, Rasband and Eliceiri3} This plugin allows the user to highlight regions of an image to inform the computer which features belong in which categories. As examples are added the computer becomes increasingly accurate at classifying the rest of the image automatically.

Using the Atomic Forge, Oak Ridge National Laboratory (ORNL) researchers brought two, three, and four silicon atoms together to build clusters (green) and make them rotate within a layer of graphene (blue). Photo credit: ORNL.

Advances in deep learning to interpret atomically resolved images are already beginning. For example, Ziatdinov et al. applied deep convolutional neural networks to automate the detection of molecular orientation, defect identification, and classification.^{Reference Ziatdinov, Dyck, Maksov, Hudak, Lupini, Song, Snijders, Vasudevan, Jesse and Kalinin4,Reference Ziatdinov, Dyck, Maksov, Li, Sang, Xiao, Unocic, Vasudevan, Jesse and Kalinin5} Maksov et al. illustrated the untangling of lattice dynamics involving the interaction of thousands of defects using deep learning and unsupervised unmixing strategies,^{Reference Maksov, Dyck, Wang, Xiao, Geohegan, Sumpter, Vasudevan, Jesse, Kalinin and Ziatdinov6} and Vasudevan et al. showed strategies for automated classification of Bravais lattice symmetries.^{Reference Vasudevan, Laanait, Ferragut, Wang, Geohegan, Xiao, Ziatdinov, Jesse, Dyck and Kalinin7} There is no fundamental impediment for implementing tools such as these in the microscope to tell the computer what to pay attention to and where to gather data. As these fields progress and as the techniques are refined and proven, integration into the microscope itself becomes increasingly attractive and powerful.

A parallel development is the recognition that the STEM can be used to tailor materials at the atomic level, termed the Atomic Forge.^{Reference Kalinin, Borisevich and Jesse8} The electron beam can alter materials but historically, steps have been taken to avoid beam alterations, as they typically preclude characterization of a pristine sample by introducing defects. Recently, however, these beam-induced modifications have been leveraged to produce remarkable demonstrations of materials manipulation at the nanometer and atomic scales. Jesse et al. demonstrated the controlled layer-by-layer growth from an amorphous precursor in strontium titanate^{Reference Jesse, He, Lupini, Leonard, Oxley, Ovchinnikov, Unocic, Tselev, Fuentes-Cabrera, Sumpter, Pennycook, Kalinin and Borisevich9} and Si.^{Reference Jesse, Hudak, Zarkadoula, Song, Maksov, Fuentes-Cabrera, Ganesh, Kravchenko, Snijders, Lupini, Borisevich and Kalinin10} Hudak et al. showed how individual Bi atoms can be moved with the electron beam through a crystalline Si lattice to form patterned structures with atomic column precision.^{Reference Hudak, Song, Sims, Troparevsky, Humble, Pantelides, Snijders and Lupini11} Dyck et al. illustrated the introduction of single dopant Si atoms in graphene with near lattice site precision.^{Reference Dyck, Kim, Kalinin and Jesse12} Susi et al. laid the groundwork for predicting and demonstrating controlled Si dopant motion through a graphene lattice.^{Reference Susi, Kotakoski, Kepaptsoglou, Mangler, Lovejoy, Krivanek, Zan, Bangert, Ayala, Meyer and Ramasse13,Reference Susi, Meyer and Kotakoski14} And Dyck et al. demonstrated the assembly of primitive structures with Si dopants in graphene.^{Reference Dyck, Kim, Jimenez-Izal, Alexandrova, Kalinin and Jesse15}

These demonstrations of such exquisite precision harken back to the initial atomic motion demonstrated in a scanning tunneling microscope by Eigler,^{Reference Crommie, Lutz and Eigler16–Reference Eigler and Schweizer19} which ignited the field of nanotechnology. However, the range of beam-driven materials transformations observed in STEM covers the gamut, including particle fragmentation, precipitation, nanotube/nanowire growth, catalytic etching and growth, phase changes, deposition, sculpting, and welding.^{Reference Dyck, Ziatdinov, Lingerfelt, Unocic, Hudak, Lupini, Jesse and Kalinin20} Most of these are not performed with single atom precision, but the examples mentioned above show that this level of precision is certainly possible. What remains is to establish the techniques and tools that will enable such modifications to be performed routinely for the construction of atomically precise extended structures. This possibility is further enhanced by the idea of a self-driving microscope. If a microscope can drive itself, knows what it is looking at, and can respond “intelligently” to this information, perhaps it can also build structures at the atomic scale.

Both the self-driving microscope and the Atomic Forge are in the early stages of development, with the concepts themselves just beginning to crystalize. Many questions and challenges remain, and these ideas could easily be dismissed as unattainable. Aberration correction itself was repeatedly dismissed after multiple failed attempts;^{Reference Pennycook and Nellist1} nevertheless, we now enjoy the fruits of those who persisted in the face of doubt, and have almost achieved Feynman’s vision of a microscope with 100 times higher resolution than his time.^{Reference Pennycook21,Reference Feynman22} Similarly, the self-driving microscope and the Atomic Forge will fulfill another Feynman prophecy: “What would happen if we could arrange the atoms one by one the way we want them?”^{Reference Feynman22}