2.1. Overall design of TomoNet

TomoNet is a Python-based software package that integrates commonly used cryoET packages to streamline the cryoET and STA pipeline, with a particular emphasis on automating particle picking of lattice-configured structures and cryoET project management. As shown in the main menu and the entire TomoNet pipeline (Figures 1 and 2), after data collection from electron microscopy, TomoNet can perform motion correction with integration of MotionCorr2;Reference Zheng, Palovcak, Armache, Verba, Cheng and Agard ⁴² tilt series assembly and tomogram reconstruction with integration of IMOD;Reference Kremer, Mastronarde and McIntosh ⁴³ CTF estimation with integration of CTFFIND4;Reference Rohou and Grigorieff ⁴⁴ manual particle picking with IMOD; particle picking using built-in geometric template matching-based algorithms with integration of PEET;Reference Heumann, Hoenger and Mastronarde ⁴⁵ automatic particle picking using built-in deep learning-based algorithms; 3D classification/particle cleaning and subtomograms placing back with built-in algorithms. This design also allows on-the-fly tomogram reconstruction processing during data collection, which facilitates a quick quality check. TomoNet generates particle-picking results in STAR format,Reference Hall ⁴⁶ which can be incorporated into Relion for high-resolution 3D refinement. It can also read Relion results in STAR format for particle cleaning and subtomograms placing back (Figure 1).

2.2. Particle picking with “Auto Expansion”

The “Auto Expansion” module is based on template matching and uses cross-correlation coefficient as a selection criterion, with a design to pick particles on flexible lattices with minimal manual inputs; its basic concept is elucidated in Figure 3. These particles exist in array-like configurations and manifest as flexible, partial, and imperfect lattices in one, two, and three dimensions (1D–3D). Examples are abound: microtubule doublets, ubiquitous in most cells, consist of 96 nm axonemal 1D translational repeat unitsReference Imhof, Zhang, Wang, Bui, Nguyen, Atanasov, Hui, Yang, Zhou and Hill ^{21 ,} Reference Shimogawa, Wijono, Wang, Zhang, Sha, Szombathy, Vadakkan, Pelayo, Jonnalagadda, Wohlschlegel, Zhou and Hill ⁴⁷ (1D rotational lattice); HIV VLPsReference Schur, Obr, Hagen, Wan, Jakobi, Kirkpatrick, Sachse, Kräusslich and Briggs ⁴⁰ and surface layer (S-layer) lattice of prokaryotic cellsReference von Kügelgen, Alva and Bharat ^{48 ,} Reference Pum, Breitwieser and Sleytr ⁴⁹ are composed of hexametric subunits (2D lattice); paraflagellar rod of protozoan species is organized into para-crystalline arrays in its distal zoneReference Zhang, Wang, Imhof, Zhou, Liao, Atanasov, Hui, Hill and Zhou ²⁰ (3D lattice). In TomoNet, each of these isolated lattice densities is called a patch, within which all subunits of the complex are connected. For instance, Figure 3 shows two patches of different sizes.

Figure 3.

Illustration of the first two iterations of “Auto Expansion” particle picking. There are two patches of a hexagonal lattice with individual particles represented by solid hexagons. At iteration 0, 18 “candidate” particles (dashed blue) were selected from the neighbors of 3 “seed” particles (orange). 14 good particles remained and will serve as “seed” particles in iteration 1, and 3 “seed” particles in iteration 0 were saved in the final particle set (green). At iteration 1, 35 “candidate” particles were selected from the neighbors of 14 “seed” particles. 29 good particles remained and will serve as “seed” particles in iteration 2, and 14 “seed” particles were saved in the final particle set. “Auto Expansion” is an iterative process and will stop when no “candidate” can be detected.

“Auto Expansion” is an iterative process; each iteration expands the particle set by adding more unpicked ones. To initiate “Auto Expansion”, users need to prepare a few “seed” particles that sparsely distribute across all observed patches. Typically, the numbers of such “seed” particles per tomogram range from 20 to 200, which depends on the number and size of patches in the input tomogram. Then, “Auto Expansion” iteratively expands the “seed” particle set to a final particle set that contains all particles on given flexible lattices, following three steps for each iteration (Figure 3). First, potential particles adjacent to each “seed” particle are calculated and selected as “candidate” particles. Second, these “candidate” particles undergo alignments to a user-provided reference and are evaluated based on cross-correlation coefficient, such that “wrong” particles with low cross-correlations are excluded. Third, qualified “candidate” particles are added to the particle set and become “seed” particles for the next iteration. During this process, only unpicked ones can be considered as “candidate” particles, and “Auto Expansion” stops either when no “candidate” particles are detected or when the user-defined maximum iteration number is reached. Doing this allows for an exhaustive exploration of particles on given lattices following their assembly topology with no restriction on geometry and outputs a final particle-picking result (Figure 2).

Compared with conventional template matching methods, “Auto Expansion” incorporates prior knowledge of lattice configuration to iteratively guide the search for “candidate” particles, i.e., unpicked particles following user-defined paths, as detailed in the Method section and TomoNet’s user manual. Thus, “Auto Expansion” significantly reduces computational complexity by searching in the regions of interest only, with restricted angular and translational search ranges defined by users. As a result, it reduces the number of incorrectly picked particles. Notably, “Auto Expansion” potentially works for any flexible, imperfect, or variable lattices in 1D, 2D, and 3D and has no intrinsic size limit of subunits.

2.3. Automatic particle picking by deep learning

The “AI AutoPicking” module is designed for automatic particle picking using supervised machine learning, which uses a U-net convolutional neural network for model training. There are three main steps in “AI AutoPicking”: training data preparation, neural network training, and particle coordinate prediction, as detailed in the Method section (Figure 4). It only requires an input training dataset consisting of 1–3 tomograms paired with their corresponding particle coordinate files. The trained model can then be applied to the entire tomography dataset and output predicted particles for each tomogram.

Figure 4.

Illustration of “AI AutoPicking” process consisting of three steps. The HIV dataset was used for this illustration, and the particles refer to Gag hexamers. (a) Training dataset preparation. Using the user-provided tomograms with associated particle coordinate files, subtomograms containing particle densities were extracted. For each subtomogram, TomoNet generated a segmentation map based on the coordinates of particles, where the voxels near a particle’s center are shown as white and the others as black. (b) Neural network training. The generated subtomograms and segmentation maps were used as the input and output to train the convolutional neural network in learning how to segment out particle densities. (c) Particle coordinate prediction. Firstly, TomoNet applied the trained neural network model to unseen tomograms and generated associated predicted segmentation maps. Then, the particle coordinate information was obtained from the segmentation maps using clustering algorithms.

Essentially, the neural network in “AI AutoPicking” is trained as a voxel-wise binary classifier, which determines whether a voxel in density maps is part of a particle (Figure 4b). To prepare for training, data pairs (ground truth) consist of extracted subtomograms coupled with their associated segmentation maps, within which each particle is labeled by a cube near its center (Figure 4a). The trained neural network model can be applied to other tomograms to perform particle segmentation. Finally, the particle coordinate information can be retrieved from the predicted segmentation maps (Figure 4c).

2.4. 3D classification using TomoNet

In addition to the above two commentary modules for particle picking, TomoNet allows users to eliminate “bad” particles based on user-defined geometric constraints, which could serve as 3D classification during high-resolution particle refinements. Lattice variation in cryoET data has multiple plausible causes. Biologically, particles may be incomplete near the lattice edge due to paused biology assembly process.Reference Liu, Zhang, Wang, Tao, Bi and Zhou ⁵⁰ Experimentally, lattices tend to become flattened near the air-water interface of the sample during imaging. These variabilities pose challenges for 3D classification in the process of high-resolution STA, making it difficult to exclude “bad” particles that exhibit unexpected coordinates and orientations assignment as subunits of lattices (Supplementary Material S1).

Removing these “bad” particles is necessary for achieving better resolutions.Reference Tan, Pak, Morado, Voth and Briggs ⁵¹ To accomplish this, TomoNet assesses each particle by counting its neighboring particles and calculating the averaged tilt angle to these neighbors to represent the local surface curvature of a lattice. TomoNet identifies particles with too few neighbors or large tilt angles to their neighbors as “bad” particles since they potentially deviate from the lattice configuration. This step can be integrated into high-resolution refinement in Relion, providing an alternative 3D classification method based on analyzing spatial relationships between particles.

2.5. Application to in situ viral protein arrays: The matrix protein lattice in HIV VLPs

To validate TomoNet as an integrated high-resolution cryoET and STA pipeline and an efficient particle-picking tool, four tomograms were processed from the HIV-1 Gag dataset which resolved the Gag hexamer structure at 3.2 Å resolution. Motion-corrected images underwent tilt series assembly, CTF estimation, and tomographic reconstruction using TomoNet. Within these tomograms, the VLP hexagonal lattice and its building blocks were observed, and some of these observed VLPs showed sphere-like geometry (Figure 5a).

Figure 5.

TomoNet application to arrays of matrix protein in HIV VLPs. (a) Illustration of picked “seed” particles on a spherical VLP. Green segments represent the particles’ Y-axis. Scale bar is 20 nm. (b) “Auto Expansion” result on three VLPs within tomogram TS_01, with yellow dots representing the center of the hexamer subunits. (c) “AI AutoPicking” particle prediction result of tomogram TS_45 shows its ability to pick particles on all lattices of different sizes and shapes. (d) Visualization of three different variations of the HIV Gag lattices generated by placing back averaged structures, two exhibiting a spherical shape, and one presented as a fragment. Blue arrows indicate defects in the lattice.

As detailed in the Method section, a combination of “Auto Expansion” and “AI AutoPicking” was applied to the above four tomograms. The result shows that particles were readily picked on all the observed lattice patches (Figure 5b,c). Then, these picked particles were imported to Relion to perform high-resolution particle refinements, the resulting reconstruction of the Gag hexamer structure (Figure 6) looks identical to the published high-resolution structure,Reference Ni, Frosio, Mendonça, Sheng, Clare, Himes and Zhang ^{11 ,} Reference Zivanov, Otón, Ke, von Kügelgen, Pyle, Qu, Morado, Castaño-Díez, Zanetti, Bharat, Briggs and Scheres ¹³, demonstrating particle-picking accuracy and efficiency of TomoNet –capable of obtaining more particles from fewer tomograms.

Figure 6.

Final map resolution of HIV Gag hexamer. (a) Final reconstruction of Gag hexamer (grey) fitted with the atomic model (PDB: 5l93). (b) One segmented Gag monomer structure, inset shows a closer view of carboxy-terminal domain overlay with the atomic model. (c) Directional Fourier shell correlation (FSC) curves for the STA of Gag hexamer structure, with a global resolution at 3.2 Å.

Using the “3D subtomogram place back” function in TomoNet, 3D visualizations were generated to illustrate the in situ assembly of the VLP lattices (Figures 5d and 7). All VLP lattices with various sizes and shapes were captured even with irregular shapes (Figure 7e and Supplementary Material S2), demonstrating TomoNet’s particle-picking ability on flexible lattices. Lattice defects on each VLP were also identified consistent with previous studies,Reference Guo, Saha, Saffarian and Johnson ⁵² enhancing the understanding of lattice assembly mechanisms.Reference Talledge, Yang, Shi, Coray, Yu, Arndt, Meng, Baxter, Mendonça, Castaño-Díez, Aihara, Mansky and Zhang ⁵³

Figure 7.

Comparative visualization of lattices obtained from TomoNet and Relion tutorial. (a, b) Visualized comparison of particles used in TomoNet and Relion tutorial within tomogram TS_01. TomoNet can pick particles not only on a sphere-like lattice but also on others with random shapes. (c, d) A comparison of particle picking results on two sphere-like shape VLPs from TomoNet and Relion tutorial. (e) A zoom-in view of an irregularly shaped lattice. Coloring is based on surface curvatures at the point of each subunit.

2.6. Application to cellular organelle sample: Eukaryotic axoneme

We validated TomoNet’s particle-picking capability for 1D lattices by processing one tomogram of extracted flagellum of Trypanosoma brucei. The axoneme consists of 9 outer doublet microtubules (DMTs) and a pair of central singlet microtubules, where each DMT is a 1D polymer of 96 nm axonemal building blocks (Figure 8a). This typical 1-D lattice often exhibits imperfections like bends and twists, posing challenges for precise particle picking (Figure 8a). Using “Auto Expansion”, TomoNet accurately picked the 96 nm-spaced axonemal subunits from all DMTs, effectively adapting to lattice imperfections (Figure 8b).

Figure 8.

TomoNet application to eukaryotic axoneme. (a) Orthogonal slice views of axoneme structure of T. brucei, showing each DMT is a one-dimensional lattice. (b) Subtomogram placing back according to TomoNet “Auto Expansion” picking result with each DMT colored differently.

2.7. Application to focused ion beam (FIB)-milled cellular sample: The S-layer lattice of prokaryotic cell

We validated TomoNet’s particle-picking capability by processing one tomogram of FIB-milled Caulobacter crescentus cells from EMD-23622.Reference Lasker, Boeynaems, Lam, Scholl, Stainton, Briner, Jacquemyn, Daelemans, Deniz, Villa, Holehouse, Gitler and Shapiro ⁵⁴ The S-layer functions as a component of the cell wall covering the cell body. Thus, its lattice geometry is typically defined by the shape of cells (Figure 9a). The pleomorphic shape of C. crescentus cell in variable sizes, with the low contrast shown in this tomogram, hindered locating subunits on the S-layer lattice and raised difficulty for efficient particle picking on its S-layer lattice (Figure 9a).

Figure 9.

TomoNet application to S-layer structure in FIB-milled cellular sample. (a) A tomographic slice view shows two C. crescentus cells in a FIB-milled lamella. (b) Orthogonal slice views of the averaged density map generated in TomoNet, showing the hexagonal distribution of S-layer inner domains. Scale bar is 20 nm. (c) Our binned averaged map (transparent) docked with 7 copies of EMD-10388 (colored). (d) Visualization of S-layer lattices generated by placing back hexamer subunit maps simulated from PDB: 6P5T. Coloring is based on surface curvatures at the center of each subunit.

TomoNet overcame the above challenges by utilizing the hexagonal configuration of S-layer lattices. With minimal manual input, “Auto Expansion” picked over a thousand hexamer S-layer subunits. The binned STA result clearly reveals the S-layer inner domain, and docking previously resolved high-resolution structureReference von Kügelgen, Tang, Hardy, Kureisaite-Ciziene, Brun, Stansfeld, Robinson and Bharat ⁵⁵ (EMD-10388) into it confirms the correct hexagonal distribution with well-fitted major domains (Figure 9b,c). Visualization of S-layer lattices also shows that the picked particles were arranged in the expected hexagonal pattern, confirming the reliability and applicability of TomoNet as a particle-picking tool (Figure 9d) and its broad application to structure determination of prokaryotic and archaeal cell walls.Reference Pum, Breitwieser and Sleytr ^{49 ,} Reference Sleytr, Schuster, Egelseer and Pum ⁵⁶

2.8. Application to in vitro assembled arrays: Nuclear egress complex (NEC) lattice

We further validated TomoNet as an integrated high-resolution STA pipeline and an efficient particle-picking tool by processing samples containing NEC lattices within budded vehicles. Nuclear egress is a pivotal step in herpes virus replication, driven by NEC and responsible for translocating nascent viral particles from nucleus to cytoplasm. In our reported dataset,Reference Draganova, Wang, Wu, Liao, Vu, Gonzalez-Del Pino, Zhou, Roller and Heldwein ⁵⁷ NEC heterodimers budded into large vesicles with diameters ranging from 100 nm to 500 nm, forming beehive-like lattices on the inner surface of these vesicles (Figure 10a,b). Because of their large sizes, noticeable compressions were observed during the sample freezing, reshaping the vesicles and NEC lattices from spherical to flattened disk shapes (Figure 10a,b). This conformational change was a consequence of the limitation in ice thickness imposed by cryoET, which restricts the sample thickness to approximately 250 nm, consequently posing challenges for particle picking.

Figure 10.

TomoNet application to in vitro assembled NEC-bound membrane. (a, b) Tomographic slice views show a large NEC lattice; the insets show different views of NEC hexamer subunits. Scale bar is 20 nm. (c) Orthogonal slice views of an averaged density map generated in TomoNet show that NEC hexamer subunits consist of UL31/UL34 heterodimers. Scale bar is 10 nm. (d) Visualization of an NEC lattice generated by placing back averaged maps shows that the large vesicle is compressed into a disk-like shape. The compression caused by sample freezing stretched the lattice, making it flat and split at the air-water surface. Coloring is based on surface curvatures at the center of each subunit. (e) Atomic model of the UL31/UL34 heterodimers fits into the final averaged map, with all helices well resolved.

TomoNet successfully picked NEC hexamer subunits following the topology of lattices. The intermediate STA result generated in TomoNet already showed the six heterodimers within one hexamer subunit (Figure 10c). With these picked particles, high-resolution 3D classifications and refinements were carried out to obtain a final reconstruction of NEC hexamer subunit at 5.4 Å resolution, without preferred orientation bias (Figure 10c,d), and all the helices were well resolved (Figure 10e). Visualization of subtomograms placing back shows that the large vesicle was compressed during sample freezing which stretched the NEC lattice, making it appear flat and split at the air-water interface, while the middle part of the lattice appears to be more curved.

2.9. Application to other types of arrays and free-floating particles

The above examples show how TomoNet’s ability to locate particles arrays arranged on flexible spheres (HIV), cell surfaces (S-layer), and nuclear membranes (NEC), which can be considered as topologically 2D lattices. In our published work of various cryoET structures, TomoNet has also been used to locate subtomograms arranged on flexible filaments (i.e., 1D arrays) such as the flagella of T. brucei Reference Imhof, Zhang, Wang, Bui, Nguyen, Atanasov, Hui, Yang, Zhou and Hill ^{21 ,} Reference Shimogawa, Wijono, Wang, Zhang, Sha, Szombathy, Vadakkan, Pelayo, Jonnalagadda, Wohlschlegel, Zhou and Hill ⁴⁷ and the amyloid-like sheath protein on β-hoops of the prototypical archaeon, Methanospirillum hungatei. Reference Wang, Zhang, Toso, Liao, Sedighian, Gunsalus and Zhou ⁵⁸ In the case of 3D lattices, TomoNet has been also used to obtain the paraflagellar rod structure of T. brucei. Reference Zhang, Wang, Imhof, Zhou, Liao, Atanasov, Hui, Hill and Zhou ²⁰ Since TomoNet has integrated packages and is designed for the entire cryoET and STA data processing pipeline, it can also be used as a general-purpose package for STA toward high resolution when particles are free floating and without local order. In the latter case, TomoNet would have the same limitation recognized for all other cryoET software packages, that is, high resolution is currently only achieved for large complexes, such as ribosomes.