Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Home
Hostname: page-component-55597f9d44-ssw5r Total loading time: 0.39 Render date: 2022-08-13T06:30:57.692Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "useRatesEcommerce": false, "useNewApi": true } hasContentIssue true
  • English
Microscopy and Microanalysis Microscopy and Microanalysis

Article contents

Quantifying Variability of Manual Annotation in Cryo-Electron Tomograms

Published online by Cambridge University Press:  26 May 2016

Corey W. Hecksel ,
Michele C. Darrow ,
Wei Dai ,
Jesús G. Galaz-Montoya ,
Jessica A. Chin ,
Patrick G. Mitchell ,
Shurui Chen ,
Jemba Jakana ,
Michael F. Schmid  and
Wah Chiu
Corey W. Hecksel
Affiliation:
Molecular Virology and Microbiology Department, Baylor College of Medicine, Houston, TX 77030, USA National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Michele C. Darrow
Affiliation:
Verna and Marrs McLean Department of Biochemistry and Molecular Biology, Baylor College of Medicine, Houston, TX 77030, USA National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Wei Dai
Affiliation:
National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Jesús G. Galaz-Montoya
Affiliation:
National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Jessica A. Chin
Affiliation:
National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Patrick G. Mitchell
Affiliation:
National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Shurui Chen
Affiliation:
National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Jemba Jakana
Affiliation:
National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Michael F. Schmid
Affiliation:
Verna and Marrs McLean Department of Biochemistry and Molecular Biology, Baylor College of Medicine, Houston, TX 77030, USA National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Wah Chiu*
Affiliation:
Molecular Virology and Microbiology Department, Baylor College of Medicine, Houston, TX 77030, USA Verna and Marrs McLean Department of Biochemistry and Molecular Biology, Baylor College of Medicine, Houston, TX 77030, USA National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
*
*Corresponding author. wah@bcm.edu
Get access Rights & Permissions[Opens in a new window]

Abstract

Although acknowledged to be variable and subjective, manual annotation of cryo-electron tomography data is commonly used to answer structural questions and to create a “ground truth” for evaluation of automated segmentation algorithms. Validation of such annotation is lacking, but is critical for understanding the reproducibility of manual annotations. Here, we used voxel-based similarity scores for a variety of specimens, ranging in complexity and segmented by several annotators, to quantify the variation among their annotations. In addition, we have identified procedures for merging annotations to reduce variability, thereby increasing the reliability of manual annotation. Based on our analyses, we find that it is necessary to combine multiple manual annotations to increase the confidence level for answering structural questions. We also make recommendations to guide algorithm development for automated annotation of features of interest.

Keywords

cryo-electron tomographysegmentationannotationvalidationDice coefficient
Type
Technique and Instrumentation Development
Information
Microscopy and Microanalysis , Volume 22 , Issue 3 , June 2016 , pp. 487 - 496
Copyright
Copyright © Microscopy Society of America 2016

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

†

Current address: Diamond Light Source Ltd, Science Division, Fermi Ave, Didcot, Oxfordshire OX11 0DX, UK.

‡

Current address: Department of Cell Biology and Neuroscience, Center for Integrative Proteomics Research, Rutgers University, 174 Frelinghuysen Road, Piscataway, NJ 08854-8076, USA.

a

Corey W. Hecksel and Michele C. Darrow contributed equally to this work.

References

Asano, S., Fukuda, Y., Beck, F., Aufderheide, A., Förster, F., Danev, R. & Baumeister, W. (2015). Proteasomes. A molecular census of 26S proteasomes in intact neurons. Science 347, 439–442.Google Scholar
Dai, W., Fu, C., Raytcheva, D., Flanagan, J., Khant, H.A., Liu, X., Rochat, R.H., Haase-Pettingell, C., Piret, J., Ludtke, S.J., Nagayama, K., Schmid, M.F., King, J.A. & Chiu, W. (2013). Visualizing virus assembly intermediates inside marine cyanobacteria. Nature 502, 707–710.Google Scholar
Darrow, M.C., Sergeeva, O.A., Isas, J.M., Galaz-Montoya, J., King, J.A., Langen, R., Schmid, M.F. & Chiu, W. (2015). Structural mechanisms of mutant huntingtin aggregation suppression by synthetic chaperonin-like CCT5 complex explained by cryo-electron tomography. J Biol Chem 290, 17451–17461.Google Scholar
Frangakis, A.S. & Förster, F. (2004). Computational exploration of structural information from cryo-electron tomograms. Curr Opin Struct Biol 14, 325–331.Google Scholar
Frangakis, A.S. & Hegerl, R. (2001). Noise reduction in electron tomographic reconstructions using nonlinear anisotropic diffusion. J Struct Biol 135, 239–250.Google Scholar
Garduño, E., Wong-Barnum, M., Volkmann, N. & Ellisman, M.H. (2008). Segmentation of electron tomographic data sets using fuzzy set theory principles. J Struct Biol 162, 368–379.Google Scholar
Gilliam, J.C., Chang, J.T., Sandoval, I.M., Zhang, Y., Li, T., Pittler, S.J., Chiu, W. & Wensel, T.G. (2012). Three-dimensional architecture of the rod sensory cilium and its disruption in retinal neurodegeneration. Cell 151, 1029–1041.Google Scholar
GraphPad QuickCalcs (2015). GraphPad QuickCalcs: T test calculator. Available at http://www.graphpad.com/quickcalcs/ttest1.cfm (retrieved July 6, 2015).Google Scholar
Hoppe, W. (1974 a). Three-demensional electron microscopic reconstruction of an object. Naturwissenschaften 61, 534–536.Google Scholar
Hoppe, W. (1974 b). Towards three-dimensional “electron microscopy” at atomic resolution. Naturwissenschaften 61, 239–249.Google Scholar
Ibiricu, I., Huiskonen, J.T., Döhner, K., Bradke, F., Sodeik, B. & Grünewald, K. (2011). Cryo electron tomography of herpes simplex virus during axonal transport and secondary envelopment in primary neurons. PLoS Pathog 7, e1002406.Google Scholar
Koning, R.I., Zovko, S., Bárcena, M., Oostergetel, G.T., Koerten, H.K., Galjart, N., Koster, A.J. & Mieke Mommaas, A. (2008). Cryo electron tomography of vitrified fibroblasts: Microtubule plus ends in situ. J Struct Biol 161, 459–468.Google Scholar
Kremer, J.R., Mastronarde, D.N. & McIntosh, J.R. (1996). Computer visualization of three-dimensional image data using IMOD. J Struct Biol 116, 71–76.Google Scholar
Lepault, J., Booy, F.P. & Dubochet, J. (1983). Electron microscopy of frozen biological suspensions. J Microsc 129, 89–102.Google Scholar
Lučić, V., Förster, F. & Baumeister, W. (2005). Structural studies by electron tomography: From cells to molecules. Annu Rev Biochem 74, 833–865.Google Scholar
Lučić, V., Rigort, A. & Baumeister, W. (2013). Cryo-electron tomography: The challenge of doing structural biology in situ. J Cell Biol 202, 407–419.Google Scholar
Maimon, T., Elad, N., Dahan, I. & Medalia, O. (2012). The human nuclear pore complex as revealed by cryo-electron tomography. Structure 20, 998–1006.Google Scholar
Martinez-Sanchez, A., Garcia, I. & Fernandez, J.-J. (2011). A differential structure approach to membrane segmentation in electron tomography. J Struct Biol 175, 372–383.Google Scholar
Martinez-Sanchez, A., Garcia, I. & Fernandez, J.-J. (2013). A ridge-based framework for segmentation of 3D electron microscopy datasets. J Struct Biol 181, 61–70.Google Scholar
Maurer, U.E., Sodeik, B. & Grünewald, K. (2008). Native 3D intermediates of membrane fusion in herpes simplex virus 1 entry. Proc Natl Acad Sci U S A 105, 10559–10564.Google Scholar
Moussavi, F., Heitz, G., Amat, F., Comolli, L.R., Koller, D. & Horowitz, M. (2010). 3D segmentation of cell boundaries from whole cell cryogenic electron tomography volumes. J Struct Biol 170, 134–145.Google Scholar
Nguyen, H. & Ji, Q. (2008). Shape-driven three-dimensional watersnake segmentation of biological membranes in electron tomography. IEEE Trans Med Imaging 27, 616–628.Google Scholar
Page, C., Hanein, D. & Volkmann, N. (2015). Accurate membrane tracing in three-dimensional reconstructions from electron cryotomography data. Ultramicroscopy 155, 20–26.Google Scholar
Patla, I., Volberg, T., Elad, N., Hirschfeld-Warneken, V., Grashoff, C., Fässler, R., Spatz, J.P., Geiger, B. & Medalia, O. (2010). Dissecting the molecular architecture of integrin adhesion sites by cryo-electron tomography. Nat Cell Biol 12, 909–915.Google Scholar
Rigort, A., Günther, D., Hegerl, R., Baum, D., Weber, B., Prohaska, S., Medalia, O., Baumeister, W. & Hege, H.-C. (2012). Automated segmentation of electron tomograms for a quantitative description of actin filament networks. J Struct Biol 177, 135–144.Google Scholar
Rusu, M., Starosolski, Z., Wahle, M., Rigort, A. & Wriggers, W. (2012). Automated tracing of filaments in 3D electron tomography reconstructions using Sculptor and Situs. J Struct Biol 178, 121–128.Google Scholar
Sandberg, K. (2007). Methods for image segmentation in cellular tomography. Methods Cell Biol 79, 769–798.Google Scholar
Shahmoradian, S.H., Galaz-Montoya, J.G., Schmid, M.F., Cong, Y., Ma, B., Spiess, C., Frydman, J., Ludtke, S.J. & Chiu, W. (2013). TRiC’s tricks inhibit huntingtin aggregation. eLife 2, e00710.Google Scholar
Shahmoradian, S.H., Galiano, M.R., Wu, C., Chen, S., Rasband, M.N., Mobley, W.C. & Chiu, W. (2014). Preparation of primary neurons for visualizing neurites in a frozen-hydrated state using cryo-electron tomography. J Vis Exp 84, e50783.Google Scholar
Tang, G., Peng, L., Baldwin, P.R., Mann, D.S., Jiang, W., Rees, I. & Ludtke, S.J. (2007). EMAN2: An extensible image processing suite for electron microscopy. J Struct Biol 157, 38–46.Google Scholar
Taylor, K.A. & Glaeser, R.M. (1974). Electron diffraction of frozen, hydrated protein crystals. Science 186, 1036–1037.Google Scholar
Tsai, W.-T., Hassan, A., Sarkar, P., Correa, J., Metlagel, Z., Jorgens, D.M. & Auer, M. (2014). From voxels to knowledge: A practical guide to the segmentation of complex electron microscopy 3D-data. J Vis Exp 90, e51673.Google Scholar
van der Heide, P., Xu, X.-P., Marsh, B.J., Hanein, D. & Volkmann, N. (2007). Efficient automatic noise reduction of electron tomographic reconstructions based on iterative median filtering. J Struct Biol 158, 196–204.Google Scholar
Volkmann, N. (2002). A novel three-dimensional variant of the watershed transform for segmentation of electron density maps. J Struct Biol 138, 123–129.Google Scholar
Volkmann, N. (2010). Methods for segmentation and interpretation of electron tomographic reconstructions. In Methods in Enzymology. Volume 483, Cryo-EM, Part C: Analyses, Interpretation, and Case studies (chapter 2), G.J. Jensen (Ed.), pp. 31–46. Cambridge: Academic Press. Available at http://www.sciencedirect.com/science/article/pii/S0076687910830022 (retrieved November 7, 2014).Google Scholar
Wang, R., Stone, R.L., Kaelber, J.T., Rochat, R.H., Nick, A.M., Vijayan, K.V., Afshar-Kharghan, V., Schmid, M.F., Dong, J.-F., Sood, A.K. & Chiu, W. (2015). Electron cryotomography reveals ultrastructure alterations in platelets from patients with ovarian cancer. Proc Natl Acad Sci U S A 112, 14266–14271.Google Scholar
Woodward, C.L., Cheng, S.N. & Jensen, G.J. (2014). Electron cryotomography studies of maturing HIV-1 particles reveal the assembly pathway of the viral core. J Virol 89, 1267–1277.Google Scholar
Zhao, X., Zhang, K., Boquoi, T., Hu, B., Motaleb, M.A., Miller, K.A., James, M.E., Charon, N.W., Manson, M.D., Norris, S.J., Li, C. & Liu, J. (2013). Cryoelectron tomography reveals the sequential assembly of bacterial flagella in Borrelia burgdorferi. Proc Natl Acad Sci U S America 110, 14390–14395.Google Scholar
Supplementary material: File

Hecksel supplementary material S1

Supplementary Table

Download Hecksel supplementary material S1(File)
File 104 KB
Supplementary material: File

Hecksel supplementary material S2

Supplementary Table

Download Hecksel supplementary material S2(File)
File 147 KB

Hecksel supplementary material S3

Supplementary Movie

Download Hecksel supplementary material S3(Video)
Video 26 MB

Hecksel supplementary material S4

Supplementary Movie

Download Hecksel supplementary material S4(Video)
Video 11 MB
11
Cited by

Save article to Kindle

To save this article to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Quantifying Variability of Manual Annotation in Cryo-Electron Tomograms
Available formats
×

Save article to Dropbox

To save this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about saving content to Dropbox.

Quantifying Variability of Manual Annotation in Cryo-Electron Tomograms
Available formats
×

Save article to Google Drive

To save this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about saving content to Google Drive.

Quantifying Variability of Manual Annotation in Cryo-Electron Tomograms
Available formats
×
×

Reply to: Submit a response

Please enter your response.

Your details

Please enter a valid email address.

Conflicting interests

Do you have any conflicting interests? *