Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Home
  • Home
Hostname: page-component-684bc48f8b-b5g75 Total loading time: 19.985 Render date: 2021-04-13T08:14:39.095Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "metricsAbstractViews": false, "figures": false, "newCiteModal": false, "newCitedByModal": true }
EnglishFrançais
Microscopy and Microanalysis Microscopy and Microanalysis

Article contents

Quantifying Variability of Manual Annotation in Cryo-Electron Tomograms

Published online by Cambridge University Press:  26 May 2016

Corey W. Hecksel ,
Michele C. Darrow ,
Wei Dai ,
Jesús G. Galaz-Montoya ,
Jessica A. Chin ,
Patrick G. Mitchell ,
Shurui Chen ,
Jemba Jakana ,
Michael F. Schmid  and
Wah Chiu
Corey W. Hecksel
Affiliation:
Molecular Virology and Microbiology Department, Baylor College of Medicine, Houston, TX 77030, USA National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Michele C. Darrow
Affiliation:
Verna and Marrs McLean Department of Biochemistry and Molecular Biology, Baylor College of Medicine, Houston, TX 77030, USA National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Wei Dai
Affiliation:
National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Jesús G. Galaz-Montoya
Affiliation:
National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Jessica A. Chin
Affiliation:
National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Patrick G. Mitchell
Affiliation:
National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Shurui Chen
Affiliation:
National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Jemba Jakana
Affiliation:
National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Michael F. Schmid
Affiliation:
Verna and Marrs McLean Department of Biochemistry and Molecular Biology, Baylor College of Medicine, Houston, TX 77030, USA National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Wah Chiu
Affiliation:
Molecular Virology and Microbiology Department, Baylor College of Medicine, Houston, TX 77030, USA Verna and Marrs McLean Department of Biochemistry and Molecular Biology, Baylor College of Medicine, Houston, TX 77030, USA National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Corresponding
E-mail address:
wah@bcm.edu
Get access
Rights & Permissions[Opens in a new window]

Abstract

Although acknowledged to be variable and subjective, manual annotation of cryo-electron tomography data is commonly used to answer structural questions and to create a “ground truth” for evaluation of automated segmentation algorithms. Validation of such annotation is lacking, but is critical for understanding the reproducibility of manual annotations. Here, we used voxel-based similarity scores for a variety of specimens, ranging in complexity and segmented by several annotators, to quantify the variation among their annotations. In addition, we have identified procedures for merging annotations to reduce variability, thereby increasing the reliability of manual annotation. Based on our analyses, we find that it is necessary to combine multiple manual annotations to increase the confidence level for answering structural questions. We also make recommendations to guide algorithm development for automated annotation of features of interest.

Keywords

cryo-electron tomography segmentation annotation validation Dice coefficient
Type
Technique and Instrumentation Development
Information
Microscopy and Microanalysis , Volume 22 , Issue 3 , June 2016 , pp. 487 - 496
Copyright
Copyright © Microscopy Society of America 2016

Access options

Get access to the full version of this content by using one of the access options below.
If you should have access and can't see this content please contact technical support.

Footnotes

†

Current address: Diamond Light Source Ltd, Science Division, Fermi Ave, Didcot, Oxfordshire OX11 0DX, UK.

‡

Current address: Department of Cell Biology and Neuroscience, Center for Integrative Proteomics Research, Rutgers University, 174 Frelinghuysen Road, Piscataway, NJ 08854-8076, USA.

a

Corey W. Hecksel and Michele C. Darrow contributed equally to this work.

References

Asano, S., Fukuda, Y., Beck, F., Aufderheide, A., Förster, F., Danev, R. & Baumeister, W. (2015). Proteasomes. A molecular census of 26S proteasomes in intact neurons. Science 347, 439–442.CrossRefGoogle ScholarPubMed
Dai, W., Fu, C., Raytcheva, D., Flanagan, J., Khant, H.A., Liu, X., Rochat, R.H., Haase-Pettingell, C., Piret, J., Ludtke, S.J., Nagayama, K., Schmid, M.F., King, J.A. & Chiu, W. (2013). Visualizing virus assembly intermediates inside marine cyanobacteria. Nature 502, 707–710.CrossRefGoogle ScholarPubMed
Darrow, M.C., Sergeeva, O.A., Isas, J.M., Galaz-Montoya, J., King, J.A., Langen, R., Schmid, M.F. & Chiu, W. (2015). Structural mechanisms of mutant huntingtin aggregation suppression by synthetic chaperonin-like CCT5 complex explained by cryo-electron tomography. J Biol Chem 290, 17451–17461.CrossRefGoogle Scholar
Frangakis, A.S. & Förster, F. (2004). Computational exploration of structural information from cryo-electron tomograms. Curr Opin Struct Biol 14, 325–331.CrossRefGoogle ScholarPubMed
Frangakis, A.S. & Hegerl, R. (2001). Noise reduction in electron tomographic reconstructions using nonlinear anisotropic diffusion. J Struct Biol 135, 239–250.CrossRefGoogle ScholarPubMed
Garduño, E., Wong-Barnum, M., Volkmann, N. & Ellisman, M.H. (2008). Segmentation of electron tomographic data sets using fuzzy set theory principles. J Struct Biol 162, 368–379.CrossRefGoogle ScholarPubMed
Gilliam, J.C., Chang, J.T., Sandoval, I.M., Zhang, Y., Li, T., Pittler, S.J., Chiu, W. & Wensel, T.G. (2012). Three-dimensional architecture of the rod sensory cilium and its disruption in retinal neurodegeneration. Cell 151, 1029–1041.CrossRefGoogle ScholarPubMed
GraphPad QuickCalcs (2015). GraphPad QuickCalcs: T test calculator. Available at http://www.graphpad.com/quickcalcs/ttest1.cfm (retrieved July 6, 2015).Google Scholar
Hoppe, W. (1974 a). Three-demensional electron microscopic reconstruction of an object. Naturwissenschaften 61, 534–536.CrossRefGoogle Scholar
Hoppe, W. (1974 b). Towards three-dimensional “electron microscopy” at atomic resolution. Naturwissenschaften 61, 239–249.CrossRefGoogle ScholarPubMed
Ibiricu, I., Huiskonen, J.T., Döhner, K., Bradke, F., Sodeik, B. & Grünewald, K. (2011). Cryo electron tomography of herpes simplex virus during axonal transport and secondary envelopment in primary neurons. PLoS Pathog 7, e1002406.CrossRefGoogle ScholarPubMed
Koning, R.I., Zovko, S., Bárcena, M., Oostergetel, G.T., Koerten, H.K., Galjart, N., Koster, A.J. & Mieke Mommaas, A. (2008). Cryo electron tomography of vitrified fibroblasts: Microtubule plus ends in situ. J Struct Biol 161, 459–468.CrossRefGoogle ScholarPubMed
Kremer, J.R., Mastronarde, D.N. & McIntosh, J.R. (1996). Computer visualization of three-dimensional image data using IMOD. J Struct Biol 116, 71–76.CrossRefGoogle ScholarPubMed
Lepault, J., Booy, F.P. & Dubochet, J. (1983). Electron microscopy of frozen biological suspensions. J Microsc 129, 89–102.CrossRefGoogle ScholarPubMed
Lučić, V., Förster, F. & Baumeister, W. (2005). Structural studies by electron tomography: From cells to molecules. Annu Rev Biochem 74, 833–865.CrossRefGoogle ScholarPubMed
Lučić, V., Rigort, A. & Baumeister, W. (2013). Cryo-electron tomography: The challenge of doing structural biology in situ. J Cell Biol 202, 407–419.CrossRefGoogle ScholarPubMed
Maimon, T., Elad, N., Dahan, I. & Medalia, O. (2012). The human nuclear pore complex as revealed by cryo-electron tomography. Structure 20, 998–1006.CrossRefGoogle ScholarPubMed
Martinez-Sanchez, A., Garcia, I. & Fernandez, J.-J. (2011). A differential structure approach to membrane segmentation in electron tomography. J Struct Biol 175, 372–383.CrossRefGoogle ScholarPubMed
Martinez-Sanchez, A., Garcia, I. & Fernandez, J.-J. (2013). A ridge-based framework for segmentation of 3D electron microscopy datasets. J Struct Biol 181, 61–70.CrossRefGoogle ScholarPubMed
Maurer, U.E., Sodeik, B. & Grünewald, K. (2008). Native 3D intermediates of membrane fusion in herpes simplex virus 1 entry. Proc Natl Acad Sci U S A 105, 10559–10564.CrossRefGoogle ScholarPubMed
Moussavi, F., Heitz, G., Amat, F., Comolli, L.R., Koller, D. & Horowitz, M. (2010). 3D segmentation of cell boundaries from whole cell cryogenic electron tomography volumes. J Struct Biol 170, 134–145.CrossRefGoogle ScholarPubMed
Nguyen, H. & Ji, Q. (2008). Shape-driven three-dimensional watersnake segmentation of biological membranes in electron tomography. IEEE Trans Med Imaging 27, 616–628.CrossRefGoogle ScholarPubMed
Page, C., Hanein, D. & Volkmann, N. (2015). Accurate membrane tracing in three-dimensional reconstructions from electron cryotomography data. Ultramicroscopy 155, 20–26.CrossRefGoogle ScholarPubMed
Patla, I., Volberg, T., Elad, N., Hirschfeld-Warneken, V., Grashoff, C., Fässler, R., Spatz, J.P., Geiger, B. & Medalia, O. (2010). Dissecting the molecular architecture of integrin adhesion sites by cryo-electron tomography. Nat Cell Biol 12, 909–915.CrossRefGoogle ScholarPubMed
Rigort, A., Günther, D., Hegerl, R., Baum, D., Weber, B., Prohaska, S., Medalia, O., Baumeister, W. & Hege, H.-C. (2012). Automated segmentation of electron tomograms for a quantitative description of actin filament networks. J Struct Biol 177, 135–144.CrossRefGoogle ScholarPubMed
Rusu, M., Starosolski, Z., Wahle, M., Rigort, A. & Wriggers, W. (2012). Automated tracing of filaments in 3D electron tomography reconstructions using Sculptor and Situs. J Struct Biol 178, 121–128.CrossRefGoogle ScholarPubMed
Sandberg, K. (2007). Methods for image segmentation in cellular tomography. Methods Cell Biol 79, 769–798.CrossRefGoogle ScholarPubMed
Shahmoradian, S.H., Galaz-Montoya, J.G., Schmid, M.F., Cong, Y., Ma, B., Spiess, C., Frydman, J., Ludtke, S.J. & Chiu, W. (2013). TRiC’s tricks inhibit huntingtin aggregation. eLife 2, e00710.CrossRefGoogle ScholarPubMed
Shahmoradian, S.H., Galiano, M.R., Wu, C., Chen, S., Rasband, M.N., Mobley, W.C. & Chiu, W. (2014). Preparation of primary neurons for visualizing neurites in a frozen-hydrated state using cryo-electron tomography. J Vis Exp 84, e50783.Google Scholar
Tang, G., Peng, L., Baldwin, P.R., Mann, D.S., Jiang, W., Rees, I. & Ludtke, S.J. (2007). EMAN2: An extensible image processing suite for electron microscopy. J Struct Biol 157, 38–46.CrossRefGoogle ScholarPubMed
Taylor, K.A. & Glaeser, R.M. (1974). Electron diffraction of frozen, hydrated protein crystals. Science 186, 1036–1037.CrossRefGoogle ScholarPubMed
Tsai, W.-T., Hassan, A., Sarkar, P., Correa, J., Metlagel, Z., Jorgens, D.M. & Auer, M. (2014). From voxels to knowledge: A practical guide to the segmentation of complex electron microscopy 3D-data. J Vis Exp 90, e51673.Google Scholar
van der Heide, P., Xu, X.-P., Marsh, B.J., Hanein, D. & Volkmann, N. (2007). Efficient automatic noise reduction of electron tomographic reconstructions based on iterative median filtering. J Struct Biol 158, 196–204.CrossRefGoogle ScholarPubMed
Volkmann, N. (2002). A novel three-dimensional variant of the watershed transform for segmentation of electron density maps. J Struct Biol 138, 123–129.CrossRefGoogle ScholarPubMed
Volkmann, N. (2010). Methods for segmentation and interpretation of electron tomographic reconstructions. In Methods in Enzymology. Volume 483, Cryo-EM, Part C: Analyses, Interpretation, and Case studies (chapter 2), G.J. Jensen (Ed.), pp. 31–46. Cambridge: Academic Press. Available at http://www.sciencedirect.com/science/article/pii/S0076687910830022 (retrieved November 7, 2014).CrossRefGoogle Scholar
Wang, R., Stone, R.L., Kaelber, J.T., Rochat, R.H., Nick, A.M., Vijayan, K.V., Afshar-Kharghan, V., Schmid, M.F., Dong, J.-F., Sood, A.K. & Chiu, W. (2015). Electron cryotomography reveals ultrastructure alterations in platelets from patients with ovarian cancer. Proc Natl Acad Sci U S A 112, 14266–14271.CrossRefGoogle ScholarPubMed
Woodward, C.L., Cheng, S.N. & Jensen, G.J. (2014). Electron cryotomography studies of maturing HIV-1 particles reveal the assembly pathway of the viral core. J Virol 89, 1267–1277.CrossRefGoogle ScholarPubMed
Zhao, X., Zhang, K., Boquoi, T., Hu, B., Motaleb, M.A., Miller, K.A., James, M.E., Charon, N.W., Manson, M.D., Norris, S.J., Li, C. & Liu, J. (2013). Cryoelectron tomography reveals the sequential assembly of bacterial flagella in Borrelia burgdorferi. Proc Natl Acad Sci U S America 110, 14390–14395.CrossRefGoogle ScholarPubMed

Hecksel supplementary material S1

Supplementary Table

File 104 KB

Hecksel supplementary material S2

Supplementary Table

File 147 KB

Hecksel supplementary material S3

Supplementary Movie

Video 26 MB

Hecksel supplementary material S4

Supplementary Movie

Video 11 MB

Altmetric attention score

Full text views

Full text views reflects PDF downloads, PDFs sent to Google Drive, Dropbox and Kindle and HTML full text views.

Total number of HTML views: 41
Total number of PDF views: 114 *
View data table for this chart

* Views captured on Cambridge Core between September 2016 - 13th April 2021. This data will be updated every 24 hours.

7
Cited by

Send article to Kindle

To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Quantifying Variability of Manual Annotation in Cryo-Electron Tomograms
Available formats
×

Send article to Dropbox

To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

Quantifying Variability of Manual Annotation in Cryo-Electron Tomograms
Available formats
×

Send article to Google Drive

To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

Quantifying Variability of Manual Annotation in Cryo-Electron Tomograms
Available formats
×
×

Reply to: Submit a response

Your details

Conflicting interests

Do you have any conflicting interests? *