Skip to main content

Diagnostic accuracy and confusability analyses: an application to the Diagnostic Interview for Genetic Studies

  • S. V. Faraone (a1), M. Blehar (a1), J. Pepple (a1), S. O. Moldin (a1), J. Norton (a1), J. I. Nurnberger (a1), D. Malaspina (a1), C. A. Kaufmann (a1), T. Reich (a1), C. R. Cloninger (a1), J. R. DePaulo (a1), K. Berg (a1), E. S. Gershon (a1), D. G. Kirch (a1) and M. T. Tsuang (a1)...

The dominant, contemporary paradigm for developing and refining diagnoses relies heavily on assessing reliability with kappa coefficients and virtually ignores a core component of psychometric practice: the theory of latent structures. This article describes a psychometric approach to psychiatric nosology that emphasizes the diagnostic accuracy and confusability of diagnostic categories. We apply these methods to the Diagnostic Interview for Genetic Studies (DIGS), a structured psychiatric interview designed by the NIMH Genetics Initiative for genetic studies of schizophrenia and bipolar disorder. Our results show that sensitivity and specificity were excellent for both DSM-III-R and RDC diagnoses of major depression, bipolar disorder, and schizophrenia. In contrast, diagnostic accuracy was substantially lower for subtypes of schizoaffective disorder – especially for the DSM-III-R definitions. Both the bipolar and depressed subtypes of DSM-III-R schizoaffective disorder had excellent specificity but poor sensitivity. The RDC definitions also had excellent specificity but were more sensitive than the DSM-III-R schizoaffective diagnoses. The source of low sensitivity for schizoaffective subtypes differed for the two diagnostic systems. For RDC criteria, the schizoaffective subtypes were frequently confused with one another; they were less frequently confused with other diagnoses. In contrast, the DSM-III-R subtypes were often confused with schizophrenia, but not with each other.

Corresponding author
1Address for correspondence: Dr Stephen V. Faraone, Psychiatry Service (116A), Veterans Affairs Medical Center, 940 Belmont Street, Brockton, MA 02401, USA.
Hide All
American Psychiatric Association (1987). Diagnostic and Statistical Manual of Mental Disorders, 3rd edn, revised. American Psychiatric Association: Washington, DC.
Andreasen, N. C., Flaum, M. & Arndt, S. (1992). The comprehensive assessment of symptoms and history (CASH). Archives of General Psychiatry 432, 615623.
Blacker, D., Lavori, P. W., Faraone, S. V. & Tsuang, M. T. (1993). Unipolar relatives in bipolar pedigrees: a search for indicators underlying bipolarity. American Journal of Medical Genetics, Neuropsychiatric Genetics 48 192199.
Chen, W. J., Faraone, S. V. & Tsuang, M. T. (1992). Linkage studies of schizophrenia: a simulation study of statistical power. Genetic Epidemiology 9, 123139.
Clogg, C. C. (1977). Unrestricted and Restricted Maximum Likelihood Latent Class Analysis: A Manual for Users. Pennsylvania State University Press: University Park, PA.
Deutsch, C. K., Matthysse, S., Swanson, J. M. & Farkas, L. G. (1990). Genetic latent structure analysis of dysmorphology in attention deficit disorder. Journal of the American Academy of Child and Adolescent Psychiatry 29, 189194.
Diamond, G. A., Rozanski, A., Forrester, J. S., Morris, D., Pollock, B. H., Staniloff, H. M., Berman, D. S. & Swan, H. J. C. (1986). A model for assessing the sensitivity and specificity of tests subject to selection bias. Journal of Chronic Diseases 39, 343355.
Espeland, M. A. & Handelman, S. L. (1989). Using latent class models to characterize and assess relative error in discrete measurements. Biometrics 45, 587599.
Espeland, M. A., Murphy, W. C. & Leverett, D. H. (1988). Assessing diagnostic reliability and estimating incidence rates associated with a strictly progressive disease: dental caries. Statistics in Medicine 7, 403416.
Faraone, S. V. & Tsuang, M. T. (1994). Measuring diagnostic accuracy in the absence of a gold standard. American Journal of Psychiatry 151, 650657.
Faraone, S. V., Biederman, J., Sprich, S., Chen, W. J. & Tsuang, M. T. (1993). Efficiency of diagnostic criteria for attention deficit disorder: toward an empirical approach to designing and validating diagnostic algorithms. Journal of the American Academy of Child and Adolescent Psychiatry 32, 166174.
Faraone, S. V., Seidman, L. J., Kremen, W. S., Pepple, J. R., Lyons, M. J. & Tsuang, M. T. (1995). Neuropsychological functioning among the nonpsychotic relatives of schizophrenic patients: a diagnostic efficiency analysis. Journal of Abnormal Psychology 104, 286304.
Gastwirth, J. L. (1987). The statistical precision of medical screening procedures: application to polygraph and AIDS antibodies test data. Statistical Science 2, 213238.
Greenberg, D. A. (1992). There is more than one way to collect data for linkage analysis. What a study of epilepsy can tell us about linkage strategy for psychiatric disease. Archives of General Psychiatry 49, 745750.
Grove, W. M., Andreasen, N. C., McDonald-Scott, P., Keller, M. B. & Shapiro, R. W. (1981). Reliability studies of psychiatric diagnosis. Theory and practice. Archives of General Psychiatry 38, 408413.
Henkelman, R. M., Kay, I. & Bronskill, M. J. (1990). Receiver operating characteristic (ROC) analysis without truth. Medical Decision Making 10, 2429.
Holzman, P. S., Kringlen, E., Matthysse, S., Falanagan, S. D., Lipton, R. B., Cramer, G., Levin, S., Lange, K. & Levy, D. L. (1988). A single dominant gene can account for eye tracking dysfunctions and schizophrenia in offspring of discordant twins. Archives of General Psychiatry 45, 641647.
Hui, S. L. & Walter, S. D. (1980). Estimating the error rates of diagnostic tests. Biometrics 36, 167171.
Kraemer, H. C. (1992). Evaluating Medical Tests: Objective and Quantitative Guidelines. Sage Publications: Newbury Park, CA.
Landis, J. R. & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics 33, 159174.
Lau, T. S. (1989). On repeated screening tests. Biometrics 45, 891898.
Lau, T. S. (1991). On dependent repeated screening tests. Biometrics 47, 7786.
Lazarsfeld, P. F. & Henry, N. W. (1968). Latent Structure Analysis. Houghton Mifflin Company: New York.
Leboyer, M., Babron, M. C. & Clerget-Darpoux, F. (1990). Sampling strategy in linkage studies of affective disorders. Psychological Medicine 20, 573579.
LeBoyer, M., Maier, W., Teherani, M., Lichtermann, D., D'Amato, T., Franke, P., Lepine, J., Minges, J. & McGuffin, P. (1991). The reliability of the SADS-LA in a family study setting. European Archives of Psychiatry and Clinical Neuroscience 241, 165169.
Leckman, J. F., Sholomska, D., Thompson, W. D., Belanger, A. & Weissman, M. M. (1982). Best estimate of lifetime diagnosis: a methodological study. Archives of General Psychiatry 39, 879883.
Lord, F. M. (1980). Applications of Item Response Theory to Practical Testing Problems. Lawrence Erlbaum Associates: Hillsdale, NJ.
Lord, F. M. & Novick, M. R. (1968). Statistical Theories of Mental Test Scores. Addison-Wesley: Reading, MA.
McClish, D. & Quade, D. (1985). Improving estimates of prevalence by repeated testing. Biometrics 41, 8189.
Marneros, A. & Tsuang, M. T., eds. (1986). Schizoaffective Psychoses. Springer-Verlag: Berlin.
Martinez, M., Khlat, M., Leboyer, M. & Clerget-Darpoux, F. (1989). Performance of linkage analysis under misclassification error when the genetic model is unknown. Genetic Epidemiology 6, 253258.
Matthysse, S., Holzman, P. S. & Lange, K. (1986). The genetic transmission of schizophrenia: application of Mendelian latent structure analysis to eye tracking dysfunctions in schizophrenia and affective disorder. Journal of Psychiatric Research 20, 5776.
Meehl, P. & Rosen, A. (1955). Antecedent probability and the efficiency of psychometric signs, patterns, or cutting scores. Psychological Bulletin 52, 194216.
Nurnberger, J. I. Jr., Blehar, M. C., Kaufmann, C. A., York-Cooler, C., Simpson, S. G., Harkavy-Friedman, J., Severe, J. B., Malaspina, D., Reich, T., Miller, M., Bowman, E. S., DePaulo, J. R., Cloninger, C. R., Robinson, G., Moldin, S., Gershon, E. S., Maxwell, E., Guroff, J. J., Kirch, D., Wynne, D., Berg, K., Tsuang, M. T., Faraone, S. V., Pepple, J. R. & Ritz, A. L. (1994). Diagnostic interview for genetic studies. Rationale, unique features, and training. Archives of General Psychiatry 51, 849859.
Ott, J. (1991). Genetic linkage analysis under uncertain disease definition. In Banbury Report 33: Genetics and Biology of Alcoholism (ed. Cloninger, C. R. and Begleiter, H.), pp. 327331. Cold Spring Harbor Laboratory Press: Cold Spring Harbor, NY.
Politser, P. (1982). Reliability, decision rules, and the value of repeated tests. Medical Decision Making 2, 4769.
Quade, D., Lachenbruch, P. A., Whaley, F. S., McClish, D. K. & Haley, R. W. (1980). Effects of misclassifications on statistical inferences in epidemiology. American Journal of Epidemiology 111, 503515.
Rice, J. P., McDonald-Scott, P., Endicott, J., Coryell, W., Grove, W. M., Keller, M. B. & Altis, D. (1986). The stability of diagnosis with an application to bipolar II disorder. Psychiatry Research 19, 285296.
Rice, J. P., Endicott, J., Knesevich, M. A. & Rochberg, N. (1987). The estimation of diagnostic sensitivity using stability data: an application to major depressive disorder. Journal of Psychiatric Research 21, 337345.
Rice, J. P., Rochberg, N., Endicott, J., Lavori, P. W. & Miller, C. (1992). Stability of psychiatric diagnoses. An application to the affective disorders. Archives of General Psychiatry 49, 824830.
Robins, E. & Guze, S. B. (1970). Establishment of diagnostic validity in psychiatric illness: its application to schizophrenia. American Journal of Psychiatry 126, 983987.
Schulzer, M., Anderson, D. R. & Drance, S. M. (1991). Sensitivity and specificity of a diagnostic test determined by repeated observations in the absence of an external standard. Journal of Clinical Epidemiology 44, 11671179.
Spitzer, R. L., Endicott, J. & Robins, E. (1978). Research diagnostic criteria: rationale and reliability. Archives of General Psychiatry 35, 773782.
Spitznagel, E. L. & Helzer, J. E. (1985). A proposed solution to the base rate problem in the kappa statistic. Archives of General Psychiatry 42, 725728.
Tsuang, M. T., Gilbertson, M. W. & Faraone, S. V. (1991). The genetics of schizophrenia: current knowledge and future directions. Schizophrenia Research 4, 157171.
Tsuang, M. T., Faraone, S. V. & Lyons, M. J. (1993 a). Advances in psychiatric genetics. In International Review of Psychiatry, Volume I (ed. Costa e Silva, J. A., Nadelson, C. C., Andreasen, N. C. and Sato, M.), pp. 395440. American Psychiatric Press: Washington, DC.
Tsuang, M. T., Faraone, S. V. & Lyons, M. J. (1993 b). Identification of the phenotype in psychiatric genetics. European Archives of Psychiatry and Clinical Neuroscience 243, 131142.
Übersax, J. S. (1983). Structural analysis of diagnostic disagreements. Journal of Nervous and Mental Disease 171, 199206.
Williams, J. B. W., Gibbon, M., First, M. B., Spitzer, R. L., Davies, M., Borus, J., Howes, M. J., Kane, J., Poper, H. G., Rounsaville, B. & Wittchen, H. (1992). The structured clinical interview for DSM-III-R(SCID). II. Multisite Test–Retest Reliability. Archives of General Psychiatry 49, 630636.
Young, M. A. (1982/1983). Evaluating diagnostic criteria: a latent class paradigm. Journal of Psychiatric Research 17, 285296.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Psychological Medicine
  • ISSN: 0033-2917
  • EISSN: 1469-8978
  • URL: /core/journals/psychological-medicine
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed