Hostname: page-component-76fb5796d-r6qrq Total loading time: 0 Render date: 2024-04-29T07:30:21.971Z Has data issue: false hasContentIssue false
  • English
  • Français

Diagnostic Tests Distinguishing Good Tests From Bad and Even Ugly Ones

Published online by Cambridge University Press:  02 January 2015

David Birnbaum ,
Barry M. Farr  and
David E. Shapiro
Barry M. Farr*
Affiliation:
University of Virginia Health System, Charlottesville, Virginia, Harvard School of Public Health, Boston, Massachusetts
David E. Shapiro
Affiliation:
Department of Biostatistics, Harvard School of Public Health, Boston, Massachusetts
*
University of Virginia Health System, Box 800473, Charlottesville, VA 22908-0473
Get access Rights & Permissions [Opens in a new window]

Abstract

This article focuses on the selection and interpretation of diagnostic tests, emphasizing the importance of understanding how their mathematical parameters affect the information they provide in various settings. The utility and limitations of sensitivity, specificity, predictive value, and receiver operating characteristic (ROC) curves are discussed using catheter-related bloodstream infections as an example. ROC curves have been used for selecting optimal cutoff values for a positive result and for selecting among several alternative diagnostic tests. For example, 16 different tests have been proposed for diagnosis of catheter-related bloodstream infection; ROC analysis provides an effective way to determine which test offers the best overall performance.

Type
Statistics for Hospital Epidemiology
Information
Infection Control & Hospital Epidemiology , Volume 21 , Issue 4 , April 2000 , pp. 278 - 284
Copyright
Copyright © The Society for Healthcare Epidemiology of America 2000

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

1.Huang, Ti. Nei Ching Su, Wên; Veith, I, trans. The Yellow Emperor's Classic of Internal Medicine. Baltimore, MD: Williams & Wilkins; 1949.Google Scholar
2.Paracelsus. Seven Defensiones; Temkin, L, trans. In: Strauss, MB, ed. Familiar Medical Quotations. Boston, MA: Little Brown & Co; 1968:95.Google Scholar
3.Feinstein, AR. The chagrin factor and qualitative decision analysis. Arch Intern Med 1985;145:1257–1259.CrossRefGoogle ScholarPubMed
4.Fletcher, RH, Fletcher, SW, Wagner, EH. Diagnosis. Clinical Epidemiology: The Essentials. 2nd ed. Philadelphia, PA: Lippincott Williams & Wilkins; 1988:42–75.Google Scholar
5.Bartlett, J, Kaplan, J, eds. Bartlett's Familiar Quotations. Boston, MA: Little Brown & Co; 1992.Google Scholar
6.Birnbaum, D, Sheps, SB. Validation of new tests. Infect Control Hosp Epidemiol 1991;12:622–624.CrossRefGoogle ScholarPubMed
7.Begg, CB. Advances in statistical methodology for diagnostic medicine in the 1980's. Statistics in Medicine 1991;10:1887–1895.CrossRefGoogle Scholar
8.Littenberg, B, Moses, LE. Estimating diagnostic accuracy from multiple conflicting reports: a new meta-analytic method. Med Decis Making 1993;13:313–321.CrossRefGoogle ScholarPubMed
9.Metz, CE. ROC methodology in radiologic imaging. Invest Radiol 1986;21:720–733.CrossRefGoogle ScholarPubMed
10.Shapiro, DE. The interpretation of diagnostic tests. Stat Meth Med Res 1999;8:113–134.CrossRefGoogle ScholarPubMed
11.Irwig, L, Tosteson, A, Gatsonis, C, Lau, J, Colditz, G, Chalmers, T, et al. Guidelines for meta-analyses evaluating diagnostic tests. Ann Intern Med 1994;120:667–676.CrossRefGoogle ScholarPubMed
12.Shapiro, DE. Issues in combining independent estimates of the sensitivity and specificity of a diagnostic test. Acad Radiol 1995;2:S37–S47.Google ScholarPubMed
13.Moses, L, Shapiro, D, Littenberg, B. Combining independent studies of a diagnostic test into a summary ROC curve: data-analytic approaches and some additional considerations. Stat Med 1993;12:1293–1316.CrossRefGoogle ScholarPubMed
14.Bannerjee, SN, Emori, TG, Culver, DH. Secular trends in nosocomial primary bloodstream infections in the United States, 1980-1989. Am J Med 1991;91(suppl3B):86S–89S.CrossRefGoogle Scholar
15.Calfee, DP, Cage, EG, Farr, BM. Secular trends in nosocomial infections during the era of managed care. Presented at the Ninth Annual Scientific Meeting of the Society for Healthcare Epidemiology of America; April 18-20,1999; San Francisco, CA. Abstract.Google Scholar
16.Fridkin, SA, Pear, SM, Williamson, TH, Galgiani, JN, Jarvis, WR. The role of understaffing in central venous catheter-associated bloodstream infections. Infect Control Hosp Epidemiol 1996;17:150–158.Google ScholarPubMed
17.Farr, BM. Understaffing: a risk factor for infection in the era of downsizing? Infect Control Hosp Epidemiol 1996;17:147.CrossRefGoogle ScholarPubMed
18.Byers, KE, Anneski, CJ, Anglim, AM, Shank, DM, Walker, DW, Farr, BM. Central venous catheter utilization and complications a university hospital. Presented at the Seventh Annual Scientific Meeting of the Society for Healthcare Epidemiology of America; April 27-29, 1997; St. Louis, MO. Abstract 29.Google Scholar
19.Centers for Disease Control and Prevention. Teleconference: Antimicrobial use and resistance: solutions to the problem. 1998.Google Scholar
20.Byers, KE, Adal, KA, Anglim, AM, Farr, BM. Case fatality rate for catheter-related bloodstream infections (CRBSI): a meta-analysis. Presented at the Fifth Annual Meeting of the Society for Healthcare Epidemiology of America; San Diego, CA; April 2-4, 1995. Infect Control Hosp Epidemiol 1995;16(suppl 2):23. Abstract 23.Google Scholar
21.Pittet, D, Chuard, C, Rae, AC. Clinical diagnosis of central venous catheter line infections: a difficult job. Presented at the 31st Interscience Conference on Antimicrobial Agents and Chemotherapy; Chicago, IL; September 29-October 2,1991. Abstract 453.Google Scholar
22.Quilici, N, Audibert, G, Conroy, MC, Bollaert, P, Guillemin, F, Welfringer, P. Differential quantitative blood cultures in the diagnosis of catheter-related sepsis in intensive care units. Clin Infect Dis 1997;25:1066–1070.CrossRefGoogle ScholarPubMed
23.Brun-Bruisson, C, Abroug, F, Legrand, P, Huet, Y, Larabi, S, Rapin, M. Diagnosis of central venous catheter-related sepsis. Critical level of quantitative rip cultures. Arch Intern Med 1987;147:873–877.CrossRefGoogle Scholar
24.Sherertz, RJ, Raad, II, Belani, A, Koo, LC, Rand, KH, Pickett, DL, et al. Three-year experience with sonicated vascular catheter cultures in a clinical microbiology laboratory. J Clin Microbiol 1990;28:76–82.CrossRefGoogle Scholar
25.Cobb, DK, High, KP, Sawyer, RG, Sable, CA, Adams, RB, Lindley, DA, et al. A controlled trial of scheduled replacement of central venous and pulmonary-artery catheters. N Engl J Med 1992;327:1062–1068.CrossRefGoogle ScholarPubMed
26.Sitzmann, JV, Townsend, TR, Siler, MC, Bartlett, JG. Septic and technical complications of central venous catheterization: a prospective study of 200 consecutive patients. Ann Surg 1985;202:766–770.CrossRefGoogle ScholarPubMed
27.Maki, DG. Nosocomial bacteremia: a epidemiologic overview. Am J Med 1981;70:719–732.CrossRefGoogle ScholarPubMed
28.Bozzetti, F, Terno, G, Camerini, E, Baticci, F, Scarpa, D, Pupa, A. Pathogenesis and predictability of central venous catheter sepsis. Surgery 1981;91:383–389.Google Scholar
29.Ryan, JAJ, Abel, RM, Abbott, WM. Catheter complications in total parenteral nutrition: a prospective study of 200 consecutive patients. N Engl J Med 1974;290:757–761.CrossRefGoogle ScholarPubMed
30.Siegman-Igra, Y, Anglim, AM, Shapiro, D, Adal, KA, Strain, B, Farr, BM. Diagnosis of vascular catheter-related bloodstream infection: a metaanalysis. J Clin Microbiol 1997;35:928–936.CrossRefGoogle ScholarPubMed
31.Kristinsson, K. Diagnosis of catheter-related infections. In: Seifert, H, Jansen, B, Farr, BM, eds. Catheter-Related Infections. New York, NY: Marcel Dekker, Inc; 1997:31–57.Google Scholar
32.Bjornson, H, Colley, R, Bower, R, Duty, V, Schwartz-Fulton, J, Fisher, J. Association between microorganism growth at the catheter insertion site and colonization of the catheter in patients receiving total parenteral nutrition. Surgery 1982;92:720–727.Google ScholarPubMed
33.Bozzetti, F, Terno, G, Bonfanti, G, Gallus, G. Blood culture as a guide for the diagnosis of central venous catheter sepsis. JPEN 1984;8:396–398.CrossRefGoogle ScholarPubMed
34.Brun-Bruisson, C, Abrouk, F, Legrand, P, Huet, Y, Larabi, S, Rapin, M. Diagnosis of central venous catheter-related sepsis: critical level of quantitative tip cultures. Arch Intern Med 1987;147:873–877.CrossRefGoogle Scholar
35.Cercenado, E, Ena, J, Rodriguez-Creixems, M, Romero, I, Bouzza, E. A conservative procedure for the diagnosis of catheter-related infections. Arch Intern Med 1990;150:1417–1420.CrossRefGoogle ScholarPubMed
36.Cleri, D, Corrado, M, Seligman, S. Quantitative culture of intravenous catheters and other intravascular inserts. J Infect Dis 1980;141:781–786.CrossRefGoogle ScholarPubMed
37.Gutierrez, J, Leon, C, Matamoros, R, Noagles, C, Martin, E. Catheter-related bacteremia and fungemia. Reliability of two methods for catheter culture. Diagn Microbiol Infect Dis 1992;15:575–578.CrossRefGoogle ScholarPubMed
38.Maki, D, Jarrett, R, Sarafin, H. A semiquantitative culture method for identification of catheter-related infection in the burn patient. Journal Surgical Residents 1977;22:513–520.CrossRefGoogle ScholarPubMed
39.Maki, D, Weise, C, Sarafin, H. A semiquantitative culture method for identifying intravenous-catheter-related infection. N Engl J Med 1977;296: 1305–1309.CrossRefGoogle ScholarPubMed
40.Raad, I, Sabbagh, M, Rand, K, Sheretz, R. Quantitative tip culture methods and the diagnosis of central venous catheter-related infections. Diagn Microbiol Infect Dis 1992;15:13–20.CrossRefGoogle ScholarPubMed
41.Raucher, HS, Hyatt, AC, Barzilai, A, Harris, MB, Weiner, MA, LeLeiko, NS, et al. Quantitative blood cultures in the evaluation of septicemia in children with Broviac catheters. J Pediatr 1984;104:29–33.CrossRefGoogle ScholarPubMed
42.Sheretz, R, Heard, S, Raad, I. Diagnosis of triple-lumen catheter infection: comparison of roll plate, sonication, and flushing methodologies. J Clin Microbiol 1997;35:641–646.CrossRefGoogle Scholar
43.Raad, I, Costerton, W, Sabharwal, U, Sacilowski, M, Anaissie, E, Bodey, GP. Ultrastructural analysis of indwelling vascular catheters: a quantitative relationship between luminal colonization and duration of placement. J Infect Dis 1993;168:400–407.CrossRefGoogle ScholarPubMed
44.Hampton, A, Sheretz, R. Vascular-access infections in hospitalized patients. Surg Clin North Am 1988;68:57–71.CrossRefGoogle ScholarPubMed
45.Collignon, P, Chan, R, Munro, R. Is semiquantitative culture of central vein catheter tips useful in the diagnosis of catheter-associated bacteremia? J Clin Microbiol 1986;24:532–535.CrossRefGoogle ScholarPubMed
46.Collignon, P, Chan, R, Munro, R. Rapid diagnosis of intravascular catheter-related sepsis. Arch Intern Med 1987;147:1603–1612.CrossRefGoogle ScholarPubMed
47.Moyer, M, Edwards, L, Farley, L. Comparative culture methods on 101 intravenous catheters. Routine, semiquantitative, and blood cultures. Arch Intern Med 1983;143:66–69.CrossRefGoogle ScholarPubMed
48.Nahass, R, Weinstein, M. Qualitative intravascular catheter tip cultures do not predict catheter-related bacteremia. Diagn Microbiol Infect Dis 1990;13:223–226.CrossRefGoogle Scholar
49.Capdevila, J, Planes, A, Palomar, M, Grasser, I, Almirante, B, Pahissa, A, et al. Value of differential quantitative blood cultures in the diagnosis of catheter-related sepsis. Eur J Clin Microbiol Dis 1992;11:403–407.CrossRefGoogle ScholarPubMed
50.Douard, M, Marie, O, Clementi, E, Arlet, G, Jacob, L, Ardoin, C, et al. Does a negative catheter tip culture exclude the diagnosis of catheter-related bacteremia? Presented at the 31st Interscience Conference on Antimicrobial Agents and Chemotherapy; Chicago, IL; September 29- October 2,1991. Abstract 455.Google Scholar
51.Paya, C, Guerra, L, Marsh, H, Farnell, M, Washington, J II, Thompson, R. Limited usefulness of quantitative culture of blood drawn through the device for diagnosis of intravascular-device-related bacteremia. J Clin Microbiol 1989;27:1431.CrossRefGoogle ScholarPubMed
52.Snydman, D, Murray, S, Kornfeld, S, Majka, J, Ellis, C. Total parenteral nutrition related infections: prospective epidemiologic study using semiquantitative methods. Am J Med 1982;73:695–699.CrossRefGoogle ScholarPubMed
53.Breslow, NE, Day, NE. The analysis of case-control studies. Statistical Methods in Cancer Research. Lyon, France: IARC; 1980:142.Google Scholar
54.Kardaun, JWPF, Kardaun, OUWF. Comparative diagnostic performance of three radiological procedures for the detection of lumbar disk herniation. Methods Inf Med 1990;29:12–22.Google ScholarPubMed
55.Widmer, AF, Nettleman, M, Flint, K, Wenzel, RP. The clinical impact of culturing central venous catheters. A prospective study. Arch Intern Med 1992;152:1299–1302.CrossRefGoogle ScholarPubMed
56.Rogers, MS, Oppenheim, BA. The use of continuous monitoring blood culture systems in the diagnosis of catheter related sepsis. J Clin Pathol 1998;51:635–637.CrossRefGoogle ScholarPubMed
57.Blot, F, Nitenberg, G, Chachaty, E, Raynard, B, Germann, N, Antoun, S, et al. Diagnosis of catheter-related bacteremia: a prospective comparison of the time to positivity of central vs peripheral blood cultures. Lancet. In press.Google Scholar
58.Mermel, LA, Josephson, L, Binns, L. Diagnosis of catheter-related bloodstream infection (CRBSI) by differential growth rates of catheter-drawn and percutaneously drawn blood cultures. Presented at the 38th Interscience Conference on Antimicrobial Agents and Chemotherapy; San Diego, CA; September 24-27,1998. Abstract K5.Google Scholar
59.Blot, F, Schmidt, E, Tancrede, C, Leclerq, B, Laplanche, A. Earlier positivity of central-venous-versus peripheral-blood cultures is highly predictive of catheter-related sepsis. J Clin Microbiol 1998;36:105–109.CrossRefGoogle ScholarPubMed
60.Capdevila, J, Segarra, A, Planes, A, Gasser, I, Gavalda, J, Pahissa, A. Long term follow-up of patients with catheter related sepsis (CRS) treated without catheter removal. Presented at the 35th Interscience Conference on Antimicrobial Agents and Chemotherapy; San Francisco, CA; September 17-20,1995. Abstract J3.Google Scholar
61.Raad, I, Davis, S, Khan, A, Tarrand, J, Elting, L, Bodey, GP. Impact of central venous catheter removal on the recurrence of catheter-related coagulase-negative staphylococcal bacteremia. Infect Control Hosp Epidemiol 1992;13:215–221.CrossRefGoogle ScholarPubMed
62.Krzywda, EA, Gotoff, RA, Andris, DA, Marciniak, TF, Edmiston, CE, Quebbeman, EJ. Antibiotic lock treatment (ALT): impact on catheter salvage and cost savings. Presented at the 35th Interscience Conference on Antimicrobial Agents and Chemotherapy; San Francisco, CA; September 17-20,1995. Abstract J4.Google Scholar
63.Benezra, D, Kiehn, TE, Gold, J, Brown, AE, Turnbull, A, Armstrong, D. Prospective study of infections in indwelling central venous catheters using quantitative blood cultures. Am J Med 1988;85:495–498.CrossRefGoogle ScholarPubMed
64.Dugdale, DC, Ramsey, PG. Staphylococcus aureus bacteremia in patients with Hickman catheters. Am J Med 1990;89:137–141.CrossRefGoogle ScholarPubMed
65.Kite, P, Dobbins, BM, Wilcox, MH, Fawley, WN, Kindon, AJ, Thomas, D, et al. Evaluation of a novel endoluminal brush method for in situ diagnosis of catheter related sepsis. J Clin Pathol 1997;50:278–282.CrossRefGoogle ScholarPubMed
66.Pettigrew, RA, Lang, SDR, Haydock, DA, Parry, BR, Bremner, DA, Hill, GL. Catheter-related sepsis in patients on intravenous nutrition: a prospective study of quantitative catheter cultures and guidewire changes for suspected sepsis. British Journal of Surgery 1985;72:52–55.CrossRefGoogle ScholarPubMed
67.Armstrong, CW, Mayhall, CG, Miller, KB, Newsome, HH Jr, Sugerman, HJ, Dalton, HP, et al. Prospective study of catheter replacement and other risk factors for infection of hyperalimentation catheters. J Infect Dis 1986;154:808–816.CrossRefGoogle ScholarPubMed
68.Hagley, MT, Martin, B, Gast, P, Traeger, SM. Infectious and mechanical complications of central venous catheters placed by percutaneous venipuncture and over guidewires. Crit Care Med 1992;20:1426–1430.CrossRefGoogle ScholarPubMed
69.Martinez, E, Mensa, J, Rovira, M, Martinez, JA, Marcos, A, Almela, M, et al. Central venous catheter exchange by guidewire for treatment of catheter-related bacteremia in patients undergoing BMT or intensive chemotherapy. Bone Marrow Transplantation 1999;23:41–44.CrossRefGoogle ScholarPubMed
70.Pittet, D. Nosocomial bloodstream infections. In: Wenzel, RP, ed. Prevention and Control of Nosocomial Infections. 3rd ed. Baltimore, MD: Williams & Wilkins; 1997:711–769.Google Scholar
71.Rello, J. Impact of nosocomial infections on outcome: myths and evidence. Infect Control Hosp Epidemiol 1999;20:392–394.CrossRefGoogle ScholarPubMed
72.Soufir, L, Timsit, J, Mahe, C, Carlet, J, Regnier, B, Chevret, S. Attributable morbidity and mortality of catheter-related septicemia in critically ill patients: a matched, risk-adjusted, cohort study. Infect Control Hosp Epidemiol 1999;20:396–401.CrossRefGoogle ScholarPubMed