Skip to main content Accessibility help
×
Home

A methodological comparison of risk scores versus decision trees for predicting drug-resistant infections: A case study using extended-spectrum beta-lactamase (ESBL) bacteremia

  • Katherine E. Goodman (a1), Justin Lessler (a1), Anthony D. Harris (a2), Aaron M. Milstone (a1) (a3) and Pranita D. Tamma (a3)...

Abstract

Background:

Timely identification of multidrug-resistant gram-negative infections remains an epidemiological challenge. Statistical models for predicting drug resistance can offer utility where rapid diagnostics are unavailable or resource-impractical. Logistic regression–derived risk scores are common in the healthcare epidemiology literature. Machine learning–derived decision trees are an alternative approach for developing decision support tools. Our group previously reported on a decision tree for predicting ESBL bloodstream infections. Our objective in the current study was to develop a risk score from the same ESBL dataset to compare these 2 methods and to offer general guiding principles for using each approach.

Methods:

Using a dataset of 1,288 patients with Escherichia coli or Klebsiella spp bacteremia, we generated a risk score to predict the likelihood that a bacteremic patient was infected with an ESBL-producer. We evaluated discrimination (original and cross-validated models) using receiver operating characteristic curves and C statistics. We compared risk score and decision tree performance, and we reviewed their practical and methodological attributes.

Results:

In total, 194 patients (15%) were infected with ESBL-producing bacteremia. The clinical risk score included 14 variables, compared to the 5 decision-tree variables. The positive and negative predictive values of the risk score and decision tree were similar (>90%), but the C statistic of the risk score (0.87) was 10% higher.

Conclusions:

A decision tree and risk score performed similarly for predicting ESBL infection. The decision tree was more user-friendly, with fewer variables for the end user, whereas the risk score offered higher discrimination and greater flexibility for adjusting sensitivity and specificity.

Copyright

Corresponding author

Author for correspondence: Katherine E. Goodman, Email: kgoodma7@jhu.edu and Pranita D. Tamma, Email: ptamma1@jhmi.edu

References

Hide All
1.Livorsi, DJ, Chorazy, ML, Schweizer, ML, et al. A systematic review of the epidemiology of carbapenem-resistant Enterobacteriaceae in the United States. Antimicrob Resist Infect Control. 2018;7:55.
2.McDanel, J, Schweizer, M, Crabb, V, et al. Incidence of extended-spectrum beta-lactamase (ESBL)-producing Escherichia coli and Klebsiella infections in the United States: a systematic literature review. Infect Control Hosp Epidemiol 2017;38:12091215.
3.Micek, ST, Hampton, N, Kollef, M. Risk factors and outcomes for ineffective empiric treatment of sepsis caused by gram-negative pathogens: stratification by onset of infection. Antimicrob Agents Chemother 2018;62(1): e01577–17.
4.Zhang, D, Micek, ST, Kollef, MH. Time to appropriate antibiotic therapy is an independent determinant of postinfection ICU and hospital lengths of stay in patients with sepsis. Crit Care Med 2015;43:21332140.
5.Clinical and Laboratory Standards Institute. Performance Standards for Antimicrobial Susceptibility Testing, 28th ed. Supplement M100S. Wayne, PA: CLSI; 2018.
6.Ledeboer, NA, Lopansri, BK, Dhiman, N, et al. Identification of gram-negative bacteria and genetic resistance determinants from positive blood culture broths by use of the verigene gram-negative blood culture multiplex microarray-based molecular assay. J Clin Microbiol 2015;53:24602472.
7.Ward, C, Stocker, K, Begum, J, Wade, P, Ebrahimsa, U, Goldenberg, SD. Performance evaluation of the Verigene (Nanosphere) and FilmArray (BioFire) molecular assays for identification of causative organisms in bacterial bloodstream infections. Eur J Clin Microbiol Infect Dis 2015;34:487496.
8.Goodman, KE, Lessler, J, Cosgrove, SE, et al. A clinical decision tree to predict whether a bacteremic patient is infected with an extended-spectrum beta-lactamase–producing organism. Clin Infect Dis 2016;63:896903.10.1093/cid/ciw425
9.Antimicrobial resistant phenotype definitions. Centers for Disease Control and Prevention webstie. https://www.cdc.gov/nhsn/pdfs/ps-analysis-resources/phenotype_definitions.pdf. Published 2016. Accessed January 15, 2019.
10.Kantele, A, Laaveri, T, Mero, S, et al. Antimicrobials increase travelers’ risk of colonization by extended-spectrum beta-lactamase–producing Enterobacteriaceae. Clin Infect Dis 2015;60:837846.
11.Ostholm-Balkhed, A, Tarnberg, M, Nilsson, M, et al. Travel-associated faecal colonization with ESBL-producing Enterobacteriaceae: incidence and risk factors. J Antimicrob Chemother 2013;68:21442153.
12.Duda, RO, Hart, PE, Stork, DG. Pattern Classification, 2nd ed. New York: Wiley-Interscience; 2001.
13.Breiman, L, Friedman, J, Stone, C, Olshen, R. Classification and Regression Trees. Boca Raton, FL: CRC/Chapman & Hall; 1984.
14.Tseng, WP, Chen, YC, Yang, BJ, et al. Predicting multidrug-resistant gram-negative bacterial colonization and associated infection on hospital admission. Infect Control Hosp Epidemiol 2017;38:12161225.
15.Rottier, WC, van Werkhoven, CH, Bamberg, YRP, et al. Development of diagnostic prediction tools for bacteraemia caused by third-generation cephalosporin-resistant enterobacteria in suspected bacterial infections: a nested case-control study. Clin Microbiol Infect 2018;24: 13151321.
16.Augustine, MR, Testerman, TL, Justo, JA, et al. Clinical risk score for prediction of extended-spectrum beta-lactamase–producing enterobacteriaceae in bloodstream isolates. Infect Control Hosp Epidemiol 2017;38:266272.
17.Kawamoto, K, Houlihan, CA, Balas, EA, Lobach, DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005;330:765.
18.Peduzzi, P, Concato, J, Kemper, E, Holford, TR, Feinstein, AR. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 1996;49:13731379.
19.Vittinghoff, E, McCulloch, CE. Relaxing the rule of ten events per variable in logistic and Cox regression. Am J Epidemiol 2007;165:710718.
20.Dietterich, T. Overfitting and undercomputing in machine learning. ACM Comput Surv 1995;27:326327.
21.Chen, X, Ishwaran, H. Pathway hunting by random survival forests. Bioinformatics 2013;29:99105.10.1093/bioinformatics/bts643
22.Strobl, C, Malley, J, Tutz, G. An introduction to recursive partitioning: rationale, application, and characteristics of classification and regression trees, bagging, and random forests. Psychol Method 2009;14:323348.
23.Drummond, C, Holte, RC. Exploiting the cost (in)sensitivity of decision tree splitting criteria. Proceedings of the Seventeenth International Conference on Machine Learning. Stanford, CA; 2000.
24.Tibshirani, R. Regression shrinkage and selection via the lasso. J Roy Stat Soc B 1996;58:267288.
25.van der Laan, MJ, Polley, EC, Hubbard, AE. Super learner. Stat Appl Genet Mol Biol 2007;6:article25. Epub 2007 Sep 16.
26.Song, YY, Lu, Y. Decision tree methods: applications for classification and prediction. Shanghai Arch Psychiatr 2015;27:130135.
27.Akaike, H. A new look at the statistical model identification. IEEE Trans Automat Control 1974;19:716723.

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed