Introduction
Clostridioides difficile is a Gram positive, spore forming, toxin producing bacterium which can colonize and infect the human intestinal track causing a robust immune response. Reference Knight, Imwattana and Collins1 Unlike most other infectious diseases, C. difficile infection (CDI) tends to relapse and re-infect despite antibiotic treatment and remission of symptoms. C. difficile spores are capable of surviving treatment, resulting in persistent carriage, serving as a nidus for recurrent infection. Most recurrent CDI episodes occur within 2–8 weeks of the initial infection and are due to the previous strain. Reference Naz and Petri2 Newer anti-CDI therapies (ie, fidaxomicin, Reference Cornely, Crook and Esposito3 bezlotoxumab, Reference Wilcox, Gerding and Poxton4 and fecal microbiota transplant Reference Kelly, Khoruts and Staley5 ) effectively prevent recurrent CDI and as of April 2023, 6 the first orally administered fecal microbiota product was approved for the prevention of recurrent CDI by the US Food and Drug Administration. However, newer therapies to prevent recurrent CDI have historically been underutilized, Reference Jiang, Sarpong, Sears and Obi7 despite adoption by recent CDI consensus guidelines. Reference Johnson, Lavergne and Skinner8 This is likely due to two major issues with CDI treatments: first, these treatments have a high cost and logistical challenges, limiting their use to patients believed to be most at risk of recurrence, Reference Lee, Lim, Bloom, Moore, Chung and Marzella9,Reference Cruz10 ; second, determining which patients are likely to recur is difficult without a widely accepted risk prediction model for recurrent C. difficile infection.
Several published outcome models (Table 1) for risk stratifying patients with CDI for developing future recurrent infection have been developed retrospectively or by using previous clinical trial data. Few recurrent CDI models have been externally validated and limited evidence suggest that they generalize poorly between centers. In addition, recently updated clinical management guidelines for C. difficile infection mention specific risk factors for recurrence, however, the performance of these features together as a risk stratification tool has not been externally evaluated. Reference Johnson, Lavergne and Skinner8 We were also interested in comparing C. difficile PCR cycle threshold (as an inverse measure of fecal organism burden), which we recently showed may be a potentially useful biomarker for recurrent infection Reference Madden, Rigo and Boone21 alongside existing clinical models. Here, we validate and compare the relative performance of the leading recurrence risk models available for hospitalized cases of C. difficile infection.
Clinical risk models for predicting recurrent C. difficile infection
C. difficile infection (CDI), methicillin-resistant S. aureus (MRSA), proton pump inhibitor (PPI), white blood cell (WBC)
Methods
Study population
A previously described Reference Madden, Enfield and Sifri22,Reference Madden, Petri, Costa, Warren, Ma and Sifri23 retrospective cohort of hospitalized adult patients with C. difficile infection at University of Virginia Medical Center (a 645-bed, tertiary care academic hospital) was used for this analysis. Hospitalized CDI cases were enrolled with a positive C. difficile polymerase chain reaction (PCR; GeneXpert®; Cepheid, Sunnyvale, CA) result and anti-C. difficile treatment. Children <18 years and episodes that were not treated with anti-C. difficile therapy (oral vancomycin, fidaxomicin, and/or IV/oral metronidazole) were excluded. Recurrent C. difficile infection was defined as the presence of a repeat positive PCR test available within 11–60 days post-infection or symptom relapse requiring retreatment/initial treatment extension for CDI within 60 days of index diagnosis. Patients without electronically available follow-up positive testing underwent additional clinician chart review (E.L., including clinician notes) to determine reports of positive C. difficile testing from outside laboratories and/or CDI symptom recurrence with retreatment/initial treatment extension. This study received approval from the University of Virginia Institutional Review Board (#20082).
Data collection/risk score calculation
To identify risk models for recurrent CDI, a literature search using medical literature databases (ie, PubMed, Google Scholar) and the bibliographies of relevant manuscripts was conducted. Manuscripts were evaluated for relevance; specifically, studies which generated clinically relevant tools for predicting recurrent CDI were of interest. So that each model could be compared in later analyses, the models had to include parameters that could be reliably gathered or imputed from the electronic medical record, and >3 ordinal scores to fit to an ROC curve. Criteria used to exclude studies from further analysis included highly specific study populations (ie, children, transplant patients), single binary predictors of recurrence, multivariable logistic regression models without a clinically applicable prediction tool (ie, integer-based scoring or similar systems by which a clinician could easily estimate relative risk), use of non-clinically accessible features, and meta-analysis and reviews. 7 models were identified which fulfilled the criteria. Reference Cobo, Merino and Martínez11,Reference Eyre, Walker and Wyllie13–Reference Zilberberg, Reske, Olsen, Yan and Dubberke18 Of these, 2 models were excluded due to inability to reproduce with our cohort (ie, Hu et al model included Horn Index, a subjective measure which was not among the cohort’s available features) Reference Hu, Katchar and Kyne17 or a lack of sufficient details for a scoring system that could be easily imputed from the electronic medical record (ie, Zilberberg et al). Reference Zilberberg, Reske, Olsen, Yan and Dubberke18 Two models, Cobo and D’Agostino, utilized definitions of recurrence (4 months and 28 days, respectively) than other models. In this study, 60 days was utilized for all models to remain consistent in analysis. In addition to the identified models, the previously established ATLAS score for predicting severe CDI was included as a control for models specific to recurrence vs general disease severity (Table 1). Reference Miller, Louie and Mullane20
Clinical data were gathered using the University of Virginia Clinical Data Warehouse, a database containing billing/coding, clinical, pharmacy, and laboratory data. Baseline features were defined as the closest available measurement within ±48 hours of C. difficile infection diagnosis (defined by the date/time of index positive C. difficile PCR specimen collection). For cases with multiple available laboratory measurements, maximum white blood cell count (WBC), creatinine, and minimum albumin measurements were used. For the Eyre model, Reference Zilberberg, Reske, Olsen, Yan and Dubberke18 all patients received 1 point for ≥3 unformed stools/day as this was a criterion for testing based on UVA diagnostic stewardship practices. Reference Madden, Mesner and Cox24 Past gastroenterology admission and C-reactive protein measurements (Eyre score) and presence/absence ≥10 unformed bowel movements/day (D’Agostino) could not be reliably collected. These were imputed as 0 added points when missing. For the IDSA/SHEA Recurrence risk stratification, one point was empirically assigned for each risk factor for recurrent CDI from the Updated 2017 IDSA/SHEA Guidelines. IDSA/SHEA Guidelines include age ≥65 years, a recurrent CDI episode within the last 6 months, immunocompromised host, and severe CDI on presentation which included the measures of white blood cell count ≥15,000 cells/ml, serum creatinine level >1.5mg/dl, or signs of fulminancy (hypotension, shock, ileus, or megacolon). Reference Johnson, Lavergne and Skinner8,Reference McDonald, Gerding and Johnson25
An immunocompromised host was defined as active (administered during hospitalization) receipt of immunosuppressant medications (≥60 mg oral daily prednisone or equivalent systemic corticosteroid, azathioprine, rapamycin derivatives, cyclosporine, tacrolimus, or mycophenolate) and/or chemotherapy (Supplementary Table 1). Reference Escobar, Baker and Kipnis19 Antimotility medications were defined as loperamide, diphenoxylate, oral opium, or bismuth subsalicylate (receipt within 7 days preceding CDI diagnosis).
Data analysis
The area under the receiver operating characteristic curve (AUROC) for each model were calculated from these score-specific diagnostic test summary indices. The Youden Index (sensitivity + specificity -1) was calculated as an overall measure of diagnostic effectiveness and as one method to identify optimal cutoffs that balance sensitivity and specificity.
DeLong’s test of variance was used to calculate two-sided statistical comparisons of the highest performing model AUROC against each of the others. This was done using correlated AUROC curves. The significance of the AUROC was calculated using only using the patients with complete data for both models being compared. Analyses were performed using statistical software R, version 4.2.3 (R Core Team, Vienna, Austria) and R packages: dpylr, Reference Wickham, Francois, Henry and Muller26 comorbidity, Reference Gutiérrez-Sacristán, Bravo, Giannoula, Mayer, Sanz and Furlong27 ROCit, Reference Khan28 pROC, Reference Robin, Turck and Hainard29 and PRROC. Reference Keilwagen, Grosse and Grau30
Results
Study population
1,519 hospitalized cases of C. difficile infection among 1,302 individuals were identified between January 2014 and April 2021 with available PCR cycle threshold, white blood cell count, and creatinine measurements to calculate risk scores. Baseline characteristics are shown in Table 2.
Study population baseline characteristics
Recurrent infection
157 recurrent CDI (rCDI) events were identified based on a repeat positive C. difficile test alone, using the UVA Health laboratory records. Clinical chart review performed for the remaining 1,362 cases found an additional 50 rCDI events with repeat positive C. difficile tests (either from the Care EverywhereTM electronic medical record-sharing feature in EpicTM or textually reported from an outside laboratory within clinician notes), and 138 rCDI events identified based on symptom relapse following anti-CDI treatment requiring retreatment without retesting. No rCDI cases were identified with negative PCR testing. In total, 345/1,519 (22.7%) patients were found to have recurrent CDI within 60 days.
1,211/1,519 (79.7%) of patients had at least one IDSA-defined risk factor for recurrent infection, all of which (recurrent CDI episode within the last 6 months, immunocompromised host, severe CDI on presentation) except one (age ≥65) was more likely among cases with subsequent recurrence. PCR Cycle threshold (Ct) among patients who went on to develop recurrent infection was lower (ie, higher stool organism burden) compared to patients without recurrence (mean PCR Ct 25.7 recurrence vs 26.8 no recurrence; t-test P = .0003). A frequency table of score distributions is shown in Table 3 and Supplementary Table 2.
Frequency table for recurrent CDI score distributions
Top: The number of patients were stratified by either numerical, integer-based scores, probability of recurrence according to the D’Agostino Model (in bold), or cycles for each of the recurrence models. Bottom: The mean of value for each model in the populations with no recurrence, recurrence, or overall population. Supplementary Table 2 includes the median.
Clinical prediction tools
AUROCs predicting recurrence of all tools are shown in Figure 1 for all patients (1A), patients with no previous recurrence (1B), and patients with 1 or more previous recurrent episodes (1C). An AUROC of 0.500 represents the prediction ability of random guessing. Each model performed poorly in this validation with the best performing model, Reveles et al, having an AUROC of 0.523, 95% CI [0.490, 0.555] when all occurrences were considered. The IDSA guideline recurrence risk criteria (AUROC of 0.520) performed as well as the trained models and 75/345 (21.7%) of patients who developed recurrent CDI had no IDSA-defined recurrence risk factors. The ATLAS score, designed to predict severe outcomes and not recurrence, had a low AUROC of 0.494. The two models which had been previously validated, Cobo et al and Larrainzar-Coghen et al, performed worse in our validation than in the derivation cohort but similar to previous validation study (Table 1).
Receiver operator curves with area under the receiver operating characteristic curve (AUROC) for C. difficile risk models with the full cohort (1,519 cases) (A), patients with no previous recurrence (1,230 cases) (B), and patients with 1 or more previous recurrent episodes (289 cases) (C). Note: Data was unavailable for some patients. See Table 3 and Supplementary Table 2 for the stratification of the data including the number of missing data points for each model.
PCR Ct had a higher AUROC (0.564) compared to any of the clinical tools, including Reveles et al (DeLong’s test of AUROC difference P = .082), and IDSA (DeLong’s test of AUROC difference P = .052), but AUROC differences were not statistically significant. Median PCR Ct was 25.3 and the positive/negative predictive values at the 1st, 2nd, and 3rd quartile cutoffs (<22.1, <25.3, <30.8) were 0.29/0.80, 0.26/0.82 and 0.23/0.82, respectively. Most tools performed better when only the likelihood of the first recurrence was evaluated (Figure 1B). However, all tools except PCR Ct (AUROC 0.591) preformed worse when only patients with at least one previous recurrence were considered (Figure 1C). Youden index (which maximizes specificity plus sensitivity) curves generated for all cases for each score are shown in Figure 2.
Youden Indices for C. difficile Severity Score Cut-offs. Youden Index is equal to 0 for tests with poor diagnostic accuracy, equal to 1 for a perfect test, and assigns equal weight to sensitivity and specificity. The Youden Index and the ideal cutoff may not apply to other patient populations. Reference Schisterman, Perkins, Liu and Bondell31
Discussion
Accurately anticipating risk is a crucial step towards improving access and stewardship of effective new therapies to prevent recurrent CDI, however, predicting future recurrence is a difficult problem. We demonstrate that existing clinical risk scores generalize poorly among a large cohort of hospitalized patients with C. difficile infection, especially early in the disease course. The rudimentary methods of generating a scoring systems used by these tools (ie, rounding logistic regression coefficients to integers) may have been inaccurate due to model bias, however, Escobar et al strove to generate more robust models using machine learning techniques and concluded that neither existing models nor their own model could be trained to accurately predict recurrent C. difficile infection in test patients (maximum achievable out-of-sample validation AUC 0.605). Reference Escobar, Baker and Kipnis19 Other factors to explain poor generalizability include significant C. difficile strain variation Reference Walker, Eyre and Wyllie32 or population-level differences (ie, location, socio-economic status, cohort recruitment, etc) between institutions that may influence recurrence risk or unmeasured putative features such as the delayed anti-toxin humoral immune response. Reference Gupta, Mehta and Dubberke33
Established risk factors for recurrent infection per recent consensus management guidelines are of marginal value for predicting recurrent C. difficile infection, with similar performance to other curated models. Discrepancy between reports of advanced age ≥65 as a positive risk factor for recurrent infection may be due to the competing risk of death within 60 days, which was two times higher among older patients ≥65 years in our cohort (139/685 (20%) patients ≥65 years vs 81/834 (9.7%) <65 years).
A significant hurdle in developing a generalizable and robust prediction tool for clinicians is the inability for the generated models to accurately predict outside of the data on which they were trained. All tools evaluated in this study performed more poorly than in the study in which they were derived. Similarly, when Rossen et al Reference van Rossen, van Dijk, Heymans, Dekkers, Vandenbroucke-Grauls and van Beurden12 validated the Cobo and Larrainzar-Coghen models, they observed lower AUROCs, with each model performing worse than random chance. (Table 1) Escobar et al Reference Escobar, Baker and Kipnis19 observed a similar phenomenon in the models which they generated and in a previously generated model. This suggests that prediction tools have been overfitted to the data on which they were trained. A deep neural network with regularization may help to optimize the bias-variance tradeoff while incorporating a higher dimension of inputs to improve prediction. However, it is also possible that the task of accurately predicting future recurrence using clinical observations at the time of index CDI diagnosis is too complex or simply not feasible. Simple measures, such PCR Ct, may be more universally predictive throughout patient populations.
This analysis was intended as a practical assessment of existing tools implemented electronically to predict rCDI and there are important limitations to this analysis. rCDI tools were not validated using the exact same rCDI time frame definitions to which they were trained but, for feasibility reasons, a single rCDI definition was used. For example, the Eyre tool (trained to predict rCDI by 4 months) may perform better predicting late rCDI or relapse. In addition, several potentially important clinical features were omitted from some models because they could not be collected electronically (eg, stool frequency). C. difficile misdiagnosis (ie, colonization versus true infection) is a substantial issue in CDI, especially in the context of PCR-based testing and may have confounded both existing tools and our validations. Reference Polage, Gyorke and Kennedy34 Recurrent C. difficile infection is typically confirmed by repeat testing, which may not be reported in the same electronic medical record or may not be feasible in favor of expediting re-initiation of anti-C. difficile treatment (eg, in outpatient settings). To avoid undercapturing outcome events, we defined recurrent CDI broadly to include cases without retesting if symptoms relapsed requiring a new round of anti-C. difficile therapy. While our rCDI may have over captured some cases, we believe a symptom-based definition is a pragmatic approach to identify clinically relevant rCDI retrospectively, especially considering that PCR testing often remains positive regardless of rCDI. Reference Saha, Yadav, Pardi, Patel, Khanna and Pardi35 Our resulting rate of recurrence (22.7%) approximates rates from prospective trials (∼25%). Reference McDonald, Gerding and Johnson25 40% of retrospective rCDI events occurred without retesting at the same health system laboratory, underscoring potential flaws in retrospective studies that use test-based rCDI definitions (and thus likely artifactually low rCDI rates) and which could help explain the poor generalizability and performance of existing rCDI tools.
A better, validated prediction tool for recurrence would be helpful to individualize therapeutic approaches to prevent recurrent infection at the time of diagnosis (eg, prioritize fidaxomicin treatment and referral for bezlotoxumab in patients with highest risk for recurrence). Interestingly, of all tools evaluated in this study, PCR Ct had the highest AUROC, similar to at least 4 other studies demonstrating that low CT and/or stool toxin concentration predict rCDI. Reference Alonso, Kelly and Garey36–Reference Sandora, Kociolek and Williams39 However, it should be noted that AUROC comparisons between categorical and quasi-continuous measures may not be valid given that scale discretization (of categorical risk scores) may reduce the precision of ROC measurements. Reference Wagner, Beiden and Metz40 While not a very useful standalone predictor, PCR Ct in a more complex model could increase the reliability of predicting recurrent CDI. Unlike the other biomarkers identified, PCR Ct is readily accessible (PCR is now used by >70% of US hospitals to diagnose CDI), Reference Guh, Mu and Winston41 for any positive C. difficile PCR Reference Sandora, Kociolek and Williams39 but Ct data are not traditionally reported to clinicians.
Future studies should carefully plan analysis to avoid common issues (eg, overfitting) and may benefit from considering novel biomarkers and/or higher-dimensional models that could augment or replace existing tools that underperform.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/ice.2024.75.
Financial support
This work was funded by the National Institutes of Health (K23AI163368 to G.R.M., T32AI055432 to R.H.B., 5R01 AI124214 to WP, 1R01 AI152477 to WP) and the National Center for Advancing Translational Science (UL1TR003015, KL2TR003016 to G.R.M.).
Competing interests
W. A. Petri is a consultant for TechLab Inc., a company that manufactures diagnostic tests for C. difficile toxins. All other authors report no conflicts of interest relevant to this article.
Patient consent statement
The study was approved by the University of Virginia Institutional Review Board for Health Sciences Research (IRB-HSR 20082), and written consent was waived due to the retrospective nature of the study.
Open access
