Introduction
Community-acquired pneumonia (CAP) is one of the most common indications for antibiotic prescribing in the inpatient setting. Reference McDermott and Roemer1 However, inappropriate antibiotic prescribing for CAP is common, particularly excess duration. Reference Magill, O’Leary and Ray2 Antibiotic stewardship (AS) programs reduce guideline-discordant antibiotic prescribing and improve outcomes for patients with CAP. Reference Nussenblatt, Avdic and Cosgrove3 Quickly identifying patients hospitalized with CAP in the electronic health record (EHR) who could benefit from stewardship remains challenging. Reference Whittle, Fine and Joyce4 Prior studies have used ICD-10 codes and claims data to identify pneumonia cases, which show mixed performance and cannot identify encounters at the point of care. Reference van de Garde, Oosterheert, Bonten, Kaplan and Leufkens5,Reference Yu, Nelson, Bounds and Jackson6 An alternative approach that has not been well-validated is using a provider-selected indication (PSI) for antibiotic orders. This would allow for real-time identification of patients treated for CAP who can be assessed for AS interventions. Our study aims to assess the ability of PSI for antibiotic orders to identify CAP cases.
Methods
Study design and population
In our health system, prescribers have been required to select an indication for antibiotic orders from a pre-specified list since 2017 (Supplement Table 1). While some indications are included in order sets, there is no pneumonia-specific order set. To assess the performance of PSI in identifying patients receiving treatment for CAP, we conducted a retrospective validation study of patients ≥18 years old admitted to one of three acute care hospitals within an academic health system from January 1, 2019 through December 31, 2021. Patients were eligible if they received at least one systemic antibiotic within 48 hours of admission; antibiotics never used to treat CAP were excluded (Supplement Table 2). We chose this source population to reflect a broad population from which AS teams might be seeking to identify pneumonia patients. Encounters with insufficient documentation of intended antibiotic use were excluded. A random sample was selected with a target of 440 encounters.
Definitions
PSI for community-acquired lower respiratory tract infection (CA-LRTI) was identified if at least one antibiotic administered in the first 48 hours of admission had a selected indication of “Pneumonia/lower respiratory infection–community acquired.” The only other lower respiratory tract option was “Pneumonia/lower respiratory infection–healthcare associated.” The reference standard was defined as documented intention to treat for pneumonia by inpatient provider(s) within 48 hours of admission, as assessed by manual review of EHR notes; accuracy of the CAP diagnosis was not assessed. Any documented terminology for pneumonia except hospital-acquired pneumonia (HAP) or ventilator-associated pneumonia (VAP) was counted in this reference standard.
Data collection
Demographic and antibiotic administration data were extracted from the EHR (Epic Clarity). Data from chart abstraction were stored in REDCap hosted at the University of Pennsylvania and included intended antibiotic indication(s) and chest radiology report results. Reference Harris, Taylor, Thielke, Payne, Gonzalez and Conde7 A medical student (DY) was trained by two infectious disease physicians (KH and LD) to conduct chart abstraction. Ten charts were reviewed by both LD and DY for training, and ambiguous cases were reviewed by KH or LD.
Statistical analysis
PSI CA-LRTI was compared against the reference standard by calculating sensitivity, specificity, and positive and negative predictive values. Secondary analyses included: (1) Including encounters documenting the terms HAP and VAP in the reference standard to determine the effect on sensitivity; (2) Adjusting PSI to include healthcare-associated LRTI (PSI HA-LRTI) to assess provider differentiation among CA-LRTI and HA-LRTI; and (3) Assessing performance characteristics of pneumonia ICD-10 codes using the same reference standard for comparison (Supplement Table 3). The proportion of patients with a new/increased infiltrate on chest radiograph (x-ray) or computed tomography scan in the first 48 hours was also assessed.
Results
There were 442 encounters screened with 440 included and two excluded due to insufficient documentation. Demographic characteristics and comorbidities are described in Table 1. Seventy-three (16.5%) patients were treated for lower respiratory tract infection (LRTI). Sixty-four (87.7%) of LRTI-treated patients were treated for pneumonia, 10 (13.7%) for COPD exacerbation, 1 (1.4%) for bronchiectasis flare, and 2 (2.7%) for other conditions. The most common antibiotics used for LRTI were vancomycin (25.0%), 4th generation cephalosporins (24.4%), macrolides (16.5%), and 3rd generation cephalosporins (12.8%).
Table 1. Demographics and clinical characteristics of included patients
a Includes antibiotic use for all included patients, not just patients with pneumonia.
PSI CA-LRTI had a sensitivity of 64.4% (95% CI, 59.9–68.9) and a specificity of 96.3% (95% CI, 94.6–98.1) for the detection of CAP treatment, with positive and negative predictive values reported in Table 2. Of 21 false negatives, all except one had HA-LRTI or sepsis as a selected indication for at least one antibiotic order. PSI CA-LRTI had comparable performance to pneumonia ICD-10 codes (Table 2). Adjusting the PSI to include selection of either CA or HA-LRTI increased sensitivity to 86.4% (95% CI, 83.2–89.6) but decreased specificity to 92.9% (95% CI, 90.5–95.3). Including all pneumonia terminology in the reference standard did not significantly change results (Table 2). The overall accuracy of PSI (sum of true positives and true negatives divided by total included) was 92.0%.
Table 2. Performance measures of methods for identifying community-acquired pneumonia
Note. CA-LRTI, community-acquired lower respiratory infection; HA-LRTI, healthcare-associated lower respiratory infection; VAP, ventilator-associated pneumonia; PPV, positive predictive value; NPV, negative predictive value.
We found that 86.2% of patients whom the provider treated for CAP had new or increased infiltrates noted on a radiology report. The EHR-based methods of CAP identification showed a similar proportion of new infiltrates, though including HA-LRTI lowered the proportion (Supplement Figure 1). Frequencies of terminology used in provider documentation for pneumonia are provided in Supplement Figure 2.
Discussion
In this study, we showed that PSI of CA-LRTI had moderate sensitivity and high specificity for identifying CAP-treated cases; given the relatively low prevalence of CAP in the population, this translated to a moderate PPV and fairly high NPV.
We found that sensitivity was primarily lowered because of (1) use of sepsis as an antibiotic indication early during admission and (2) lack of differentiation between HAP and CAP when selecting the indication, which was supported by an increase in sensitivity when using both CA and HA-LRTI PSIs. Decreases in PSI accuracy for similar reasons have been reported by other studies. Reference Heil, Pineles and Mathur8,Reference van den Broek, Beishuizen and Haak9 The Infectious Disease Society of America removed the term “healthcare-associated pneumonia” from guidelines in 2016. Reference Kalil, Metersky and Klompas10 While this was not updated in our EHR antibiotic indications during the study period, it has since occurred to improve accurate categorization of pneumonia.
Like other studies assessing PSI accuracy, we show a high agreement rate (92%) between PSI and clinical documentation. Reference Heil, Pineles and Mathur8,Reference van den Broek, Beishuizen and Haak9 Our study additionally calculates the sensitivity and specificity of PSI, which better assesses PSI performance for potential stewardship activities within a general population of patients receiving antibiotics. Though the PPV was only moderate, PSI could likely be used to increase the efficiency of identifying pneumonia cases for stewardship purposes, acknowledging that some non-pneumonia cases would still be identified. Additionally, despite the high NPV, not all pneumonia cases will be identified, and supplementation with other identification methods would be warranted. While PSI has similar sensitivity to ICD-10 codes for identifying CAP, PSI allows real-time identification of patients treated for CAP.
Our study has several limitations. Since the provider’s intention to treat CAP was the reference standard rather than an objective definition, some included patients may not in fact have pneumonia. Most providers used the general term “pneumonia” rather than “community acquired pneumonia,” which could result in misclassification of some encounters. Additionally, our antibiotic indication field may not be generalizable to other institutions.
In summary, we show PSI for antibiotic orders has potential for usage in real-time AS intervention for CAP patients. Methods with higher sensitivity should continue to be explored.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/ash.2024.41.
Acknowledgments
We acknowledge the Infectious Disease Society of America Grants for Emerging Researchers/Clinicians Mentorship Program and the Perelman School of Medicine Center for Clinical Epidemiology and Biostatistics Summer Research Fellowship for their support of DY. Research Transparency and Reproducibility: Data can be made available upon request.
Author contribution
DY: Conceptualization, data curation, analysis, methodology, validation, visualization, writing–original draft, writing–review and editing.
KWH: Conceptualization, data curation, analysis, methodology, validation, writing–review, and editing.
LC: Data curation, formal analysis, writing–review, and editing.
LD: Conceptualization, data curation, analysis, methodology, validation, writing–original draft, writing–review, and editing.
Financial support
None.
Competing interests
The authors declare that they have no relevant conflicts of interest.
Open access
