Introduction
Clinical decision support systems (CDSSs) have been shown to increase adherence to clinical guidelines Reference Sutton, Pincock, Baumgart, Sadowski, Fedorak and Kroeker1 and augment diagnostic and management behavior in several infectious syndromes. Reference Dezman, Lemkin, Papier and Browne2–Reference Shaikh, Hoberman and Hum4 CDSSs assist in diagnosis by allowing correct application of disease-specific criteria, serving as educational tools about unfamiliar syndromes, improving the appropriateness of laboratory testing, Reference Algaze, Wood, Pageler, Sharek, Longhurst and Shin5 and assisting with the application of isolation precautions. Reference Dugdale, Rubins and Lee6
Emerging infectious diseases place a high cognitive burden on frontline clinicians for several reasons: the clinical presentation is unfamiliar, testing algorithms may change rapidly, detailed epidemiologic history is crucial for identifying at-risk patients, and unfamiliar infection control protocols make applying isolation precautions challenging. The mpox outbreak of 2022–2023 exemplified all these conditions. Since the eradication of smallpox, few clinicians were familiar with poxvirus infections. At the beginning of the epidemic, testing was scarce. And, since infections were concentrated in gay, bisexual, and other men who have sex with men, Reference Thornhill, Barkati and Walmsley7 it was critical that history taking focus on Centers for Disease Control and Prevention (CDC) epidemiologic criteria for a person under investigation (PUI). These questions, centering on a patient’s sexual health and behaviors, are not universally asked. Reference Goyal, McCutcheon, Hayes and Mollen8
To support the identification, isolation, and diagnosis of people presenting with a rash and possible mpox, the “Evaluate for Mpox” (EvalMpox) CDSS was incorporated into the electronic health record (EHR) of a large integrated healthcare system.
Methods
Based on experience developing CDSS for coronavirus disease 2019 (COVID-19), Reference Dugdale, Rubins and Lee6 a team of infectious diseases, infection control, and information technology experts constructed EvalMpox. Toward quick dissemination at the beginning of the outbreak, we previously communicated a rapid report on the initial CDSS applied to the first 55 evaluated patients ending July 20, 2022. Reference Albin, Lazarus and Hysell9 This manuscript analyzes the performance of an enhanced CDSS and all 668 encounters through April 12, 2023.
EvalMpox assists clinicians in identifying patients with mpox by guiding the collection of information regarding epidemiologic criteria for CDC PUI status in patients with a new, unexplained rash (Figure 1). Epidemiologic criteria were updated throughout the epidemic to conform to evolving CDC criteria. If a clinician inputs both clinical and epidemiologic criteria for mpox, EvalMpox classifies the patient as a PUI and recommends testing. If the patient does not meet clinical and epidemiologic criteria for mpox, EvalMpox recommends against testing unless clinical suspicion is high. EvalMpox then generates a risk assessment note in the EHR, coordinates the application of mpox-related infection statuses in the patient’s electronic chart, and orders appropriate isolation (Supplemental Figure 1 online).
From the top left, EvalMpox guides clinicians to sample images of mpox rashes and guides history taking to allow a standardized collection of information on rash onset, location, qualities, and associated systemic symptoms. It also prompts the clinician to document the rash photographically to assist in the evaluation of rash evolution over time. This standardized approach also accomplishes clinician teaching on features of this unfamiliar disease and ensures evaluation for signs or symptoms that may not be part of a routine evaluation (eg, pharyngitis, proctitis). From the top right, risk factor identification assists with contact tracing. By collecting information on challenges to discharge home, EvalMpox facilitates early involvement of in-house case management and Department of Public Health input. For patients classified as PUI, EvalMpox provides local site contact information to assist HCW in patient triage and testing. Finally, EvalMpox automatically coordinates the application of mpox-related infection statuses and isolation. Example screenshot from Epic™ (Epic Systems Corporation).Note: HCW, healthcare workers
Data on EvalMpox encounters exported from the EHR (Epic) included patient demographics, encounter date/time, practice location/setting, clinician-user role, and PUI/non-PUI status. Mpox testing results performed in our system were separately exported. Data were inspected for duplicate encounters, and charts were manually reviewed to ensure data integrity. Categorical data were analyzed with χ2 testing and continuous data by t test. Negative predictive value (NPV) and positive predictive value (PPV) were calculated over the total analyzed period. Data were collected under MGB IRB protocol 2012P002359.
Results
Tool utilization
EvalMpox was used in 668 encounters, originating from over 100 clinical locations across Greater Boston, Nantucket, Martha’s Vineyard, western Massachusetts, and southern New Hampshire (Supplemental Figure 2 online). Encounters originated in the emergency department (n = 219, 33%), urgent care (n = 202, 30%), outpatient (n = 199, 30%), and inpatient (n = 48, 7%) settings (Supplemental Figure 3 online) and peaked in early August 2022 (Supplemental Figure 4 online). EvalMpox was completed by clinicians in diverse role groups, including attending physicians, advanced practice providers, postgraduate trainees, and registered nurses (Supplemental Figure 5 online).
Patient characteristics
Based on the presence or absence of epidemiologic criteria, EvalMpox classified 275 patients as PUI and 393 patients as non-PUI, respectively. Consistent with national case characteristics reported to the CDC, patients designated PUIs by EvalMpox were significantly younger than those designated non-PUIs (mean age 34 vs 40, P value < .001 by t test) (Table 1 and Supplemental Figure 6 online). Similarly, PUIs were also significantly more likely to have a recorded legal sex as male (82% vs 55%, P < .001 by χ2, Table 1 and Supplemental Figure 7 online).
Characteristics of persons under investigation (PUI) and non-PUI as designated by EvalMpox. One PUI had an unknown legal sex
Mpox testing
PUIs were significantly more likely to be tested for mpox compared with non-PUIs (210 of 275 compared with 53 of 393, P < .001 by χ2, Table 1 and Supplemental Figure 8 online). Among the tested PUIs, 126 (60%), 74 (35%), and 10 (5%) tested negative, positive, or inconclusive by polymerase chain reaction (PCR), respectively. Among the tested non-PUIs, 49 (92%), 3 (6%), and 1 (2%) tested negative, positive, and inconclusive by PCR, respectively (Table 1 and Supplemental Figure 9 online). Patients designated PUI were significantly more likely to test positive for mpox (P < .001 by χ2). The PPV of an EvalMpox PUI designation for a positive PCR was 35% (95% CI 29%–42%) and the NPV was 99% (95% CI 98%–100%). One hundred sixteen PCR tests were sent without a corresponding encounter where EvalMpox was performed. Ninety-seven (84%), 13 (11%), and 6 (5%) were negative, positive, and inconclusive by PCR, respectively (Supplemental Figure 10 online).
Discussion
We describe the performance of EvalMpox, a novel CDSS for the identification, evaluation, and management of patients meeting CDC PUI criteria for mpox. There was widespread adoption of EvalMpox across our large, integrated healthcare system among diverse provider roles and in all care settings. The CDSS performed well; our patients classified as PUI had similar patient demographics compared with CDC mpox case demographics, and PUI were more likely to test positive for mpox than non-PUI. The NPV of EvalMpox was high.
There are several limitations to the conclusions that can be drawn from our report. First, this study was conducted in a single health system, potentially limiting generalizability. Second, though we performed extensive education prior to and during implementation, uptake was not universal. If patient characteristics influenced clinician decisions of whether to use EvalMpox, this utilization behavior may have biased the observed test characteristics of the CDSS. However, EvalMpox was used in most mpox testing encounters. Third, though we find that the NPV for EvalMpox was high, despite this being the largest worldwide mpox outbreak, low overall community prevalence certainly contributes to this result. Fourth, as in any CDSS that relies on provider data entry, errors in tool use can lead to inappropriate recommendations. A focused chart review of the three patients designated non-PUI by EvalMpox who tested positive by PCR revealed that one of those patients reported epidemiologic risk factors for mpox that were not input correctly into EvalMpox.
Finally, clinician judgment remains necessary when interpreting the recommendations from any CDSS. Two individuals who tested positive for mpox reported no epidemiologic risk factors to multiple interviewers and so, following CDC PUI criteria, were designated non-PUI. EvalMpox does direct users to additional clinical resources, and these patients eventually underwent mpox testing due to repeat presentations to care.
In conclusion, our data support the potential for CDSS to assist in the identification, evaluation, and management of patients with emerging infectious diseases, supporting laboratory stewardship and appropriate implementation of transmission-based precautions. Our findings lay the groundwork for future investigations, including into which factors influence healthcare workers (HCW) toward using or not using an available CDSS. It will also be useful, during future outbreaks of emerging infections, to consider randomizing HCW to CDSS use or standard of care, to allow rigorous interrogation of the ability of CDSS to improve diagnostic accuracy and disease-specific knowledge.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/ice.2024.51.
Acknowledgments
The authors would like to thank Jasmine B. Ha, MS, Mass General Brigham Digital Health, for project management support of the MGB MPX Digital Health response, and Zoe Vernick, for her project management of the preparation of this manuscript, as well as a poster presentation to IDWeek.
Financial support
This work was supported by a cooperative agreement from the Centers for Disease Control and Prevention (CK22-2203). The Centers for Disease Control and Prevention had no involvement in the preparation, submission, or review of the manuscript.
Competing interests
All authors report no conflicts of interest relevant to this article.
Open access
