Multiple studies have reported a 2- to 3-fold rise in severe sepsis incidence over the past several decades, accompanied by substantial decreases in case fatality rates.Reference Martin, Mannino, Eaton and Moss 1 – Reference Kumar, Kumar and Taneja 6 Almost all of these estimates are based upon claims data, however, and may therefore be biased by increasingly vigilant diagnosis and coding practices.Reference Lindenauer, Lagu, Shieh, Pekow and Rothberg 7 – Reference Rhee, Gohil and Klompas 9 Indeed, we previously demonstrated that the sensitivity of sepsis codes for capturing the most overt form of sepsis, bacteremia with concurrent vasopressors or lactic acidosis, has increased significantly over time, and that improving documentation of acute organ dysfunction is also likely biasing estimates of changing sepsis severity and burden.Reference Rhee, Murphy and Li 10 , Reference Rhee, Murphy and Li 11
Given the questionable reliability of administrative claims to track severe sepsis incidence and outcomes, we developed a surveillance definition that uses clinical data instead of diagnosis codes and is potentially applicable using electronic health record (EHR) data. Our aim was to characterize the accuracy and stability of this definition over time and compare it with claims-based definitions, using manual medical chart reviews with the international consensus definition as the reference standard. We then estimated and compared changes in severe sepsis incidence and mortality rates using the clinical vs claims-based surveillance definitions.
METHODS
This was a retrospective cohort study at Massachusetts General Hospital and Brigham and Women’s Hospital in Boston, Massachusetts, involving all patients at least 18 years old hospitalized from January 1, 2003, through December 31, 2012. The study was approved by the Partners Healthcare Institutional Review Board.
Surveillance Definitions Based on EHR Clinical Data
Our surveillance definition for severe sepsis required clinical indicators of suspected infection and acute organ dysfunction (Table 1). “Suspected infection” was defined as a blood culture order and at least 4 consecutive days of antibiotics (or <4 days if antibiotics were continued until at least 1 day prior to death or discharge), with the first day of antibiotics required to be a new parenteral agent. Four days was chosen as a minimum duration of antibiotic therapy because empirical antibiotics are often stopped after 48–72 hours when cultures return negative and the patient’s condition is deemed noninfectious. We required the antibiotic start date and at least 1 organ dysfunction criterion to occur within ±2 calendar days of the blood culture order. Organ dysfunction and hypoperfusion were defined using criteria and thresholds adapted from the Surviving Sepsis Guidelines, but modified to take into account baseline levels of organ dysfunction.Reference Dellinger, Levy and Rhodes 12
Comparison of the International Consensus Definition of Severe Sepsis and Surveillance Definition Based on Electronic Health Record (EHR) Clinical Data
NOTE. INR, international normalized ratio.
a Adapted from the 1991 American College of Chest Physicians/Society of Critical Care Medicine Consensus Conference Committee, 2001 Society of Critical Care Medicine/ European Society of Intensive Care Medicine/American College of Chest Physicians/ American Thoracic Society/ Surgical Infection Society International Sepsis Definitions Conference, and 2012 Surviving Sepsis Campaign Guidelines.
b New intravenous antibiotic defined by washout period of 2 days. Any subsequent oral antibiotic also requires washout period of 2 days.
c Vasopressor=norepinephrine, dopamine, vasopressin, epinephrine, phenylephrine.
d Baseline values are defined as lowest for creatinine, bilirubin, and INR from day −30 of hospitalization admission to day of discharge, or the highest for platelets from day −30 of admission to day of discharge.
e End-stage renal disease defined by International Classification of Diseases, Ninth Revision, Clinical Modification code of 585.6.
f Any order for warfarin from day −30 of admission to day of hospital discharge.
g Serum lactate was not included in the primary surveillance definition, but was included in a secondary definition.
We excluded lactate levels from our primary definition because the use of serum lactate orders has increased dramatically over the past decade.Reference Casserly, Phillips and Schorr 13 – Reference Rhee, Murphy and Li 16 We performed a sensitivity analysis, however, to assess whether and how the inclusion of lactate (with a level ≥2.0 mmol/L) would affect the perceived incidence of severe sepsis over time. We also explored the utility of a parsimonious “simplified surveillance definition” that required only evidence of suspected infection and vasopressors, initiation of mechanical ventilation for at least 2 days, or an acute rise in creatinine—reflecting the most common forms of organ dysfunction associated with sepsis.Reference Blanco, Muriel-Bombin and Sagredo 17 – Reference Guidet, Aegerter, Gauzit, Meshaka and Dreyfuss 19 We reasoned that if a simpler definition can mirror trends detected with a comprehensive definition then it might facilitate future public health surveillance efforts across a broader array of hospitals.
Surveillance Definitions Based on Claims Data
We identified patients with claims-based indicators of severe sepsis on hospital discharge using the method of Angus et alReference Angus, Linde-Zwirble, Lidicker, Clermont, Carcillo and Pinsky 20 as modified by Iwashyna et al.Reference Iwashyna, Odden and Rohde 21 This “implicit” definition flags patients with concurrent International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes for infection and organ dysfunction, or “explicit” ICD-9-CM codes for severe sepsis or septic shock (995.92, 785.52). Secondarily, we examined just the patients with explicit severe sepsis or septic shock codes.
Assessment of Surveillance Definition Accuracy
We compared the accuracy of our clinical and claims-based definitions by reviewing 1,000 randomly selected medical charts of patients with at least 1 blood culture order while hospitalized. We reasoned that blood culture orders were a simple marker that would capture the great majority of patients with severe sepsis. We drew 600 medical charts from hospitalizations from 2003–2009 and 400 from 2012 in order to be able to assess for changes in the sensitivity and positive predictive value (PPV) of our various definitions over time while retaining precision of our estimates for the current era. An intensivist (C.R.) systematically reviewed each patient’s progress notes, discharge summaries, nursing flow sheets, medication records, and microbiology, laboratory, and radiology findings using a standardized data collection tool in REDCapReference Harris, Taylor, Thielke, Payne, Gonzalez and Conde 22 to determine whether the patient met criteria for severe sepsis using the international consensus definition.Reference Levy, Fink and Marshall 23 A second intensivist (S.K.) independently reviewed 60 randomly selected medical charts (split evenly between those initially classified as severe sepsis, septic shock, and non-severe sepsis/septic shock). Each reviewer was masked to the other’s findings as well as to patients’ ICD-9-CM codes and whether patients were positive with respect to the electronic clinical surveillance definitions. Interobserver agreement was assessed using the kappa statistic. After all medical chart reviews were complete and surveillance definitions applied, we examined discrepant cases to understand reasons for false-positives and false-negatives.
Incidence and Mortality Trends
We applied all surveillance definitions to all patients hospitalized at Massachusetts General Hospital and Brigham and Women’s Hospital in 2003–2012 and calculated annual incidence and in-hospital mortality rates for patients flagged by each definition.
Data Source
Patients’ demographic characteristics, ICD-9-CM codes, medications, laboratory results, and dates of admission, discharge, and death were retrieved from the Partners Research Patient Data Registry, a centralized clinical data warehouse that has been in full production since February 2002 and is populated with data extracted from Partners’ home-built EHR system.Reference Murphy and Chueh 24 We obtained blood culture data from the clinical microbiology laboratories. The dates of initiation and discontinuation of mechanical ventilation of all hospitalized patients were obtained from the respiratory therapy departments for the Massachusetts General Hospital cohort for the entire study period and from Brigham and Women’s Hospital for the years 2005–2012. We used ICD-9-CM codes (96.7x) or Current Procedural Terminology codes (94002, 94003, or 94004) to identify mechanical ventilation in the Brigham and Women’s Hospital population for the years 2003–2004.
Statistical Analyses
Exact 95% binominal CIs were calculated for sensitivity and PPV. Differences in sensitivity and PPV in 2012 vs 2003–2009, and between the clinical and claims definitions in 2012, were analyzed using the z test for 2 proportions. Ten-year incidence and mortality trends were assessed by fitting time series models with linear trends to the observed annual rates. The 10-year fitted percent change for incidence was calculated as the ratio between the fitted absolute annual change multiplied by 10 and the observed baseline incidence rate in 2003. Trends imputed from clinical and claims data were compared through the z score by dividing the difference between each slope by the square root of the sum of the variance of each fitted trend line. We considered P<.05 to be statistically significant and used 2-sided tests. All analyses were performed using SAS, version 9.4 (SAS Institute).
RESULTS
For 2003–2012, there were a total of 901,466 hospitalizations at both hospitals. Of these, 64,199 (7.1%) flagged the EHR clinical surveillance definition for severe sepsis, 69,075 (7.7%) met Angus criteria, and 11,096 (1.2%) had explicit severe sepsis or septic shock codes.
Accuracy of Surveillance Definitions
Of the 1,000 medical charts reviewed, 220 met criteria for severe sepsis. Agreement between the 2 chart reviewers was very good (kappa, 0.8). The characteristics of severe sepsis and non-severe sepsis patients are summarized in Table 2. The most common types of sepsis-induced organ dysfunction were hypotension (72.3% of cases), acute kidney injury (55.9%), and acute respiratory distress syndrome (24.1%). Septic shock was present in 88 (40%) of the 220 patients with severe sepsis. Lactate levels were measured in an increasing fraction of severe sepsis patients over time: 16 (29.6%) of 54 cases in 2003 and 71 (77.2%) of 92 cases in 2012 (P<.01). Forty-eight (21.8%) of the 220 patients with severe sepsis died in the hospital, compared with 15 (1.9%) of the 780 patients without severe sepsis (P<.01) (Table 2).
Characteristics of Patients With Severe Sepsis Determined by Medical Record Review
NOTE. Data are no. (%) of patients, unless otherwise specified. IQR, interquartile range.
The accuracy of all surveillance definitions is summarized in Table 3. The sensitivity of the primary EHR clinical surveillance definition was stable between 2003–2009 and 2012 (77.3% vs 80.4%, P=.58), as was the PPV (51.0% vs 52.9%, P=.74). In contrast, the sensitivity of Angus codes increased from 51.6% in 2003–2009 to 67.4% in 2012 (P=.02) and sensitivity of explicit severe sepsis/septic shock codes increased from 9.4% in 2003–2009 to 26.1% in 2012 (P<.01). The PPVs of both claims-based definitions were stable across time. Reasons for false-negatives and false-positives for the clinical and Angus claims surveillance definitions after review of discrepant cases are detailed in Table 4.
Accuracy of Surveillance Definitions Based on Electronic Health Record (EHR) Clinical and Claims Data for Identifying Hospitalizations With Severe Sepsis Determined by Medical Record Review in 2012 vs 2003–2009
NOTE. ICD-9-CM, International Classification of Diseases, Ninth Revision, Clinical Modification; INR, international normalized ratio.
a Indicates statistically significant differences.
Reasons for False-Negatives and False-Positives for Surveillance Definitions Relative to Severe Sepsis Determined by Medical Record Review
NOTE. EHR, electronic health record; INR, international normalized ratio.
a Numbers in each cell do not necessarily add up to total N because several cases had multiple reasons for false-negative or false-positive status.
In 2012, the EHR clinical surveillance definition had superior sensitivity compared with the Angus definition (80.4% vs 67.4%, P=.04) and the explicit severe sepsis/septic shock codes (80.4% vs 26.1%, P<.01). PPV for the clinical surveillance definition (52.9%) was similar to the Angus definition (55.4%, P=.69) but less than the PPV for explicit severe sepsis/septic shock codes (96.0%, P<.01).
On sensitivity analysis, adding lactate at least 2.0 mmol/L as a criterion increased sensitivity and PPV, whereas the simplified definition (suspected infection and vasopressors, mechanical ventilation for ≥2 days, or creatinine increase of ≥0.5 mg/dL) had slightly lower sensitivity and higher PPV. Both definitions also had stable performance over time.
Of the 48 chart review–confirmed severe sepsis patients who died, the EHR clinical surveillance definition flagged 41 (85.4%), the definition with lactates included flagged 42 (87.5%), and the simplified definition flagged 40 (83.3%). In contrast, the Angus method flagged 29 deaths (60.4%) and explicit severe sepsis/septic shock codes flagged 14 (29.2%) (P<.01 for all clinical definitions vs either claims definition).
Incidence and Mortality Trends
There were substantial discrepancies between incidence trends derived from surveillance using clinical vs claims data (Figure 1). The incidence of severe sepsis using Angus codes increased from 519 per 10,000 hospitalizations in 2003 to 867 in 2012 (fitted 10-year increase of 72.4% [95% CI, 57.1%–87.6%], P<.01 for linear trend). With explicit severe sepsis/septic shock codes, the incidence increased from 21 to 160 per 10,000 hospitalizations (572% increase, [95% CI, 395%–750%], P<.01). In contrast, there was only a modest trend toward more cases using the EHR-based clinical surveillance definition, changing from 709 to 757 cases per 10,000 hospitalizations (7.7% increase [95% CI, −1.1% to +16.5%], P=.14). When including lactate at least 2.0 mmol/L as an additional criterion, however, there was a significant increase in incidence, from 712 to 824 cases per 10,000 hospitalizations (18.6% increase [95% CI, 6.5%–30.6%], P=.02). Annual incidence using the simplified definition was stable, changing from 597 to 574 cases per 10,000 hospitalizations (−4.7% change [95% CI, −15.3% to +5.9%], P=.42).
Severe sepsis incidence trends using surveillance definitions based on electronic health record (EHR) clinical data versus claims data, 2003-2012. Percentages next to each method refer to fitted 10-year change relative to 2003, with associated 95% confidence limits.
Annual changes in hospital mortality rates varied similarly (Figure 2). There was a substantial mortality decrease using Angus codes from 17.6% in 2003 to 12.6% in 2012 (fitted absolute 10-year decline of 5.4% [95% CI, 4.1%–6.7%], P<.01), whereas mortality using explicit severe sepsis/septic shock codes decreased from 50.0% to 32.2% (17.3% decline [95% CI, 15.2%–19.5%], P<.01). In contrast, mortality rates decreased more modestly amongst patients meeting the EHR definition, from 15.8% in 2003 to 14.2% in 2012 (1.7% decline [95% CI, 1.1%–2.3%], P<.01). The mortality rate associated with the definition including lactates decreased from 15.8% to 13.6% (2.6% decline [95% CI, 2.0%–3.2%], P<.01), while mortality associated with the simplified definition was stable: 17.9% in 2003 vs 17.5% in 2012 (0.5% decline [95% CI, 0.4%–1.4%], P=.30). All incidence and mortality trends imputed from clinical data were significantly different than trends from claims data (P<.01 for all comparisons).
Severe sepsis mortality trends using surveillance definitions based on electronic health record (EHR) clinical data versus claims data, 2003-2012. Percentages next to each method refer to fitted absolute 10-year change, with associated 95% confidence limits.
DISCUSSION
We found that a surveillance definition based on EHR clinical data had superior sensitivity for identifying hospitalizations with severe sepsis compared with claims-based methods. The PPV of this definition was similar to the widely used modified Angus method but lower than explicit severe sepsis/septic shock codes. However, the clinical surveillance definition demonstrated stable performance over time, whereas the sensitivity of both claims methods increased, particularly the explicit severe sepsis/septic shock codes. On applying the definition to all hospitalized patients over 10 years, we found a much more modest increase in severe sepsis incidence and decline in hospital mortality compared with claims-based estimates.
The increasing sensitivity of claims over the past decade suggests that clinicians and coders are becoming better at appropriately recognizing and coding for sepsis and/or organ dysfunction over time. This is likely due to increasing awareness of sepsis, financial incentives to code for higher patient complexity, and the introduction of new diagnosis codes.Reference Lindenauer, Lagu, Shieh, Pekow and Rothberg 7 – Reference Rhee, Murphy and Li 11 The larger decrease in mortality rates of patients with sepsis codes compared with that of patients fulfilling our clinical surveillance definition suggests that sepsis codes are being assigned to less ill patients over time. The stable PPV for these codes, however, suggests that they are nonetheless appropriate. Our findings do, however, underscore the need for more credible ways to measure sepsis rates over time, which is particularly important in light of the profusion of sepsis prevention initiatives and new governmental regulations compelling hospitals to adopt sepsis protocols and report adherence to quality measures.Reference Rhee, Gohil and Klompas 9 The EHR-based clinical surveillance method had stable performance over time, suggesting that it may give more reliable estimates of changes in disease incidence and mortality than claims.
Our analysis of the true severe sepsis cases missed by the EHR clinical surveillance definition revealed that most were either sepsis manifested solely by transient hypotension (a less severely ill group whose importance of capturing for surveillance can be debated) or due to failure of the antibiotic criteria to capture septic patients. Reassuringly, only 2 cases were missed because organ dysfunction occurred outside the ±2 day infection window period, suggesting that this is a reasonable timeframe for surveillance. Unsurprisingly, most false-positive cases with the EHR definition were due to the lack of a causal association attributed by the reviewer between infection and organ dysfunction. However, for purposes of standardizing surveillance, removing this potentially subjective assessment is likely to be beneficial.
Our study supports the feasibility of large-scale, automated, objective sepsis surveillance using EHR clinical data. At present there are relatively few large healthcare networks with complete electronic data spanning many years; however, the increasing adoption of EHR systems is making automatable public surveillance increasingly feasible. We note that surveillance using a parsimonious set of organ dysfunction criteria (vasopressors, mechanical ventilation, and rise in creatinine) had slightly lower sensitivity, and slightly higher PPV, compared with our full surveillance definition, with similar observed trends in incidence over time. This type of definition, with its relatively simple criteria, may facilitate surveillance across a broader array of facilities. In addition, the EHR-based clinical surveillance definitions captured more of the severe sepsis deaths compared with claims. Resources spent on surveillance should ideally capture events associated with the greatest risk of adverse outcomes in order to help target and monitor prevention efforts most efficiently.
When lactate was added as a surveillance criterion, there was a significant increase in sepsis incidence, though still less than that imputed with claims. This is likely due to the increasing use of lactate testing over time and extension to patients without evidence of other organ dysfunction.Reference Rhee, Murphy and Li 16 This may reflect the impact of the Surviving Sepsis Campaign guidelines and numerous publications suggesting that measuring lactate levels is beneficial.Reference Dellinger, Levy and Rhodes 12 , Reference Shapiro, Howell and Talmor 25 – Reference Singer, Taylor, LeBlanc, Williams and Thode 28 Our findings suggest that although lactate measurement may be important for the clinical care of individual patients, it is less suited to helping estimate changes in population-level sepsis burden over time because of significant increases in testing rates.
Our study has several important limitations. First, we included data from only 2 academic hospitals, which may limit the generalizability of our findings. However, the incidence, clinical characteristics, and mortality rate of our severe sepsis cohort resembles that reported in other studies,Reference Sands, Bates and Lanken 29 – Reference Angus and van der Poll 31 as do the performance characteristics of the claims methods we examined and their associated incidence and mortality trends.Reference Iwashyna, Odden and Rohde 21 , Reference Whittaker, Mikkelsen, Gaieski, Koshy, Kean and Fuchs 32 , Reference Gaieski, Edwards, Kallan and Carr 33 Second, a single reviewer classified cases as severe sepsis using international consensus definitions. Cases were reviewed using structured forms, however, and there was excellent agreement with a second masked reviewer on a random subset of medical charts. Third, we limited our surveillance to patients with blood culture orders. It is unlikely, however, that many patients with severe sepsis did not have blood cultures ordered since the threshold for ordering blood cultures is very low in severely ill patients.Reference Sands, Bates and Lanken 29 , Reference Shafazand and Weinacker 34 Fourth, several components of our surveillance definition rely on therapeutic maneuvers that may have varying thresholds between clinicians and institutions, such as initiation of antibiotics, vasopressors, and mechanical ventilation. However, claims-based methods also rely on capturing treatment modalities including mechanical ventilation.Reference Angus and van der Poll 31 Fifth, the modified Angus definition includes codes for neurologic dysfunction, but we did not include this as one of the clinical criteria. We felt this was justified because altered mental status was not included in the most recent Surviving Sepsis Guidelines’ definition of severe sepsis.Reference Dellinger, Levy and Rhodes 12 Sixth, our data do not directly speak to the performance of ICD-10 codes, which are now being implemented in the United States. However, having a surveillance measure that is stable over time will only become more important as clinicians become more comfortable and proficient with the new coding scheme. Moreover, the clinical forces leading clinicians to be increasingly more vigilant about detecting and treating sepsis will continue independently of the introduction of ICD-10. Notably, a recent systematic review of the validity of sepsis claims found that the sensitivity of ICD-10 codes in other countries ranged from 5.9% to 52.5%—well below the sensitivity of our EHR definitions.Reference Jolley, Sawka, Yergens, Quan, Jette and Doig 35 Lastly, examination of trends using clinical data cannot be viewed as definitive since both sensitivity and PPV of the surveillance definitions were imperfect. Nonetheless, the stable sensitivity and PPV of the clinical surveillance definitions across time support their use as proxies for severe sepsis over the entire decade.
In conclusion, a surveillance definition for severe sepsis based upon routinely collected EHR clinical data was more sensitive and resistant to changes in performance over time compared with claims-based methods. When we applied the surveillance definition to 10 years of data from 2 large hospitals, we found that severe sepsis incidence has been rising more modestly and mortality declining less rapidly compared with claims-based estimates. The use of EHR clinical data warrants further exploration for routine public health surveillance.
ACKNOWLEDGMENTS
Financial support: Prevention Epicenters Program of the Centers for Disease Control and Prevention (grant 3U54 CK000172-04S1); National Institutes of Health (grant T32 AI007061 to C.R.).
Potential conflicts of interest. All authors report no conflicts of interest relevant to this article.
