The European Centre for Disease Prevention and Control (ECDC), and their US counterpart, the CDC, defines criteria for surgical site infection (SSI) occurring within 30 days for a surgery without implant and 90 days for a surgery with implant. In a recent review of 26 studies investigating SSIs, Badia et alReference Badia, Casey, Pretrosillo, Hudson and Mitchell2 point out that “SSIs were consistently associated with elevated costs, relative to uninfected patients… [the studies reporting] that SSI patients required prolonged hospitalization, reoperation, readmission, and that SSIs increased mortality rates.” Furthermore, those patients developing an SSI generate increased costs compared to those not developing an SSI,Reference Broex, van Asselt, Bruggeman and van Tiel3-Reference Atkinson, Jones, Ousey and Stephenson5 leading to additional healthcare costs running potentially into billions of euros per year.Reference Leaper, van Goor and Reilly6 Many countries have recognized the importance of surveillance programs to determine not only the instantaneous rate of infections but also their trajectory over time, with Switzerland introducing such a program in 2009.Reference Troillet, Aghayev, Eisenring and Widmer7 In turn, these efforts have led to a wealth of new ideas for improving procedures and processes (see, eg, the reviews by Liu et alReference Liu, Dumville and Norman8 and Ailen et alReference Allen, David and Veerman9). In addition, innovative measures are collected to determine standardized SSI rates to allow intra- and interhospital comparisons.Reference Ejaz, Schmidt, Johnston, Frank and Pawlik10-Reference Fakuda12 However, the quality of surveillance itself has also come under scrutiny, in particular in the age of increased automation (eg, Pindyck et alReference Pindyck, Gupta and Strymish13 and van Mourik et alReference van Mourik, van Duijn, Moons, Bonten and Lee14), and, at least in some countries, linkage of financial penalties with SSI rates. In their 2017 article, Bordeianou et alReference Bordeianou, Cauley and Antonelli15 point out that “overall, hospitals using less rigorous capture methods had improved surgical-site infection rates for National Healthcare Safety Network [NHSN] compared to standardized National Surgical Quality Improvement Program reports,” and “studies have found large discrepancies in the SSI rates reported in NHSN data compared to internal audits.” In Switzerland, repetitive audits of the surveillance processes and outcomes were introduced in 2011, with many hospitals in the Swissnoso surveillance program having 3 audits (in our analysis 61 of 148 hospitals, 41%) in the period up to the end of 2019.Reference Kuster, Eisenring, Sax and Troillet16 The accuracy of SSI detection is estimated using a semi-quantitative score that reflects the quality of surveillance.
In this study, we compared and contrasted standardized SSI rates for 2 of the most common surgeries in Switzerland, knee and hip arthroplasty and colorectal procedures, with audit scores. In contrast to the situation in the United States and other countries, there are as yet no financial penalties for having a high infection rate in Switzerland. Nonetheless, hospital rates could theoretically be manipulated (or perhaps better, “gamed”) in some way because the rates are reported publicly (eg, Swissnono17). To this end, we considered options for detecting and potentially “correcting” scenarios in which underreporting could be occurring.
We hypothesized that audit score would be correlated with the SSI rate, meaning that hospitals with lower quality of surveillance would also report lower rates of infection, and that conversely, those with better audit quality would report higher rates of infection. We aimed (1) to determine whether SSI rates for prosthetic knee and hip surgeries and colorectal surgeries are correlated with the audit quality score; (2) to compare these 2 different surgeries to determine whether the correlation is different based on surgery type; (3) to quantify this distortion, assuming mismatch between what is measured in terms of SSI rates and reality; and (4) to propose methods for detecting underreporting and potential methods investigating scenarios in which underreporting is corrected in the form of a sensitivity analysis.
Methods
We selected adult patients with complete follow-up information for knee and hip elective first arthroplasties (January 1, 2009, until September 30, 2018, 156,467 procedures, 148 hospitals) and colorectal surgeries (January 1, 2009, until September 30, 2019, 61,410 procedures, 135 hospitals) from the Swissnoso database. For the analysis, we only included surgeries in the year previous to the audit and the actual year of the audit(s) for a hospital (ie, 2 years per audit). We assumed that this would provide an adequate summary of the infection rate at the time of the audit. The resulting analysis set consisted of 81,957 surgeries for knee and hip arthroplasty and 33,315 for colorectal procedures.
Standardized SSI rate
We defined a composite end point of 30-day or 1-year infection for an SSI, with the latter specifically relating to knee and hip arthroplasty that include permanent implants. (Note that these time points only partially coincide with the CDC/NHSN definition).
We standardized the hospital-specific SSI rate using the recognized National Healthcare Safety Network (NHSN) risk index, which comprises the American Society of Anesthesiology (ASA) score, the wound class and procedure duration.Reference Culver, Horan and Gaynes18 Each surgery is scored leading to an SSI rate and risk index score of 0, 1, 2 or 3, which then provides a composite score, the NHSN-adjusted SSI rate per hospital and procedure type.
Surveillance audit quality score
The 50-point score per audit is based on the CDC/NHSN method for SSI monitoring, which requires active, patient-based, prospective, and postdischarge surveillance, including review of medical records. Hospitals were visited on site by 1 of 3 specifically trained investigators (2 registered nurses, 1 physician), each with profound knowledge of the Swissnoso SSI surveillance methodology. The quality score and report (including false-negative or false-positive case reviews) are validated by a senior trained investigator, and in case of discrepancies in the case reviews, by a senior physician. A full description of the auditing procedures is provided in the article by Kuster et alReference Kuster, Eisenring, Sax and Troillet16 (Supplementary Tables S1–S3 online). After each onsite visit, the hospital receives a detailed validation report including specific recommendations to improve the quality of the surveillance process. Since its introduction, the scoring system and the use of the quality score values, have gained acceptance in Swiss hospitals, and the quality scores are reported publicly along with the SSI rates.
Primary analysis: Correlation of standardized SSI rate and surveillance audit quality score
The primary analysis considered the correlation between the standardized SSI rate on a hospital level and its audit score, both of which were averaged over all relevant years. Since there are, at most, 3 audit scores for each hospital for both knee and hip and colorectal procedures, we calculated the weighted mean of the NHSN risk index adjusted infection rates for the specific surgery type for the year previous to the audit and the year in which the hospital was audited. The weights were proportional to the number of surgeries of the type performed. The weighted mean audit quality score was calculated in an analogous manner.
Scatter plots were used to present the data graphically, whereby the mean audit quality score (x-axis) was plotted against the NHSN risk index adjusted infection rate for each hospital (y-axis). The degree of correlation was estimated using the slope parameter of the linear model, with guidelines added for the median of the respective distributions of both metrics to aid the reader with interpretation.
Table 1. Patient Characteristics for the included knee and hip Arthroplasties and Colorectal Procedures
Note. IQR, interquartile range; ASA American Society of Anesthesiology.
a Timing of the first antibiotic, negative is prior to incision.
Supplementary analyses
Infection rate trajectory
The trajectory of the average NHSN risk index adjusted SSI rates for the audit scores in each of the 2–3 years in which audits took place, and the previous year to the audit, was also plotted, considering the audit number (1, 2, or 3) as independent variable. For those hospitals with 3 audits, the mean slope of the trajectory was estimated by fitting a linear mixed-effects model with a random intercept per hospital.
Hospitals with low infection rate and low audit score
Many hospitals reported few or no infections for knee and hip surgeries. In a post hoc sensitivity analysis, we hypothesized that those hospitals with an infection rate and audit score in the lower quartile of the respective distribution were potentially underreporting the number of infections. Treating low (or no) infection rate at the hospital level as a missing data problem, we used multiple imputation to generate hypothetical infections under a missing at random assumption (refer to supplementary material).
We used the observed data from other hospitals to impute a hypothetical infection rate for those hospitals potentially underreporting their rate. The multiply imputed data were then integrated in the scatter plots from the primary analysis. All analyses were performed in R version 3.6.1 software.19
Ethical approval
The Cantonal Ethics Committee (Berne, Project ID 2019-00294) approved the study.
Results
Knee and hip arthroplasty
The analysis of knee and hip arthroplasty included 81,957 patients from 125 hospitals (Table 1 and Supplementary Fig. S1 online) with median NHSN adjusted infection rate 1.0% (IQR, 0.6%–1.5%) per hospital and median audit score 37 (IQR, 33–42). We detected no significant difference between the surgeries performed in all hospitals, and those in hospitals audited (Table 1 and Supplementary Table S1 online).
Figure 1 shows the NHSN risk index adjusted infection rate plotted against audit score for each hospital stratified by hospital type, along with the overall linear fit (solid black line) and 95% confidence intervals (grey). We observed large variability in rates and scores for knee and hip arthroplasty (Fig. 1A), with a noticeable increase in infection rate as audit score increased (P = .002). The variability is pronounced for public (black) compared to private (grey) hospitals, although there are some prominent outliers. Overall, both the infection rates and the audit scores were lower for private compared to public hospitals. The median infection rate for private hospitals was 0.8% (interquartile range [IQR], 0.5–1.2), and the median infection rate for public hospitals was 1.1% (IQR, 0.7–1.7) (P = .04). The median audit score for private hospitals was 33 (IQR, 30–36), and the median audit score for public hospitals was 38 (IQR, 35–42) (P < .001).
Fig. 1. Average audit score plotted against average National Healthcare Safety Network (NHSN) adjusted infection rate hospital type: private in grey, public in black; bubble sizes are proportional to standard error of the infection rate; linear model shown (black solid) with 95% confidence interval (shaded grey), along with median SSI rate and audit score (dotted).
Using the dotted lines denoting the median infection and audit scores to divide the plot into 4 quadrants in Figure 1, we observed a predominance of private hospitals with both low audit scores and low infection rates (bottom left), whereas public hospitals are predominantly in the top-right quadrant with higher audit scores and infection rates. The bottom-right quadrant might be considered the “sweet spot” where hospitals have both higher audit and lower infection rates.
In terms of the trajectory of audit scores per hospital over time, we detected a clear increasing trend in the quality of the surveillance (P < .001 for the increasing slope, Fig. S2).
Colorectal procedures
Our analysis of colorectal procedures included 33,315 patients from 110 hospitals (Table 1 and Supplementary Fig. S1 online) with a median NHSN risk index adjusted infection rate of 12.7% (IQR, 9.0–16.6) per hospital and a median audit score of 38 (IQR, 35–42). We detected no significant difference between the surgeries performed in all hospitals, and those in audited hospitals (Supplementary Table S1 online).
As with knee and hip arthroplasty, there is large variability in rates and scores for colorectal procedures. We observed a noticeable difference between public and private hospitals for planned procedures (Fig. 1B, top panel, N = 20,951 (74.6%) surgeries), with public hospitals having a much higher infection rate (12.9; [IQR, 9.8–15.1] vs 7.5 [IQR, 5.8–10.1]; P < .001); although the audit scores were comparable (40 [IQR, 36–43] vs 38 [IQR, 33–39]; P = .06), those for private hospitals were lower. The overall infection rate increased with audit score for planned procedures (P for the slope = .002), but in this case the increase was attributable to public hospitals, private hospitals showed no clear increase with audit score (Fig. 1B, top panel). For elective procedures, there was again a clear difference between public and private hospitals in terms of the infection rate (N = 9,119; 19.7 [IQR, 17.5–25.5] vs 8.4 [IQR, 6.9–13.0]; P < .001), with only a slight difference in audit scores (41 [IQR, 37–43] vs 37 [IQR, 35–38]; P = .02). As for the elective surgeries, the increasing trend with audit score was more pronounced for public hospitals (Fig. 1B, bottom). The audit score trajectory for the hospitals performing colorectal surgeries was similar to that for knee and hip procedures (not shown).
Hospitals with low infection rate and low audit score
In the sensitivity analysis of those hospitals potentially underreporting their knee and hip infection rate, there were 9 hospitals (4,488 patients, 15 infections) with both infection rate and audit score in the lower quartile of the respective distribution (Supplementary Table S2 online). These hospitals all had <100 beds and were predominantly privately run (77.8%).
We set the infection indicator variable to missing for the 4,488 patients, and we multiply imputed 50 complete data sets with additional “new” hypothetical infections based on the observed data (ie, assuming missingness was at random). Each of these completed data sets was aggregated at the hospital level to calculate the NHSN risk index and audit score in exactly the same way as for the primary analysis, and the estimates were then combined using Rubin rules (Supplementary Table S2, column 4: hospital fixed effect [jomo]). The observed data, along with the multiply imputed estimates, were plotted and annotated with the observed and imputed points for each of the 9 hospitals (Fig. 2). The slope of the linear model increased again, but less steeply, and was only marginally significant at the 5% level (P = .02). Other imputation approaches, along with results, are available in the Supplementary Material (online).
Fig. 2. Multiply imputed infection rates for those hospitals with lower audit score and infection rate are shown as black stars (*), with the updated linear model (dashed black [overall slope effect (P = .02); private hospitals only (P = .60), public hospitals only (P = .20)]. Rubric: Per hospital average audit score plotted against average NHSN adjusted infection rate for knee and hip arthroplasty; stratified by type of hospital (left panel private in grey, right panel public in black); bubble sizes are proportional to standard error of the infection rate; linear model shown from the primary analysis (black solid) with 95% confidence interval (shaded grey).
Discussion
Approximately 300 million surgeries take place worldwide each year, and this number is estimated to grow by 38% every 8 years.Reference Abbas and Pittet20 SSIs are a major concern not only in terms of mortality and morbidity but also for patient quality of life and health costs. Turner et alReference Turner and Migaly21 estimated that for the United States, “The true cost of SSI is difficult to quantify given variations in methodology utilizing inconsistent intrinsic and extrinsic costs. National estimates reach >$3 billion annually.”Reference Turner and Migaly21
Given the global significance of such infections, it is of paramount importance that surveillance is carried out in a consistent and quality-assured manner. In this national surveillance of postoperative knee and hip infections and colorectal infections, we observed a wide range of SSI rates and surveillance quality, with discernible clustering based on whether the hospital was public or private. Public hospitals tended to have higher infection rates and audit scores, whereas private hospitals tended to have lower infection rates, but also lower audit scores. In terms of correlation, the analysis indicated that, counterintuitively, infection rate increased with audit score. However, this rate was highly influenced by a concentration of mostly private hospitals with both low infection rate and audit score. We attempted to correct this phenomenon by multiply imputing hypothetical infection rates for those hospitals in which we assumed that the infection rate was being underreported.
For colorectal surgery, there was no discernible clustering of private and public hospitals by infection rate and score, although there was a clear difference in terms of infection rates for such surgeries, with private hospitals having considerably lower infection rates.
The trajectory of the audit score over time increased for both knee and hip arthroplasties and colorectal surgeries, showing that the quality of surveillance improved noticeably with longer experience with surveillance.Reference Abbas, de Kraker and Aghayev22
In terms of methodology, those hospitals having both low infection rates and low audit scores might be considered to be potentially underreporting their infection rate due to lower quality of their surveillance. We proposed multiple imputation as a method for potentially correcting the infection rate based on observed data from other hospitals. Such a “correction” would probably be considered controversial, but sensitivity analysis is commonplace, for example, in clinical trials when investigating the potential effect of missing data on outcome measures.
To our knowledge, no studies have formally correlated SSI rates with audit quality. Nonetheless, a number of studies have validated reported data by auditing. Haley et alReference Haley, Van Antwerpen and Tserenpuntsag23 validated the accuracy of SSIs using 6% of the procedures from 176 hospitals, comparing the data reported to the National Healthcare Safety Network (NHSN). They found the highest numbers of errors reported for colon surgery. In our study, those hospitals performing colorectal surgeries had a slightly higher audit score compared to those performing knee and hip arthroplasties, but this was not a statistically significant difference. A more recent study by Manivannan et alReference Manivannan, Gowda, Bulgonda, Rao, Raman and Natarajan24 highlighted the beneficial effects of performing the surveillance, auditing and feedback process itself, in both orthopedic and cardiac surgeries involving implants. They found that SSI rates were “inversely correlated with the rate of compliance with preoperative [correlation coefficient] r = −0.738; P = .037), perioperative (r = −0.802; P = .017), and postoperative (r = −0.762; P = .028) care bundles,” so that the higher the compliance, the lower the rate. Of course, these results are not directly comparable with those from our study because they consider implementation of clinical care and not audit quality. Nonetheless, we found positive correlation between infection rate and audit quality, which was not what was expected. However, as previously discussed, the effect of potentially underreporting of infection rate, combined with some hospitals performing very few surgeries, skewed our analysis and led to the consideration of sensitivity analysis methods.
Our study has a number of limitations. The surveillance audit pertains to the people recording surgical site infections, their processes, and training in the respective hospital, whereas the SSI rate is for a specific procedure. Thus, the average audit quality score for a hospital lacks the required granularity to enable a fair comparison with the infection rate for a specific surgery type. We arbitrarily used a weighted average of the infection rate in the year previous to the audit and the year in which the audit took place. We made this decision feeling that it struck a balance between simplicity and providing a suitable summary measure of the infection rate when the audit took place.
The NHSN adjusted infection rate is the subject of considerable discussion, with some arguing, perhaps justifiably, for the inclusion of more information summarizing the specific hospital case mix. The results from our study support this, with larger public hospitals having higher infection rates (top right quadrant), mirroring their more complex case load. Estimating the SSI rate for hospitals with a combination of a low number or no infections and audit score, led to the use of multiply imputation to “correct” for underreporting.
This study provides unique insight into the correlation between SSI rates and the quality of surveillance. We present a method to correct for potential underreporting, and this might be included in public reporting in the form of a sensitivity analysis. Surveillance systems without routine evaluation of validity may underestimate the true incidence of SSIs. Audit quality should be taken into account when interpreting SSI rates, perhaps by adjusting infection rates for those hospitals with lower audit scores.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/ice.2021.14
Acknowledgments
The authors thank all participating centers for providing their surveillance data. These data were collected in collaboration with the Swiss National Association for the Development of Quality in Hospitals and Clinics (ANQ).
Financial support
This work was funded by the Swiss National Science Foundation (grant no. 32003B_179500, primary investigator J. Marschall). A.A. is partially funded by the Swiss National Science Foundation (grant no. CRSK-3_190977) and is also employed at the University Childrens Hospital in Basel, Switzerland.
Conflicts of interest
All authors report no conflicts of interest relevant to this article.
Open access
