INTRODUCTION
Utilization of computed tomography (CT) scans in the emergency department (ED) has increased by more than 300% over the last two decades.Reference Kocher, Meurer, Fazel, Scott and Krumholz 1 Studies have demonstrated that emergency physicians (EPs) vary in their test-ordering behaviour.Reference Jain, Elon, Johnson, Frank and Deguzman 2 - Reference Marin, Weaver and Barnato 7 Inter-physician variation in the ordering of any test may introduce unnecessary costsReference Cutler, Skinner, Stern and Wennberg 8 or subject a patient to the risks of over-diagnosis, under-diagnosis, false positives, or further unnecessary testing.Reference Patz, Pinsky and Gatsonis 9 - Reference Thompson, Wojcik, Grant and Ko 11 Variation in the use of CT scans is of special significance because CT scans expose patients to ionizing radiation at doses that likely increase a patient’s lifetime risk of cancer.Reference Brenner and Hall 12 - Reference Eisenberg, Afilalo and Lawler 17 This practice variation may imply that some patients are being exposed to radiation unnecessarily.
Inter-physician variation has significant cost implications. Within a given department, test ordering may vary by two- to threefold, with corresponding variation in length of stay, admission rate, and downstream testing.Reference Klassen, Reed and Stiell 3 , Reference Levine, Moore, Franck, Li and Kuehl 4
While studies have demonstrated variation in CT ordering among EPs, these studies have not determined whether individual physicians are aware of their utilization rates.Reference Klassen, Reed and Stiell 3 - Reference Marin, Weaver and Barnato 7
Proposed drivers of physician variation include fear of malpractice and physician risk tolerance.Reference Katz, Williams and Brown 18 - Reference Wong, Kowalenko and Roahen-Harrison 24 However, physician variation is not completely understood.Reference Rohacek, Buatsi and Szucs-Farkas 23 , Reference Feinglass, Martin and Sen 25 The objectives of this study were to describe variation in CT-ordering rates among EPs, to measure the correlation between perceived and actual CT-ordering rates, to assess attitudes that might influence a decision to order an imaging test, and to identify which of these attitudes are associated with higher CT utilization.
METHODS
Study design
This study included a retrospective review of imaging and administrative billing records at two ED sites of a tertiary-care adult teaching hospital (Ottawa, Ontario), and a cross-sectional survey of EPs at this hospital. We obtained approval from The Ottawa Hospital Research Ethics Board.
Study setting and population
The Ottawa Hospital includes two urban EDs with a combined 140,000 annual patient visits and is a referral centre for the region of Eastern Ontario. One site is the regional trauma and stroke centre; the other site is the regional oncology centre. The hospital is the principal teaching site for an emergency medicine residency. Medical students and junior residents from other specialties rotate through the ED. The study population included all patients attending the ED during the study period, as well as all the EPs employed at this institution during that time.
Study protocol
Departmental billing records were used to determine the number of patients seen per EP for the months of January, March, June, and September of 2012. These months were selected due to the availability of detailed billing data. An imaging database, the Picture Archiving and Communication System (PACS), was employed to determine the number of CTs ordered by each EP. To order a CT, EPs or residents completed paper requisitions that were scanned into PACS. We searched PACS by ordering physician and then read the scanned requisition to minimize errors in order attribution. If no attending physician was listed on the requisition, the CT was excluded from our analysis.
Although most residents at our institution discuss CT ordering decisions with attending EPs, senior emergency medicine residents in their 3rd, 4th, or 5th year might not. CTs ordered after 11:00 pm were often delayed until morning, but were still attributed to the original ordering physician.
We defined a single CT study by the scan protocol at our institution. For instance, a CT of the abdomen and pelvis was considered a single study. If a patient had more than one CT ordered with different protocols (e.g., CT head and CT pulmonary angiogram), we counted both CTs.
We did not review the ED charts or collect patient data (e.g., ESI/CTAS score, patient age, diagnosis).
The survey was conducted in three stages. First, study authors chose, by consensus, factors that might plausibly influence a physician’s decision to order a CT. Second, the survey was pilot-tested with senior emergency medicine residents to ensure comprehensibility, and to obtain feedback on redundant questions, which we eliminated. Third, the paper survey was delivered to the hospital mailbox of all EPs at The Ottawa Hospital in June 2013. The survey had 21 questions and took about 5 minutes to complete. Two reminder emails, to encourage completion, were sent monthly after initial survey distribution. Physicians who completed the survey were entered in a draw for a $50 gift certificate. The final survey included: 1) demographic data; 2) 10 self-reported factors that influence physicians’ ordering practices rated on a 4-point Likert-type scale; 3) five beliefs on CT utilization rated on a 5-point Likert-type scale; and 4) an estimate of the perceived rate of self-ordering.
Data analysis
Ordering rates were calculated for each physician and expressed as a cumulative percentage over the study period. The average across physicians was calculated as an unweighted mean. The ED duty schedule at The Ottawa Hospital includes four different shift types: urgent (ambulatory care); observation (stretchers without cardiac monitors); resuscitation (trauma, resuscitation, and cardiac monitoring); and night (entire department). We adjusted each physician’s ordering rate to account for differences in shift distribution, and to identify the “outlier” status of physicians, using the approach described by DeLong et al.Reference DeLong, Peterson and DeLong 26 We constructed a random-effects logistic regression model in which shift type was a fixed covariate, and physicians were considered random effects, using SAS software (v. 9.1). The physician random effects were estimated as best linear unbiased predictors, which are shrunken estimates, closer to the mean CT-ordering rate, and are more robust against small sample sizes. The random-effects estimates obtained from the logistical regression model were expressed as observed versus expected ratios of CT-ordering rate, with their 95% confidence intervals, for each physician. Physicians whose confidence intervals did not include 1 were considered outliers.
The correlation between perceived and actual CT-ordering rates was calculated using Pearson’s correlation coefficient, as well as using mean difference with a 95% confidence interval (CI). Physicians were categorized into quintiles based on their actual ordering rates and compared to the perceived rates based on survey responses.
We used multivariable regression to identify important independent predictors of higher CT utilization.Reference Harrell 27 Physicians who had seen fewer than 300 patients during the study period were excluded from the multivariable model. Potential predictors of higher CT utilization were first assessed using bivariate tests of association; in particular, demographic characteristics (physician sex, categorized age) were assessed using chi-squared tests. Ten survey factors measured on a Likert-type scale were analyzed as ordinal variables, in which 0=not influential and 3=heavily influential, and assessed using chi-squared tests/Cochran–Armitage trend tests. Shift type was analyzed as a categorical variable and assessed using chi-squared tests. Multicollinearity among candidate predictor variables was assessed using a variance-clustering algorithm,Reference Nelson 28 and variables involved in near-linear dependencies were excluded from consideration. Variables significantly associated (p<0.05) with CT utilization were considered for entry into a multivariable generalized linear regression model using the log-link function and Poisson distribution. The CT-ordering rate for each shift type was entered as the dependent variable, and the natural log of the number of patients seen was specified as an offset term. We accounted for over-dispersion by using a Pearson scale parameter. Stepwise backward elimination with an alpha level for staying in the model set at 0.2 was used to determine independent predictors of CT utilization. Regression coefficients were estimated as relative risks with CIs. All analyses were completed using Statistical Analysis Software (v. 9.3; SAS Institute, Cary, NC).
RESULTS
Physician characteristics
CT ordering data were available for 59 physicians, of whom 45 (76%) completed the survey. Another six physicians completed the survey, but CT ordering data were unavailable for them (due to recent entry to practice or leave from clinical duties). Demographic data were therefore available for 51 EPs, which are summarized in Table 1.
Characteristics of physicians included in the study (n=51)
CCFP (EM)=Canadian College of Family Physicians–Emergency Medicine certificate (three-year training program); FRCPC=Fellow of the Royal College of Physicians of Canada (five-year training program).
Chart review findings and actual ordering rates
Unadjusted CT-ordering rates are displayed in Figure 1. During the study period, 59 EPs saw 45,854 patients and ordered 6,609 CTs—an ordering rate of 14.4% (standard deviation (SD)=4.3%). The average numbers of patients and CTs per physician were 777 and 112, respectively. The ordering rate for individual physicians ranged from 5.9 to 25.9%. There were 11 physicians whose CT-ordering rate was >1 SD (or 3 physicians whose CT-ordering rate was >2 SD) above the mean.
Variation in CT-ordering rates among emergency physicians.
The ratios of observed to expected CT-ordering rates, adjusted for shift type, are presented in Figure 2, with their CIs. MDs are numbered in the same order as in Figure 1. In this analysis, 13 physicians were low-ordering outliers, and 12 physicians were high-ordering outliers. Three physicians whose unadjusted rate was in the lowest quintile and two physicians whose unadjusted rate was in the highest quintile were not outliers in this analysis. This likely reflects uneven shift allotment within the department. Seven physicians who saw fewer than 300 patients during the study period were excluded from the multivariable model, but none were outliers.
Standardized CT-ordering rate with 95% confidence limits.
Survey findings
CT data were available for 59 of 65 EPs. Of those, 45 completed the survey, for an overall survey response rate of 76.3%. The mean perceived ordering rate was 12.6% (SD=7.1). As anticipated, the perceived ordering rate was weakly correlated with actual ordering for individual physicians (r=0.19, CI: –0.11 to 0.46, p=0.21) (Figure 3).
Scatter plot of actual versus perceived CT-ordering rates.
The distributions of physicians, categorized into quintiles of perceived and actual CT ordering, are depicted in Figure 4.
Perceived versus actual ordering rates, by quintile, from lowest (Q1) to highest (Q5).
Only 29% of EPs were able to predict their ordering behaviour to the correct quintile.
From 45 EPs, 42 responders (93.3%) believed they ordered “about the same” or “fewer” CTs than their peers. Of the 17 EPs in the two highest-ordering quintiles, only 3 (18%) knew they were high CT utilizers (Figure 4).
The self-reported factors that influenced physicians’ decisions to order CTs are presented in Figure 5.
Self-reported factors that influence physicians’ decision to order CT.
Emergency physicians’ views on CT ordering are presented in Figure 6. A total of 15 (29%) of respondents disagreed that they should order fewer CTs, but only 5 (10%) disagreed with the statement that colleagues should order fewer CTs. A total of 9 (18%) agreed that they should order fewer CTs, whereas 17 (33%) agreed that colleagues should order fewer CTs. The majority of physicians agreed that CT increases the risk of cancer (90%) and that EPs order more CTs than they used to (96%).
Physicians’ views on CT ordering.
Multivariable regression analysis
The following variables were not significantly associated with CT-ordering rates using bivariate tests of association: physician gender (p=0.42), age (p=0.19), residency program (p=0.20), and residency location (p=0.69). These variables were therefore excluded from the multivariable model. Moreover, the Likert-type scale responses to “lifetime number of CTs a patient already had” and “risk of radiation” were collinear, as were “ED flow” and “medico-legal risk.” We therefore excluded “lifetime number of CTs a patient already had” and “ED flow” from consideration in the multivariable model. Higher Likert-type scale responses to the following predictors were associated with higher ordering rates: medico-legal risk (RR=1.18, CI: 1.03–1.21); risk of contrast (RR=1.14, CI: 1.07–1.22); what colleagues would do (RR=1.09, CI: 0.99–1.19); risk of missing a diagnosis (RR=1.08, CI: 0.98–1.21); and patient wishes (RR=1.07, CI: 0.97–1.17).
DISCUSSION
We demonstrate important variation in CT-ordering rates and that a subset of physicians order far more CT scans than their colleagues. This is consistent with the existing literature.Reference Levine, Moore, Franck, Li and Kuehl 4 , Reference Prevedello, Raja and Zane 6 , Reference Kim, Sidney, Klingman and Johnston 29
This study also examined physicians’ self-awareness of their test-ordering behaviour. To our knowledge, this is the first ED-based study to address this question. When physicians were asked to estimate their absolute rate of CT ordering (as a percentage), we found only a weak correlation with their actual rate of ordering. This suggests that physicians have a poor awareness of their personal ordering rate.
Although physicians might not be able to estimate their absolute ordering rate, we considered that they might know how they compare to peers. When asked to rank their ordering rate compared to colleagues, the majority of physicians believed that their practice was in the middle quintile. Only 3 of 45 physicians believed they fell in the upper two quintiles.
Despite this apparent lack of self-awareness, physicians were aware of the risks of radiation. Nonetheless, only 18% felt that they should order fewer CT scans. In comparison, 33% of physicians felt that their colleagues should order fewer CT scans. These data suggest that few physicians in this study recognized over-utilization in their own practice, but rather attributed it to others. Physicians clearly believe that the tests they order are appropriate. We were unable to assess the appropriateness of the CT scans ordered in the study, as we did not collect clinical information on the patients. Previous studies, however, suggest that positive findings are no more common in high-ordering centres,Reference Klassen, Reed and Stiell 3 , Reference Stiell, Wells and Vandemheen 30 so that some CT scans are likely inappropriate.
The problem of variation in physician practice remains vexing. Numerous studies and several reviews have evaluated strategies to improve physician test-ordering behaviour.Reference Oxman, Thomson, Davis and Haynes 31 , Reference Solomon, Hashimoto, Daltroy and Liang 32 Approaches to modifying physician behaviour include the use of clinical practice guidelines, academic detailing, audit and feedback, pay-for-performance, and computer-based decision support, but no strategy has demonstrated consistent success in all settings.Reference Hartig and Allison 33 Our findings suggest that the highest CT orderers did not realize that they ordered more than their colleagues. Although peer comparisons have not been shown to be useful for changing practiceReference Jamtvedt, Young, Kristoffersen, O’Brien and Oxman 34 in other ways, high-ordering physicians might be more willing to reduce the use of CT, as it has the potential to directly harm patients.
EDs should also consider their aggregate utilization compared to comparator hospitals. Although different hospitals serve different populations, it is unlikely that patient factors can account for all institutional variability,Reference Klassen, Reed and Stiell 3 , Reference Mannix, Meehan, Monuteaux and Bachur 5 , Reference Marin, Weaver and Barnato 7 , Reference Cutler, Skinner, Stern and Wennberg 8 , Reference Stiell, Wells and Vandemheen 30 and there is room for entire departments to change their practice.
In the absence of clear evidence-based approaches to the evaluation of many chief complaints, some inter-physician variation in practice is to be expected. This study, however, demonstrates that measuring raw ordering rates alone may not be sufficient to identify outlier physicians, as some EPs in our cohort with apparently high or low ordering rates were not outliers once we adjusted for shift distribution. Adjusting for individual patient factors might also be considered, but this was beyond the scope of our study. Nonetheless, our study, like others, suggests that there is a subset of high-ordering physicians who might benefit from measures to reduce utilization. There is also a subset of low-ordering physicians. There is some evidence to suggest that low orderers do not miss important diagnoses,Reference Stiell, Wells and Vandemheen 30 but more research is necessary.
We attempted to find predictors of high CT ordering using muiltivariate regression analysis. We did not find that physician age, sex, and training program were associated with higher CT ordering. However, some responses on our survey of physician attitudes predicted higher rates of CT ordering. In our reduced multivariate model, higher CT ordering was predicted by a stronger response to questions rating concern for medical-legal risk, risk of contrast, what colleagues would do, patient wishes, or risk of missing a diagnosis. These findings are not all intuitive. In Canada, medico-legal risk is comparatively low compared to that in the United States,Reference Fleeter 35 but medico-legal concerns were still our strongest predictor of increased imaging. Concern regarding risk of contrast should predict decreased CT ordering, but, paradoxically, we found the opposite.
Factors that have previously been shown to be associated with test utilization include the patient’s geographic location,Reference Klassen, Reed and Stiell 3 , Reference Mannix, Meehan, Monuteaux and Bachur 5 , Reference Cutler, Skinner, Stern and Wennberg 8 , Reference Stiell, Wells and Vandemheen 30 , Reference Couchman, Forjuoh, Reis, Bartels and Lindzey 36 race,Reference Pezzin, Keyl and Green 37 , Reference Harris 38 insurance status,Reference Couchman, Forjuoh, Reis, Bartels and Lindzey 36 and physician involvement in guideline development.Reference Verstappen, ter Riet and Dubois 39 Physician risk tolerance and fear of malpractice have also been associated with differences in test utilization.Reference Katz, Williams and Brown 18 , Reference Pines, Hollander and Isserman 19 , Reference Cheng, Wu, Syue, Yen and Wu 22 - Reference Wong, Kowalenko and Roahen-Harrison 24 A study of internal medicine physiciansReference Feinglass, Martin and Sen 25 found that variation in practice style accounted for a 40% variation in hospital charges.
There is unlikely to be a single solution to the problem of physician variation in test utilization.Reference Oxman, Thomson, Davis and Haynes 31 This study demonstrates that physicians have poor awareness of how their test ordering compares to that of their colleagues. We argue that it is unlikely that physicians with high levels of utilization will be willing to change test-ordering behaviour without feedback. Improved physician self-awareness could be part of the solution.
LIMITATIONS
This study is limited by data collection from two EDs within a single Canadian university system. Our results may not be generalizable to other hospitals or other countries. Although we collected data on over 6,500 CTs, the average number of CTs per physician was 112.
Some CTs might have been ordered by residents without prior approval by their attending physicians. Nevertheless, we believe that the effect on CT-ordering rate would have been small. Most CT-ordering decisions are discussed with attending physicians. Given the size of our department and residency program, attending physicians work with senior residents on a minority of shifts. Residents are not preferentially assigned to any physician, so the effect of senior resident ordering should be evenly distributed.
The survey included responses from only 45 physicians; a larger number of physicians might have produced a stronger multivariate model. Physicians who did not respond to the survey might have held different attitudes toward CT ordering from those who responded, and this could have affected the results of our multivariable regression. In our model, risk of contrast was associated with higher, not lower, ordering rates. This may represent a limitation of the model, as risk of contrast should be a deterrent to CT ordering.
Finally, the factors that physicians rated on our survey instrument were chosen by consensus among the investigators. It is possible that other factors we did not measure would be more predictive. It is also possible that physician self-report of the importance of a factor is an inaccurate representation of their behaviour.
CONCLUSIONS
There is large variation in CT ordering among EPs—in our centre, we measured a fourfold variation between the lowest and highest utilizers. Physicians’ self-reported ordering rates correlate poorly with actual ordering. Only a small proportion of high CT orderers were aware that they ordered more than their colleagues. Higher rates of ordering were observed among physicians who reported increased concern with: 1) risk of missing a diagnosis, 2) medico-legal risk, 3) risk of contrast, 4) patient wishes, and 5) what colleagues would do.
Acknowledgements
Aspects of this research study were presented in abstract form at the Meeting of the Society of Academic Emergency Medicine (SAEM), Dallas, Texas, May 2014 and the Meeting of the Canadian Association of Emergency Physicians (CAEP), Ottawa, Ontario, June 2014.
Competing interests: This study was funded through a research grant from the Department of Emergency Medicine, University of Ottawa, and through a student research grant from the Faculty of Medicine, University of Ottawa. The authors have no financial relationships that may represent a conflict of interest.
