BACKGROUND

Computerized provider order entry (CPOE) has emerged in North America as a means to standardize and improve the delivery of health care services to patients.Reference Kuperman and Gibson ⁴ CPOE involves the utilization of electronic systems to track health care provider orders but can also include more advanced functionality such as clinical decision support.Reference Miller, Waitman and Chen ⁵

The debate surrounding CPOE has focused on the clinical benefits versus costs of implementation. Studied clinical benefits of CPOE include reduced prescribing errors and adverse medication interactions,Reference Nuckols, Smith-Spangler and Morton ⁶ improved adherence to evidence-based protocols for specific presentations, such as renal colic and acute ischemic stroke,Reference Blankerfield, Rogers and White ^{7 -} Reference Yang, Park and Chung ⁹ increased legibility and accessibility of documentation on record,Reference Cresswell, Bates and Williams ¹⁰ and potential secondary uses of data by health care organizations for outcome tracking and quality assessment.Reference Cresswell, Bates and Williams ¹⁰ CPOE’s benefits, however, may come at the cost of decreased patient and physician satisfaction,Reference Bastani, Walch and Todd ^{11 ,} Reference Shanafelt, Dyrbye and Sinsky ¹² impaired emergency physician productivity,Reference Schiff, Amato and Egulae ¹³ and increased length of stay (LOS) in the emergency department (ED) for admitted patients.Reference Spalding, Mayer and Ginde ¹⁴ There is no strong evidence to suggest that CPOE improves patient mortality,Reference Brunette, Tersteeg and Brown ¹⁵ and the literature has not yet provided consensus evidence in favour of CPOE when weighing its benefits and drawbacks.Reference Georgiou, Prgomet and Paoloni ¹⁶

ARTICLE SUMMARY

Gray and colleagues conducted a retrospective cohort study to evaluate the impact of CPOE implementation on ED workflow at a quaternary care centre in London, Ontario, comprising two separate ED campuses. They adopted a pre-post implementation design to evaluate the effect of CPOE on three primary ED flow metrics: LOS, wait time (WT), and the proportion of patients who left without being seen (LWBS).

The study included all ED patients 18 years and older who were triaged during July and August 2013 (pre-implementation of CPOE) and July and August 2014 (post-implementation of CPOE). Data were extracted from the London Health Sciences Centre (LHSC) electronic database and health records. Any patients with incomplete or incorrect ED charts, negative WTs or LOS, extreme outliers with WT > 24 hours, or those missing vital statistics were excluded. The investigators also completed a subgroup analysis looking at variability by the Canadian Triage and Acuity Scale (CTAS) stratification and admitted patients to evaluate whether CPOE had differential effects based on acuity of illness or disposition status.

The authors analysed a combined data set of 36,758 ED visits: 18,872 visits in 2013 and 17,886 visits in 2014. Median age, gender distribution, CTAS stratification, and rate of admission were similar between the two groups at baseline. Statistical analyses were conducted to determine significant changes in WT, LOS, and LWBS between pre- and post-implementation of CPOE.

KEY RESULTS

The authors concluded that CPOE implementation detrimentally impacted patient flow in the two EDs that they studied. They found statistically significant increases in median WT (increased from 78 to 83 minutes), median LOS (increased from 254 to 264 minutes), proportion of patients who LWBS (increased from 7.2% to 8.1%), and the LOS for admitted patients (increased from 713 to 776 minutes) (Table 1).

Table 1

Key results from Gray et al.’s CPOE studyReference Gray, Fernandes and Van Aarsen ³

QUALITY ASSESSMENT

The study by Gray et al.Reference Gray, Fernandes and Van Aarsen ³ was a retrospective cohort design comparing ED patient flow pre- and post-implementation of CPOE using administrative data. The authors had clearly defined outcomes and collected data with objective variables. The inclusion and exclusion criteria were appropriate and acceptable for cohort recruitment. Nevertheless, this study had several limitations.

The CPOE in this study was introduced to the two EDs in April 2014, only 3 months prior to the data sampling set for post-implementation. The results seen in this study could potentially be confounded by inexperience and lack of adequate training or uptake during the early stages of implementation. It would be useful to collect data further from implementation to determine whether user familiarity with CPOE reduces its impact on workflow over time given previously described learning curves with CPOE implementation.Reference Abramson, Patel and Malhotra ¹⁷

The retrospective cohort design encompasses potential sampling errors that limit the generalizability of the authors’ findings.Reference Kaji, Schriger and Green ¹⁸ Gray et al.Reference Gray, Fernandes and Van Aarsen ³ drew data from only July and August during the pre- and post-implementation phases of CPOE in one Canadian city. It is difficult to state whether this would be a representative sampling for a number of reasons.

First, the authors did not collect or report on potential confounding characteristics between the pre- and post-implementation time periods such as mean boarding times, hospital over-capacity statistics, population and ED utilization rates, and bed allocation resources.

Second, the choice of July and August for data extraction may have impacted outcomes, given that July 1 marks the beginning of residency for new trainees in Canada, and junior residents in the ED environment would be adjusting to CPOE in addition to new clinical responsibilities. The existence of the “July effect” (whereby medical trainees beginning new clinical roles affect medical error rates, patient outcomes, and departmental productivity) remains uncertain in the literature.Reference Young, Ranji and Wachter ¹⁹

Third, in the post-implementation group, a substantial proportion of patient data was excluded, mostly due to clerical and administrative data error. It is impossible to determine what impact this may have had on the results.

Finally, CPOE software programs can be highly variable in their design, implementation strategy, and user interfaces. Some CPOE programs may include components of clinical decision support and order set bundles to prompt evidence-based reminders during clinical encounters, whereas others may simply be a collection of individual medications and orders requiring manual entry. Without further detail on the specifics of the CPOE design and implementation in the authors’ study, it is difficult to draw parallels between this study’s results and CPOE use in other centres.

TAKE-TO-WORK POINTS

CPOE implementation may statistically impair ED flow metrics, although this is of uncertain clinical significance given the slight differences in flow metric results (5 and 10 minutes’ difference in median WT and LOS, respectively) and the large interquartile ranges measured. The initial stages of implementing CPOE may lead to increased ED WTs, LOS, and more patients leaving without being seen by an emergency physician. The generalizability of Gray et al.’s studyReference Gray, Fernandes and Van Aarsen ³ is uncertain because the impact of CPOE on ED work flow may be highly dependent on both software characteristics and implementation strategy. Further studies looking at the long-term outcomes of CPOE implementation are still needed.

METHODOLOGY OF THE SOCIAL MEDIA RESPONSE ANALYSIS

The social media discussion started with the launch of the blog post and podcast on July 5, 2016, and continued for 2 weeks until July 19, 2016. An invitation to comment on the article was included in the audio of the podcast, the text of the blog post, and on social media communications (Twitter and Facebook). Social media responses written in the SGEM blog’s comment section, the SGEM Facebook page, and on Twitter (directed at @thesgem, @socmobem, or using the #SGEMHOP hashtag) between July 5 and July 19, 2016, were reviewed by the authorship team. KL compiled and reviewed all aforementioned social media commentary to identify tweets and posts related to the CPOE SGEM podcast and blog post. A thematic analysis was conducted using a qualitative framework approach as outlined below.

Framework approach for thematic analysis:

• ∙ Provisional classification: content from each of the various analysed social media platforms was classified as either promotional (i.e., containing only a link to the blog post with no further content) or commentary-based.

• ∙ Thematic framework development: each commentary-type item was evaluated individually to identify key issues, concepts, and themes raised.

• ∙ Indexing: commonly identified themes across all of the commentary-type items was compiled and coded with short phrases for ease of comparison and tracking.

• ∙ Charting: the thematic framework was organized into a comparison chart presented in Table 2.

• ∙ Mapping and interpretation: as soon as common thematic groupings were identified and a comparison chart was created, all authors participated in a consensus-based analysis to determine which comments were most representative of the general themes of the discussion.

Table 2

Framework approach for thematic analysis – comparison chart

Multiple metrics of dissemination were further tracked by the SGEM HOP team for analysis:

• ∙ Blog post page views were monitored using the Jet Pack plugin by Wordpress.com (available from https://wordpress.org/plugins/jetpack/).

• ∙ Facebook “reach” analytics were provided by Facebook and represented the number of users who saw the original SGEM Facebook post on their own newsfeeds. ²⁰

• ∙ Twitter impressions (the number of users whose newsfeeds contained a tweet featuring the #SGEMHOP hashtag) were tracked using Symplur, a software program that monitors health care related twitter conversations.Reference Symplur ²¹ Tweets not containing the hashtag were not tracked by Symplur. The number of impressions was calculated by taking the number of tweets per twitter user using the #SGEMHOP hashtag and multiplying it by the number of followers that each participant had.

• ∙ The altmetric score is a proprietary, standardized tool that tracks the disseminative impact of research articles in social media forums (e.g., Facebook, Twitter) and on blogs, podcasts, and news outlets. ²² The altmetrics of the featured article by Gray et al.Reference Gray, Fernandes and Van Aarsen ³ (Figure 1) were compared to all other articles published in CJEM, all published research analysed by altmetrics, and the articles covered in the first three CJEM-SGEM HOPs.Reference Luckett-Gatopoulos, Thoma and Milne ^{23 -} Reference McKenna, Thoma and Milne ²⁵

Figure 1

Screen capture of the altmetrics data retrieved August 3, 2016, from https://cambridgejournals.altmetric.com/details/9299925.

RESULTS OF THE SOCIAL MEDIA RESPONSE

Table 3 provides details on the social media reach of Gray et al.’s articleReference Gray, Fernandes and Van Aarsen ³ during the SGEM HOP campaign. During the 2-week period following the podcast release, #SGEMHOP was used in 84 tweets by 43 individual users, representing 216,926 Twitter impressions. Thirty-four of these tweets were from the study’s authors or CJEM personnel prompting social media responses. Online conversation through Twitter and the SGEM blog remained active for 14 days following podcast release. The altmetrics score for Gray et al.’s articleReference Gray, Fernandes and Van Aarsen ³ was 41, placing it 10th highest amongst all previous CJEM publications and within the top 5% of all published research. Prior articles featured in the SGEM HOPs series scored in the 46–71 range.Reference Luckett-Gatopoulos, Thoma and Milne ^{23 -} Reference McKenna, Thoma and Milne ²⁵ The mean altmetrics score for CJEM publications is 5.9. ²⁶

Table 3

Aggregate analytic data of social media platform discussions following the SGEM blog posting

ONLINE DISCUSSION SUMMARY

The majority of the discourse was held directly on the SGEM blog with dissemination of the URL link through Twitter, Facebook, and Google +. The SGEM blog hosted a total of 17 comments from 9 discrete users, including direct feedback from the study’s primary author, Dr. Andrew Gray.Reference Milne ² Clinicians across Canada engaged in conversation to share their experiences with CPOE implementation at their institutions. Consistently identified themes included the limited generalizability of the study’s results due to confounding factors, variability of CPOE software design across centres, and the importance of a collaborative implementation strategy when introducing CPOE to a department.

There were multiple confounding variables that could have influenced the study’s results. Dr. Eddy Lang, Academic Department Head for Emergency Medicine at the University of Calgary, cautioned that the study’s results needed to be considered within a systemic context of ED overcrowding: “CPOE may have an association with increased [length of stay] for admitted patients but without adjustment for boarding and [emergency inpatients] there is no way to know if any relationship exists.” This was consistent with the discussion held on the SGEM podcast, where the authors’ lack of reporting for important potential confounding factors was addressed. Variables, such as boarding time and proportion of emergency inpatients, were not tracked between the pre- and post-implementation time periods.

Physicians across the country shared their experiences with CPOE implementation and highlighted the variability of CPOE software across different centres. Dr. Wurster felt that implementation of CPOE at his centre was “nothing short of a disaster. […] Our training was not adequate and we were given little input in how we would like the computer system to work for us. No one examined how using CPOE would affect our daily processes in the emergency department.” This was contrasted with Dr. Lalani’s experiences with CPOE in Calgary, where physician input is a critical component of CPOE development and maintenance: “I really value my colleagues […] who have worked so hard to generate easily workable order sets that include things like ‘Well’s Score’ for VTE. I like that if I type ‘ED meningitis’ I get all the orders for this presentation (including CT and all four LP tubes pre-labelled), and I like that if I go to any of the four hospitals [in Calgary] that I work at, it’s the exact same process.”

Finally, the importance of a collaborative implementation strategy was summarized by Dr. Shawn Dowling, an emergency clinician-researcher with a funded role for maintenance and optimization of CPOE order sets in Calgary: “[CPOE’s] success and failure is dependent on the components (order sets), its fit (usability of the software), the engine (physician buy in/engagement) and its mechanics (the team responsible for maintaining and optimizing the CPOE environment). CPOE is most powerful when it’s part of an integrated knowledge translation strategy.”

LIMITATIONS OF SOCIAL MEDIA ANALYSIS

There are a number of inherent limitations to any study evaluating social media engagement. The utilization of social media data involves sampling bias, and we recognize that the online discussion summarized in this paper likely represents the subset of emergency physicians who are heavily engaged in free open access medical education (FOAMed) activities. As such, the opinions of the wider audience may not have been captured through the social media platforms analysed.

Furthermore, commonly used social media analytic scores, such as the number of Twitter impressions or the altmetrics score, are useful for quantifying article dissemination, but do not necessarily provide information on the quality of the discourse generated.Reference Eysenbach ^{27 -} Reference Fox, Gurary and Ryan ²⁹ While the blog post hosted informative discussion around Gray et al.’s publication,Reference Gray, Fernandes and Van Aarsen ³ virtually all of the tweets containing the #SGEMHOP hashtag and various Facebook posts simply directed readers to the blog itself and did not directly add to the discussion.

We have summarized the social media dialogue in this paper; however, it remains important for readers to independently evaluate the primary literature and continue to critically appraise the social media feedback.