Research in context
Evidence before this study
Dynamic emergency responses in which the scientific community’s collective knowledge base is limited require innovative methods for rapid development and dissemination of science and clinical practice to health care providers. In early 2020, the global health care community suddenly faced a new disease, Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2), with no prior experience with the disease and its effects. In response to a very limited evidence base for clinical practice, Project ECHO worked with the US Department of Health and Human Services’ (HHS) Office of the Assistant Secretary for Preparedness and Response (ASPR) and the Healthcare Resilience Task Force/Working Group (HRWG) under the Incident Management Structure to create a platform for rapid peer-to-peer learning at scale among front line clinicians across the United States, US territories, and beyond. Reference Bryant, Sosin, Weidrich and Redd1
To understand the existing evidence related to peer-to-peer knowledge development and dissemination in past emergencies, we conducted a systematic review of available literature. We searched PubMed and Google Scholar with no date limitation for search terms including ‘peer-to-peer learning,’ ‘telementoring,’ ‘knowledge dissemination in emergencies,’ ‘crowdsourcing clinical practice in emergencies,’ ‘digital platforms in emergencies or pandemic response,’ and other related terms. We found several references to dissemination of clinical practice during crises, digital learning platforms for health care providers, and uses of technology during emergency responses. Reference Abdel-Massih, Ben-Joseph and Borio2 We also found 1 study from a single hospital system in Switzerland that utilized a digital platform to communicate current knowledge to its providers. Reference Windisch, Zamberg and Zanella3 No other studies were presented on the use of digital platforms for peer-to-peer learning to generate and disseminate evolving clinical practice during health emergencies.
Added value of this study
We demonstrated a successful implementation of a peer-to-peer learning platform that was resilient to constraints presented by the global COVID-19 pandemic. While there may be other studies of unidirectional dissemination of knowledge during emergencies, the HHS ASPR - Project ECHO Clinical Rounds experience demonstrates a scalable method to gather and disseminate insights and experiences from clinical providers in near real time, adapting a tele-mentoring digital health solution to support development of dynamic learning health systems.
Implications of all the available evidence
Due to the increasing threat presented by novel pathogens, 4 an effective method for peer-to-peer learning during low-evidence infectious disease response where perhaps information is ubiquitous, but evidence verified through scientific testing or clinical practice is scarce, is likely to be crucial for future health emergencies. Building upon a previous partnership between HHS ASPR and Project ECHO during the response to Zika, the data collected during this use of the Project ECHO learning model provided early indicators that it is effective for enhancing clinician learning and improving clinical care during low-evidence infectious disease responses. Reference Heard-Garris, Arora and Lurie5
Introduction
On January 1, 2020, the World Health Organization (WHO) activated its Incident Management Support Team to investigate reports by Chinese authorities of atypical pneumonia cases in Wuhan. These cases were the first internationally reported cases of a novel coronavirus disease. By March 11, 2020, the novel coronavirus disease, COVID-19, was declared a pandemic by the WHO, which was followed 2 days later by a declaration of emergency in the United States. 6
In the weeks and months following the declaration of the pandemic, processing and disseminating the evolving information regarding COVID-19 prevention and care presented enormous challenges. Indeed, past health emergencies have demonstrated a deficiency in effective knowledge dissemination to health care workers during emergencies. Reference Dearinger, Howard and Ingram7 While some communication approaches have shown promise in reaching health care workers in localized environments, few to none are known to have the ability to gather clinical practice advice from clinicians while concurrently disseminating the latest known information to them at scale. Reference Abdel-Massih, Ben-Joseph and Borio2
Project ECHO is a continuing education and workforce capacity building initiative developed by the University of New Mexico in 2003. Reference Arora, Kalishman and Thornton9 It is a guided practice model that uses case-based learning to help participants manage their own cases and acquire generalizable knowledge to provide specialized care to patients locally. Reference Furlan, Pajer, Gardner and MacLeod10,Reference Arora, Kalishman and Dion11 HHS ASPR, in collaboration with Project ECHO, more than 20 medical professional societies, and the National Emerging Special Pathogens Training and Education Centers (NETEC), launched a series of Project ECHO COVID-19 Clinical Rounds (COVID-19 Clinical Rounds) on March 24, 2020. The purpose of COVID-19 Clinical Rounds is peer-to-peer, real-time sharing of COVID-19 clinical care challenges and successes, as circumstances rapidly evolve.
History of project ECHO
Project ECHO was launched in 2003 as a health care workforce capacity building initiative, when Sanjeev Arora MD, a liver disease specialist at the University of New Mexico Health Sciences Center in Albuquerque, developed a telementoring platform to scale and decentralize access to services for the state’s hepatitis C patients. Reference Arora, Thornton and Murata12 A New England Journal of Medicine study found that care provided by Project ECHO trained community providers was as good as care provided by specialists at a university. Reference Arora, Thornton and Murata12 These results have been replicated multiple times in diverse contexts ranging from Buenos Aires, Argentina, to Punjab, India, to Native American reservations in the Midwest and Southwestern United States. Reference Mendizabal, Ridruejo and Ceballos13–Reference Page, Qeadan, Qualls, Thornton and Arora17 While traditional telemedicine involves a specialist directly caring for the patient, Project ECHO can be classified as ‘telementoring,’ a ‘guided practice model where the participating clinician retains responsibility for managing the patient.’ 18
The ECHO Institute now supports a global network of partners and programs, with more than 400 partner organizations implementing more than 900 programs in 44+ countries and engaging participants in more than 150 countries. 19 Since COVID-19 Clinical Rounds began, numerous other Project ECHO programs related to COVID-19 have been launched globally (e.g., an Indian Health Service COVID-19 Clinical Readiness and Patient Care series; a USAID supported 1Health Workforce COVID-19 series; an Infection Prevention and Control Global Webinar Series in partnership with the US Centers for Disease Control and Prevention, and the WHO; and a WHO Africa Region COVID-19 Case Management series). 19–Reference Katzman, Tomedi and Thornton21 In this paper, we will focus our study on the first 4 months of the response, from the first session on March 24, 2020 through July 28, 2020. It should be noted that COVID-19 Clinical Rounds continued outside the period of study; as of December 2020, more than 100 ECHO COVID-19 Clinical Rounds had occurred.
Project ECHO COVID-19 clinical rounds
COVID-19 Clinical Rounds are thrice-weekly peer-to-peer telementoring sessions, generally focused once per week on each topic (critical care, emergency department, and emergency medical services (EMS)), with ad hoc special sessions. A wide range of clinicians, including physicians, nurses, and EMS clinicians, attended the sessions. Session topics were selected based on new knowledge and/ or presenter availability, requests from the HRWG, and by assessing clinician demand through registration questions, through discussion and polling in prior sessions, as well as routine CME survey feedback. During each session, 1 or 2 relevant experts describe a case, the state of COVID-19 in their community or facility, their specific clinical or operational challenges, and how they have addressed those challenges. Much of each session is spent discussing questions generated by the participants. Recorded sessions are made available through the Project ECHO website, which serves as a public repository. Reference Carifio and Perla22 Participation details by type are presented in Table 1. The average number of participants per sesssion was 569 participants (n = 52, SD = 350).
Average participant counts by session type (March 24, 2020 through July 30, 2020)
Methodology
In the course of this study, a range of analytical methodologies were used to analyze the data gathered through COVID-19 Clinical Rounds sessions. A primary source of data was Continuing Medical Education (CME) surveys. CME surveys included a series of Likert scale and free-text questions regarding perceptions of presentation quality, information learned, intended use of information learned, and barriers to use of information learned. CME surveys also include an open-ended question around improving the session quality, responses to which were used to improve the COVID-19 Clinical Rounds over time. A total of 7778 survey responses from 52 sessions were received between March 24, 2020, and July 30, 2020 (an average of 150 responses per session).
Impact surveys were also conducted in November 2020. Unlike CME surveys, which were associated with a specific session, impact surveys sought to understand overall impressions of COVID-19 Clinical Rounds from participants who had attended 1 or more sessions since March 2020. Impact surveys were emailed to all 6053 registrants. The COVID-19 Clinical Rounds chat feature was used for 2 weeks in November 2020 to remind participants to complete the survey. Impact surveys asked respondents questions related to 4 topics: (1) rounds’ influence on patient care, (2) changed clinical practices because of rounds, (3) changed operational practices because of rounds, and (4) respondents’ likelihood of joining COVID-19 Clinical Rounds in future emergency scenarios. Each question received between 220 and 260 responses.
Table 2 illustrates additional contextual data available from the sessions.
Additional data sources
Analysis of clinical learning
To assess clinical learning, we analyzed individual-level change in knowledge before and after each COVID-19 Clinical Rounds session, establishing a direct link between a participant’s self-reported learning and their participation in 1 COVID-19 Clinical Rounds session. Additionally, self-reported knowledge data were disaggregated by session type to identify differences in self-reported learning across session types. The Mann-Whitney test was used to determine statistical significance of self-reported knowledge changes before and after COVID-19 Clinical Round Sessions.
Analysis of use of learning
To assess clinicians’ intention to use what they learned during COVID-19 Clinical Rounds, we analyzed quantitative and qualitative data from CME surveys. Respondents were asked to respond to the question ‘How relevant is this session to your current work?’ on a Likert scale from 1 (‘not at all relevant’) through 5 (‘extremely relevant’). Respondents were also asked to respond to the question ‘Will you use what you learned in this session in your work?’ on a Likert scale from 0 (‘N/ A did not learn’) to 5 (‘definitely yes’). We calculated the mean response for both groups as well as the mean response disaggregated by session type.
CME surveys also provided qualitative data on respondents’ planned use of COVID-19 Clinical Rounds. Inductive thematic analysis was conducted to identify themes and patterns. Results represent participant responses from April 2, 2020, to July 30, 2020.
Impact survey responses were aggregated to obtain frequencies for each answer choice on the Likert scale. Respondents were also asked 2 open-ended response questions about changed clinical and operational practices. Inductive thematic analysis was conducted to determine the highest-reported categories of changed practices.
Results
Clinical learning
On average, across all session types, participants rated their knowledge of the session topic after a session 0.43 points higher on a Likert scale ranked from 1 (‘not at all knowledgeable’) through 5 (‘extremely knowledgeable’) than they did before the session (reference Table 4 below for averages). A paired Wilcoxon sign test for non-parametric variables was used to determine statistical significance; through P -values of < 0.0001 we determined there was significant difference between the before/after self-scoring.
Self-reported knowledge before and after COVID-19 Clinical Rounds sessions
Reported intended uses of COVID-19 Clinical Rounds sessions by type (March 24 – July 30)
Reported barriers to use of session information (March 24 – July 30)
Changed clinical practices (reported November of 2020)
Changed operational practices (reported November of 2020)
The project ECHO model.
Improved patient care (reported November of 2020).
Willingness to join ECHO clinical rounds in future (reported November of 2020).
Intention to use clinical learning
The mean of session participant responses to the CME survey question, ‘How relevant is this session to your current work?’ was 3.85 on a Likert scale from 1 (‘not at all relevant’) through 5 (‘extremely relevant’). CME surveys included a Likert scale question asking respondents to rate whether they will use the content of the session from a scale of 0 (‘N/ A did not learn’) to 5 (‘definitely yes’). Most CME survey respondents (75.8%) reported they would ‘definitely yes’ (response = 5) or ‘probably yes’ (response = 4) use content gleaned from the session. The mean response was 4.19.
CME surveys also asked respondents what they planned to use from the session, in free-response format. Of all the forecasted uses, the top specific 15 types of information clinicians planned to use are reported in Table 4.
While many respondents (71.7%) reported there were no barriers to using the information learned in the sessions, a small percentage of respondents indicated there were some barriers to using session information.
Actual use: changed clinical and operational practices
Impact surveys asked respondents, ‘What specific clinical practices have you changed or implemented as a result of this training?’ In some cases, 1 response provided multiple changed clinical practices; thus, changed clinical practices outnumber the actual number of responses (n = 243).
Impact surveys also asked respondents, ‘What specific operational practices have you changed or implemented as a result of this training?’ In some cases, 1 response provided multiple changed operational practices; thus, total changed operational practices outnumber the actual number of responses (n = 223).
In aggregate, 94.6% of respondents reported that COVID-19 Clinical Rounds helped provide better care to patients.
All but 1 impact survey respondent reported that they would join COVID-19 Clinical Rounds again in the event of a future national or local emergency, with 89.9% of respondents indicating they ‘strongly agree’ that they would join COVID-19 Clinical Rounds again.
Testimonies from participating clinicians have cited COVID-19 Clinical Rounds as influential in their decision-making and clinical care. Anecdotal feedback from clinicians is often both positive and tangible. For the April 2020 critical care sessions, 1 survey respondent said, ‘materials and experiences shared [in the session] have already been incorporated into a virtual independent learning module for medical clerks who are currently unable to rotate [with]in [the] hospital. Information on experience with COVID-19 critical care, especially ventilator strategies, was placed in a Medical Mass Casualty Triage in a Critical Access Hospital scenario to guide the clerks’ facilitated study.’ Another said he or she ‘was planning to use information about utilization of non-ICU physicians in the ICU setting with oversight from the ICU attending.’ A participant commented, ‘these clinical exchanges are critical for clinicians to validate what they are seeing and consider when clinical care and research practices/ directions might need to be modified.’ The range of chat responses and CME survey responses support this point: respondents found COVID-19 Clinical Rounds across the range of session topics directly useful in their clinical settings.
Discussion
Through comments in surveys and in the chat feature during the ECHO Clinical Rounds, it is evident that providers found the rounds to be a place to ask questions, get the most up-to-date guidance from the government and their peers, and to discuss the guidance and how it affected their work. The real-time structure of COVID-19 Clinical Rounds allowed federal leaders from HHS ASPR and HRWG to access information on clinical trends, challenges, and successes from clinicians’ perspectives. These bodies utilized information gathered from the rounds to develop up-to-date, topical guidance.
One instance of such guidance relates to the use of Elastomeric Half Mask Respirators (EHMRs), reusable PPE that have been authorized in the health care setting and can be used in place of disposable N95 respirators. Early COVID-19 Clinical Round polling data found that EHMRs were minimally used (15 - 20% of respondents’ organizations). Despite low usage, approximately 75% of respondents expressed a willingness to use EHMRs. The HRWG used this information to strengthen outreach efforts on EHMRs. HRWG outreach on PPE preservation strategies specifically highlights the benefits of EHMRs to provide cost-effective, reusable PPE in the face of N95 shortages. As a result of this information, numerous webinars on the benefits of elastomerics were released and conversations were held with the federal Supply Chain Advisory Group on sourcing elastomerics.
Study limitations
Participants in the COVID-19 Clinical Rounds may differ from the general population of health care providers. Due to limited demographic data on session participants, these differences are unknown. These clinicians may be more heavily impacted by the burden of COVID-19 patients. Clinical shift schedules and scheduled clinical staff meetings were also factors impacting an individual’s attendance.
The analysis conducted on Likert scale data (as in, ‘on a scale of 1 to 5, rate your knowledge of the session topic’) assumed these data to be ordinal data, and thus used non-parametric tests to determine statistical significance. Likert scale data can sometimes be classified as parametric, however given the subjective nature of Likert scale questions based on participant, the non-parametric route was chosen, even though non-parametric tests are slightly less powerful than parametric tests.
Each session’s participant responses to surveys or other questions may be subject to social desirability bias. This bias may be amplified when participants are offering data connected with their registration or discussing topics publicly by chat. Some attendees may have watched as a group instead of individually, potentially encouraging discussion among the viewing group that was not captured in the session chat. In the survey and other data, there were no unique identifiers for participants. As participants could join multiple sessions each week and several clinicians could join on 1 computer connection, it was not possible to calculate the number of unique participants across sessions although unique participants in each individual session was recorded.
Conclusions
The successful implementation of the HHS ASPR and Project ECHO COVID-19 Clinical Rounds as a peer-to-peer learning platform, which was resilient to restrictions presented by the global COVID-19 pandemic, offers a promising model to governments, and to the domestic and international communities. While the COVID-19 Clinical Rounds launched less than 2 weeks after HHS ASPR requested support from Project ECHO, it took effort to establish the COVID-19 Clinical Rounds. The relationships already established among Project ECHO and HHS ASPR, developed initially during the Zika epidemic, and between HHS ASPR, and the professional society partners, were instrumental in HHS ASPR’s ability to rapidly establish such a peer-to-peer learning platform and attract so many practitioners to sessions. 4 The presented evidence of clinician learning and clinician intent to use information gained from COVID-19 Clinical Rounds sessions offer early insight into the effectiveness of this scalable method to gather and disseminate insights and experiences from clinicians in near to real time. The use of clinician information for a national response demonstrates the benefit of the telementoring model in concert with national and/or international governments and organization response. Developing partnerships to support increased national and international cooperation and creation of dynamic peer-to-peer telementoring platforms and communities can be an impactful step toward preparing for and responding to future national and international emergency responses. Such is especially the case for low or no-notice response where scientifically tested or clinically verified practice evidence is limited.
Open access
