Hostname: page-component-797576ffbb-jhnrh Total loading time: 0 Render date: 2023-12-02T01:17:16.440Z Has data issue: false Feature Flags: { "corePageComponentGetUserInfoFromSharedSession": true, "coreDisableEcommerce": false, "useRatesEcommerce": true } hasContentIssue false

The use of free online educational resources by Canadian emergency medicine residents and program directors

Published online by Cambridge University Press:  01 April 2015

Eve Purdy
School of Medicine, Queens University, Kingston, ON
Brent Thoma*
Learning Laboratory and Division of Medical Simulation, Department of Emergency Medicine, Massachusetts General Hospital, Boston, MA Emergency Medicine, University of Saskatchewan, Saskatoon, SK
Joseph Bednarczyk
Department of Emergency Medicine, University of Manitoba, Winnipeg, MB
David Migneault
Department of Emergency Medicine, University of British Columbia, Vancouver, BC
Jonathan Sherbino
Division of Emergency Medicine, McMaster University, Hamilton, ON
Correspondence to: Dr. Brent Thoma, Emergency Medicine Residency Program, Room 2686, Royal University Hospital, 103 Hospital Drive, Saskatoon, SK S7N 0W8; Email:



Online educational resources (OERs) are increasingly available for emergency medicine (EM) education. This study describes and compares the use of free OERs by the Royal College of Physicians and Surgeons of Canada (RCPSC) EM residents and program directors (PDs) and investigates the relationship between the use of OERs and peer-reviewed literature.


A bilingual, online survey was distributed to RCPSC-EM residents and PDs using a modified Dillman method. The chi-square test and Fisher’s exact test were used to compare the responses of residents and PDs.


The survey was completed by 214/350 (61%) residents and 11/14 (79%) PDs. Free OERs were used by residents most frequently for general EM education (99.5%), procedural skills training (96%), and learning to interpret diagnostic tests (92%). OER modalities used most frequently included wikis (95%), file-sharing websites (95%), e-textbooks (94%), and podcasts (91%). Residents used wikis, podcasts, vodcasts, and file-sharing websites significantly more frequently than PDs. Relative to PDs, residents found entertainment value to be more important for choosing OERs (p<0.01). Some residents (23%) did not feel that literature references were important, whereas all PDs did. Both groups reported that OERs increased the amount of peer-reviewed literature (75% and 60%, respectively) that they read.


EM residents make extensive use of OERs and differ from their PDs in the importance that they place on their entertainment value and incorporation of peer-reviewed references. OERs may increase the use of peer-reviewed literature in both groups. Given the prevalence of OER use for core educational goals among RCPSC-EM trainees, future efforts to facilitate critical appraisal and appropriate resource selection are warranted.



Il existe de plus en plus de ressources didactiques en ligne (RDL) pouvant servir à la formation en médecine d’urgence (MU). L’étude dont il question ici vise à décrire et à comparer l’utilisation des RDL gratuites par les résidents en MU du Collège royal des médecins et chirurgiens du Canada et les directeurs de programme (DP), ainsi qu’à examiner le lien entre l’utilisation des RDL et la lecture de documents évalués par les pairs.


Un questionnaire d’enquête bilingue, en ligne a été envoyé, selon une version modifiée de la méthode de Dillman, à tous les résidents en MU du Collège royal et aux DP. Les auteurs ont appliqué le test Khi2 et la méthode exacte de Fisher pour comparer les réponses des résidents et celles des DP.


Dans l’ensemble, 214 résidents sur 350 (61 %) et 11 DP sur 14 (79 %) ont participé à l’enquête. Les RDL gratuites utilisées par les résidents concernaient surtout la formation générale en MU (99,5 %), l’acquisition de compétences techniques (96 %) et l’apprentissage de l’interprétation des examens de diagnostic (92 %). Les formes de RDL très souvent utilisées comprenaient les sites wikis (95 %), les sites Web de partage de fichiers (95 %), les livres numériques (94 %) et les fichiers balados (91 %). Les résidents utilisaient les sites wikis, les fichiers balados, les fichiers balados vidéos et les sites Web de partage de fichiers beaucoup plus souvent que les DP. En outre, les résidents accordaient plus d’importance à la valeur du divertissement que les DP dans le choix des RDL (p < 0,01). En revanche, quelques résidents (23 %) estimaient que les références bibliographiques n’étaient pas importantes, contrairement à tous les DP. Enfin, les participants dans les deux groupes ont indiqué que l’utilisation des RDL avait pour effet d’accroître la lecture de documents évalués par les pairs (75 % et 60 %, respectivement).


Les résidents en MU font un usage très répandu des RDL et ils se distinguent des DP par la valeur accordée au divertissement et à l’indication des références dans les documents évalués par les pairs. Les RDL peuvent accroître la lecture de documents évalués par les pairs dans les deux groupes. Compte tenu de la grande utilisation des RDL que font les stagiaires en MU du Collège royal pour atteindre des buts pédagogiques de base, il est justifié de faire de la recherche sur les moyens visant à faciliter l’évaluation critique et la sélection appropriée des ressources.

Original Research
Copyright © Canadian Association of Emergency Physicians 2015 


Most medical trainees are members of Generation Y. This cohort, born in the 1980s and 1990s, is characterized as being more technologically savvy than preceding generations.Reference Weiler 1 They also increasingly use social media (defined as the sharing of information and ideas within an online community) to communicate. The health profession trainees in this cohort increasingly rely on the Internet and other technology to retrieve medical information.Reference Weiler 1 Reference Matava, Rosen and Siu 5

Online educational resources (OERs) are becoming increasingly prominent in medical education. Initially, these resources were paid for by institutions (e.g., UpToDate) and individuals (e.g., the podcast EM:RAP); however, free OERs are growing in popularity.Reference Nickson and Cadogan 6 Reference Mallin, Schlein and Stroud 8 Emergency medicine (EM) has become a leader in this area.Reference Nickson and Cadogan 6 Reference Mallin, Schlein and Stroud 8

Understanding whether learners and their teachers differ in use and choice of free resources is an important next step to comprehending the impact of OERs on medical education.Reference George and Green 9 , Reference Cheston, Flickinger and Chisolm 10 Free OERs vary in content; some are highly referenced, registered with universities, incorporate peer-review processes, and offer continuing medical education credit, whereas others propagate inaccurate information.Reference Brabazon 11 , Reference Thoma, Chan and Desouza 12 Generational trends suggest that the current cohort of EM learners will be drawn to these nontraditional resources to a greater extent than their teachers, but how students and teachers choose resources has not been well defined.Reference Weiler 1 Reference Barker, Wehbe-Janek and Bhandari 3 To teach appropriate stewardship of this new and evolving repository of educational material, the use of free OER must first be described and quantified.

The primary purpose of this study was to quantitatively describe free OER use by the Royal College of Physicians and Surgeons of Canada (RCPSC) EM residents. Secondary objectives were to compare the use of OERs between RCPSC-EM residents and program directors (PDs) and to investigate the effect of OERs on the use of peer-reviewed literature. A survey methodology was selected to explore these questions and to allow data to be collected from this large, distributed population in a generalizable, cost-effective, and standardized fashion.


This study received ethical approval from the Research Ethics Board of McMaster University. Survey results were reported in accordance with best practices as outlined in Academic Emergency Medicine.Reference Mello and Merchant 13

The survey was designed by consensus of the five authors who were informed by similar surveys in anesthesia and EM residents.Reference Matava, Rosen and Siu 5 , Reference Mallin, Schlein and Stroud 8 , Reference Wong 14 It was hosted on Fluid Surveys. The program director and a resident from postgraduate years one through four from the University of Saskatchewan RCSPC-EM residency program completed a pilot survey between September 27, 2013 and October 10, 2013. Their qualitative feedback was used to clarify questions and refine content. The final version of the survey was translated into French by a bilingual co-author (Migneault). The English version is available in Appendix 1.

A survey link was distributed by email to RCPSC-EM residents and PDs of the 14 training programs in Canada. Respondents were recruited using a modified Dillman methodReference Hoddinott and Bass 15 that involved a maximum of five emails from multiple sources, including the director of the Canadian Association of Emergency Physicians resident section, chief residents, and PDs. Responses were collected from November 19 to December 31, 2013. All responses were anonymous, and completion of the survey was voluntary. Entry into a draw for a $500 gift certificate was used as an incentive for survey participation.

The statistical analysis was performed using SPSS Version 21.0 (SPSS IBM, New York, USA). The chi-square test and Fisher’s exact test (<5 variables) were used to calculate p-values for categorical variables. A p-value ≤0.05 was considered statistically significant. Data entered into incomplete surveys were included in the statistical analysis.


As outlined in Figure 1, 214/350 (61%) of RCPSC residents and 11/14 PDs (79%) participated in the study. There were three incomplete responses included in the analysis, for which participants entered demographic information and answered some questions but did not reach the final page of the survey. The demographics of the participants are presented in Table 1.

Figure 1 Flow diagram of study participation and survey completion.

Table 1 Demographics

Free OER use: frequency, type, and comparison to traditional resources

Residents used wikis (95%), e-textbooks (93%), file-sharing sites (91%), podcasts (90%), video sharing sites (88%), medical blogs (80%), and vodcasts (71%) at least monthly. Textbooks (86%), subscription-based resources (65%), and podcasts/vodcasts (40%) were ranked as the top three resources that contributed most to a resident’s education, more frequently than primary literature (33%), e-textbooks (26%), medical blogs (26%), and wikis (11%).

Comparison of the types of OERs used by RCPSC-EM residents and PDs (Figure 2), respectively, showed that residents over PDs used podcasts (90% v. 45%, p<0.01), wikis (96% v. 73%, p<0.01), online file-sharing sites (91% v. 54%, p<0.01), and vodcasts (71% v. 27%, p<0.01) more frequently.

Figure 2 Monthly or more frequent use of OERs by residents compared to PDs (*p<0.05).

Reason for free OER use

OERs were used by residents and PDs, respectively, for a wide variety of purposes (Figure 3), including core EM education (98% v. 82%, p=0.01), procedural skills training (95% v. 73%, p=0.01), diagnostic/imaging interpretation (92% v. 73%, p>0.05), answering questions at the point of patient care (86% v. 82%, p>0.05), and investigating controversial topics in EM (82% v. 73%, p>0.05).

Figure 3 Purpose of OER use by residents compared to PDs (*p<0.05).

Resource selection

Figure 4 compares the factors that residents and PDs used for selecting resources. The factors that ranked more frequently as important or very important by residents when compared to PDs, respectively, were entertainment value (41% v. 9%, p=0.02) and peer referral (50% v. 36%, p=0.05). Inclusion of references was not statistically different between the groups, but only 77% of residents compared to 100% of PDs thought references were important or very important.

Figure 4 Perceived value of factors important for selecting OERs by residents compared to PDs (*p<0.05).

Free OERs and peer-reviewed literature use

Overall, residents (80%) and PDs (73%) self-reported that free OERs increased their use of primary literature. Both groups (72% residents, 55% PDs) felt that they read more studies in full as a result of OERs, and both groups (79% residents, 73% PDs) felt that they read more critical appraisals of primary studies. A minority of residents (25%) and PDs (27%) access more than half of the articles cited by OERs.


This study found that RCPSC-EM residents and PDs use free OERs regularly. Residents used nearly all OERs more frequently than PDs and considered different factors when selecting resources. OERs increased self-reported primary literature use. Despite prevalent OER use in our study, residents ranked textbooks and subscription-based resources as their top information sources. These results are similar to other reports of OER use by residents.Reference Matava, Rosen and Siu 5 , Reference Mallin, Schlein and Stroud 8 , Reference Wong 14

There were several differences between the types of OERs used by residents and PDs. The use of podcasts was the greatest discrepancy with 90% of residents and 45% of PDs listening to them monthly. This difference is important because we also found that learners value direction from faculty in selecting resources. If PDs are not frequently using podcasts or other OERs, they may be ill equipped to suggest appropriate resources. It will be increasingly important for educators to understand the usage patterns and motivations behind learner use of OERs.

Residents’ and PDs’ opinions also differed on the importance of entertainment value and referenced content when selecting OERs. The increased value placed on entertainment by residents is congruent with a recent report highlighting how Generation Y EM learners are most receptive to education that is fast-paced and entertaining.Reference Santella, Kaneva and Petrucci 16 , Reference Moreno-Walton, Brunett and Akhtar 17 OERs more frequently offer this “edu-tainment” factor than traditional educational resources. The increasing importance of engaging and entertaining educational materials is a trend that warrants recognition by educational leaders.Reference Moreno-Walton, Brunett and Akhtar 17 , Reference Sandars and Morrison 18 The importance of references in the selection of OERs also differed. All PDs valued appropriately referenced OERs, whereas nearly a quarter of residents did not. Although this trend was not statistically significant, efforts to further understand this discrepancy should be a focus of both formal research and discussions between residents and faculty.

By recognizing these differences in OER resource selection, PDs may be better equipped to encourage high quality resource choice among their trainees. Interestingly, a recent survey of RCPSC-EM PDs endorsed the use of high quality OERs and highlighted the importance of developing residents’ abilities to critically appraise them.Reference Bednarczyk, Pauls and Fridfinnson 19 These skills are important because the quality of information delivered in OERs can be a variable. For example, the reliability of wikis for medical information is unclearReference Lavsa, Corman and Culley 20 , Reference Haigh 21 with one study showing that EM trainees more frequently answered clinical questions incorrectly with greater confidence when using Google, which often links to wikis as a resource.Reference Krause, Moscati and Halpern 22

One of the strongest criticisms of traditional resources (i.e., textbooks, systematic reviews, and subscription-based) is that they are quickly outdated. Subscription-based resources and e-textbooks have considerable variability in the proportion of updated topics with 52% of 200 articles on UpToDate classified as out of date.Reference Jeffery, Navarro and Lokker 23 If even continuously updated resources are potentially expired, it follows that printed textbooks are even more outdated. Systematic reviews are subject to similar limitations, with 7% in need of updating at the time of publication.Reference Shojania, Sampson and Ansari 24 OERs have the potential to address this limitation because they can be published quickly and are easily updated.

The delayed update of traditional resources would be less of a problem if it was possible to read all of the latest primary literature. However, this is time-consuming with estimates suggesting a number needed to read 13–14 articles from the top 20 journals to find one practice-changing article.Reference McKibbon, Wilczynski and Haynes 25 OERs may ameliorate this problem by delivering important information from and increasing awareness of the literature in a timely and digestible fashion. In our study, both residents and PDs reported increased use of primary literature related to OER use, which may be due to increased awareness of relevant articles.

Weighing the strengths and weaknesses of any learning tool should be the shared responsibility of residents and their teachers. Free OERs are no different, but our study demonstrates discrepancies in OER use and the factors that residents and PDs see as important in this decision-making process. OERs offer unique benefits over traditional resources, including ease of access, frequent updates, real-time feedback from a worldwide audience, and the opportunity for active learning through content creation and interaction with content producers. The asynchronous learning model, central to OERs, makes it a natural choice for Generation Y learners who value accessibility highly when seeking information.Reference Weiler 1 , Reference Sandars and Morrison 18 Those involved with residency education need to be aware of these benefits and also be prepared find ways to help learners identify the highest quality resources. This will require knowledge of the ever-increasing body of available free OERs.


Our survey methodology was susceptible to various forms of bias (e.g., nonresponse, recall, and social desirability); however, many forms of validity evidence were also inherent to our study design. The survey content was based on previously used instruments, the survey was piloted with a representative group of residents and educators, the study population was well defined, reliable contact information was available for all potential participants, the response rate was relatively high for a national survey,Reference Barker, Wehbe-Janek and Bhandari 3 , Reference Matava, Rosen and Siu 5 , Reference Mallin, Schlein and Stroud 8 and residents from all programs and regions of Canada were included.

A second limitation is the large number of comparisons between the two populations, which makes the results susceptible to a type I error. However, the majority of differences that we found were highly significant (generally p<0.01). The small population of PDs also put the study at risk of a type II error.

It is arguable that the chosen study population and focus on free OERs limit the generalizability of our results. Exclusive study of RCPSC-EM residents and PDs means that the results are not fully generalizable to other residency programs, but it did allow for specificity to our population of interest. PDs are a specific and unique population of educators and may not be representative of all teachers involved in the education of RCPSC-EM residents. It was not feasible to survey all educators, so PDs were chosen because they are educational opinion leaders who have consistent contact with the surveyed population of residents and who are responsible for developing longitudinal resident curricula. We focused on free OERs because paid OERs are better established, less accessible, seem to face less scrutiny, and have had less explosive growth.

We were unable to provide descriptive interpretation of our results because the survey included only closed-ended questions. Open-ended questions were not included because it was felt that they would decrease survey completion.


This study informs educators about residents’ OER use. Its findings should stimulate discussions about OER use and their critical appraisal within EM education. EM residents use OERs frequently for general EM education, procedural training, and learning interpretive skills. Relative to their PDs, residents use OERs more frequently and place greater value on entertainment. Self-reported data suggest that OERs increase the use of peer-reviewed literature for both residents and PDs. Future research that prospectively assesses residents’ use of OERs would deepen our understanding of their role in resident education.

Competing interests: None declared.

Supplementary material

To view supplementary material for this article, please visit


1. Weiler, A. Information-seeking behavior in Generation Y students: motivation, critical thinking, and learning theory. The Journal of Academic Librarianship. Elsevier 2005;31(1):46-53.Google Scholar
2. Canadian Medical Association. Social media use by physicians. 2011. Available at: (accessed February 2014).Google Scholar
3. Barker, AL, Wehbe-Janek, H, Bhandari, NS, et al. A national cross-sectional survey of social networking practices of U.S. anesthesiology residency program directors. J Clin Anesth 2012;24(8):618-624.Google Scholar
4. von Muhlen, M, Ohno-Machado, L. Reviewing social media use by clinicians. J Am Med Inform Assoc 2012;19(5):777-781.Google Scholar
5. Matava, CT, Rosen, D, Siu, E, et al. eLearning among Canadian anesthesia residents: a survey of podcast use and content needs. BMC Med Educ 2013;13(1):59.Google Scholar
6. Nickson, CP, Cadogan, MD. Free Open Access Medical education (FOAM) for the emergency physician. Emerg Med Australas 2014;26(1):76-83.Google Scholar
7. Cadogan, MD, Thoma, B, Chan, TM, et al. Free Open Access Meducation (FOAM): the rise of emergency medicine and critical care blogs and podcasts (2002-2013). Emerg Med J 2014;31(e1):e76-7; online first. Available at: (accessed March 2014).Google Scholar
8. Mallin, M, Schlein, S, Stroud, S, et al. A survey of the current utilization of asynchronous education among emergency medicine residents in the United States. 2014;89(4):598-601.Google Scholar
9. George, DR, Green, MJ. Beyond good and evil: exploring medical trainee use of social media. Teach Learn Med 2012;24(2):155-157.Google Scholar
10. Cheston, CC, Flickinger, TE, Chisolm, MS. Social media use in medical education. Acad Med 2013;88(6):1.Google Scholar
11. Brabazon, T. The Google effect: googling, blogging, wikis and the flattening of expertise. Libri 2006;56(3):157-167.Google Scholar
12. Thoma, B, Chan, TM, Desouza, N, et al. Implementing peer review at an emergency medicine blog bridging the gap between educators and clinical experts. CJEM 2014;16(0):1-4.Google Scholar
13. Mello, MJ, Merchant, RC. Surveying emergency medicine. Acad Emerg Med 2013;20:409-412.Google Scholar
14. Wong, N. EM education survey. EM Resident 2013:8-9.Google Scholar
15. Hoddinott, SN, Bass, MJ. The Dillman total design survey method. Can Fam Physician 1986;32:2366-2368.Google Scholar
16. Santella, T, Kaneva, P, Petrucci, A, et al. Using social media for surgical education: distraction or opportunity? 2013: 1-2. Available at: (accessed February 2014).Google Scholar
17. Moreno-Walton, L, Brunett, P, Akhtar, S, et al. Teaching across the generation gap: a consensus from the Council of Emergency Medicine Residency Directors 2009 Academic Assembly. Acad Emerg Med 2009;16(2):S19-S24.Google Scholar
18. Sandars, J, Morrison, C. What is the Net Generation? The challenge for future medical education. Med Teach 2007;29(2-3):85-88.Google Scholar
19. Bednarczyk, J, Pauls, M, Fridfinnson, J, et al. Characteristics of evidence-based medicine training in Royal College of Physicians and Surgeons of Canada emergency medicine residencies—a national survey of program directors. BMC Med Ed 2014;14(1):57.Google Scholar
20. Lavsa, SM, Corman, SL, Culley, CM. Reliability of Wikipedia as a medication information source for pharmacy students. Current Pharm Teach Learn 2011;3(2):154-158.Google Scholar
21. Haigh, CA. Wikipedia as an evidence source for nursing and healthcare students. Nurse Educ Today 2011;31(2):135-139.Google Scholar
22. Krause, R, Moscati, R, Halpern, S, et al. Can emergency medicine residents reliably use the Internet to answer clinical questions? West J Emerg Med 2011;12(4):442-447.Google Scholar
23. Jeffery, R, Navarro, T, Lokker, C, et al. How current are leading evidence-based medical textbooks? An analytic survey of four online textbooks. J Med Internet Res 2012;14(6):e175.Google Scholar
24. Shojania, KG, Sampson, M, Ansari, MT, et al. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med 2007;147(4):224-234.Google Scholar
25. McKibbon, KA, Wilczynski, NL, Haynes, RB. What do evidence-based secondary journals tell us about the publication of clinically important articles in primary healthcare journals? BMC Med 2004;2:33-46.Google Scholar
Figure 0

Figure 1 Flow diagram of study participation and survey completion.

Figure 1

Table 1 Demographics

Figure 2

Figure 2 Monthly or more frequent use of OERs by residents compared to PDs (*p<0.05).

Figure 3

Figure 3 Purpose of OER use by residents compared to PDs (*p<0.05).

Figure 4

Figure 4 Perceived value of factors important for selecting OERs by residents compared to PDs (*p<0.05).

Supplementary material: File

Purdy supplementary material

Appendix 1

Download Purdy supplementary material(File)
File 16 KB