Background
The advent of the World Wide WebReference MacKenzie and Greenes 1 in 1991 led to an enormous range of innovations in medical education.Reference Friedman 2 , Reference Webb, Joseph and Yardley 3 Now, over two decades later, the development of social media with its interactive content is expected to change the way medicine is taught.Reference Boulos, Maramba and Wheeler 4 – Reference Sandars 8 “Social media” is defined as a group of network technologies that share a participatory approach for creating content through open architecture that facilitates collaboration.Reference McGee and Begg 9 Collaborative writing applications (CWAs) are a category of social media that allows multiple authors to contribute synchronously and asynchronously to a single document.Reference Kaplan and Haenlein 10 , Reference Archambault, Van De Belt and Grajales 11 New CWAs, such as Google DocsTM and wikis, are rapidly gaining popularity in medical education because they support free and open collaboration and decentralize content production.Reference Boulos, Maramba and Wheeler 4 , Reference Varga-Atkins, Dangerfield and Brigden 7 , Reference McGee and Begg 9 , Reference Archambault, Blouin and Poitras 12 – 18 Constructivist learning principles support the use of social media in medical education. Instead of passively receiving information, students can actively create course content, and by doing so increase their comprehension and knowledge retention.Reference Morley 19 Constructivism purports that students learn best when they construct their own meaning from experiences and develop their own solutions to problems. In the context of using CWAs in a collaborative writing project, the concept of individual constructivist learning is expanded to include communal constructivist learning, which engages students in developing their own information and creating knowledge that will benefit other students.Reference Holmes, Tangney and Fitzgibbon 20 In this model, students leave their own imprint in the development of their course, university, and ideally, discipline.
A meta-analysis assessing the efficacy of Internet-based learning in medical education found that such approaches have a positive effect.Reference Cook, Levinson and Garside 21 However, the analysis did not include social media or CWA interventions.Reference Cook, Levinson and Garside 22 Three recent systematic reviews found that social media improved skill acquisition, knowledge retention, student satisfaction, and teacher supervision.Reference Archambault, Van De Belt and Grajales 11 , Reference Cheston, Flickinger and Chisolm 23 , Reference Hollinderbaumer, Hartz and Uckert 24 A scoping review of the literature,Reference Archambault, Van De Belt and Grajales 11 focused specifically on CWAs, found three randomized trialsReference Moeller, Spitzer and Spreckelsen 25 – Reference Stutsky 27 and concluded that CWAs had positive effects on scientific writing skills, leadership skills, and problem-based learning processes. However, low contribution rates have been found to be a major obstacle to CWA use in medical education.Reference Archambault, Van De Belt and Grajales 11 , Reference Cheston, Flickinger and Chisolm 23 , Reference Johnson, Subak and Brown 28 , Reference Thompson, Schulz and Terrence 29 This is a common problem with CWAs: for example, 44% of all contributions to Wikipedia (Wikimedia Foundation, USA) are made by 0.1% of editors.Reference Heilman, Kemmann and Bonert 30 , Reference Priedhorsky, Chen and Lam 31 This contribution process has been described as a “long tail” distribution, whereby a large number of people make small contributions.Reference Anderson 32 – Reference Yan and Gerstein 34
Since 2008, the authors have used Google Docs (now called Google DriveTM) to share a collaborative online presentation designed for use by emergency medicine (EM) residents graduating from the Royal College of Physicians and Surgeons of Canada (RCPSC) training programs to allow them to share summaries and brief critical appraisals of landmark articles that they have reviewed in preparation for their RCPSC certification exam. The purpose of this presentation is to foster collaboration among EM residents across Canada and maintain an up-to-date database of article summaries. However, the contribution rate for article summaries has remained under 2%.Reference Archambault, Blouin and Poitras 12 , Reference Archambault, Blouin and Poitras 35 The purpose of this study was to identify residents’ beliefs about contributing summaries of landmark articles to an online national collaborative writing project using the Google Docs slideshow application, with the overall goal of designing strategies to improve contribution rates.
Methods
Theoretical basis
The Theory of Planned Behaviour (TPB) (Figure 1)Reference Ajzen 36 , Reference Conner and Norman 37 was used to identify residents' beliefs. This theory has been applied successfully to the study of health professional behaviours to help tailor implementation interventions.Reference Godin and Kok 38 , Reference Millstein 39 A systematic review found that Internet-based implementation interventions informed by the TPB tend to have substantial effects on health behaviour change.Reference Webb, Joseph and Yardley 3 The TPB helps implementation scientists focus on key elements known to influence the adoption of clinical behaviours. Indeed, attitude, subjective norm, and perceived behavioural control are key TPB elements that must be considered when designing an effective implementation intervention. Together, these three factors explain a significant amount of behavioural change.Reference Rutter and Quine 40 When an individual has some control over a situation for modifying his or her clinical behaviour, intention is the immediate determinant of this behaviour.Reference Ajzen 41 In other words, before clinicians change behaviour, they must intend to do so. In turn, this behavioural intention is itself under the influence of a set of personal beliefs. According to Ajzen, an individual’s salient beliefs are the beliefs most frequently reported with respect to the key elements: attitude, social norm, and perceived behavioural control. “Attitude” refers to a person’s evaluation of the consequences (advantages and disadvantages) of adopting the behaviour. “Subjective norm” refers to a person’s perceived social pressure to engage or not engage in a given behaviour, including beliefs about how people who are considered important (referents) would like us to behave. “Perceived behavioural control” refers to a person’s perception of how easy or difficult it is to perform the behaviour in question in light of perceived barriers and facilitators (control beliefs).
Theory of planned behaviour
Study design
The protocol for this study has been published elsewhereReference Archambault, Blouin and Poitras 35 and was approved by the Ethics Review Board at Centre de santé et services sociaux Alphonse-Desjardins.
Drawing on the TPB, we conducted semi-structured telephone interviews with graduating RCPSC EM residents. Interview questions were based on a vignette that clearly described the behaviour of interest that we wanted residents to reflect (Appendix 1, available at http://dx.doi.org/10.1017/cem.2014.49). The behaviour of interest was defined using Fishbein’s target-action-context-time (“TACT”) principles:Reference Fishbein and Yzer 42 (1) target: a new Google Docs slide summarizing an important article missing from the presentation; (2) action: to contribute; (3) context: in preparation for the RCPSC certification exam in EM; and (4) timeline: within six months.
Study setting and recruitment of participants
Participants in the study had to be RCPSC EM residents in their last year of training. Thirteen of the 14 RCPSC EM programs in Canada had eligible residents at the time of the study. Potential participants were recruited via the annual National Emergency Medicine Review course at Queen’s University. During the course, the principal investigator presented a 90-minute lecture reviewing the most important literature of the year. A similar lecture has been given every year since 2008 using the Google Docs slideshow application. After the lectures in 2010, 2011, and 2012, an email was sent to all attendees inviting them to freely access, edit, and update the slideshow based on their own studies and readings. The email included an invitation to participate in a telephone interview, the vignette, and the link to the study’s SurveyMonkeyTM questionnaire. Email reminders were sent one week and two weeks after the initial email. In addition, invitations were sent to specific participants to obtain a representative sample of residents from each training program, and to elicit the widest possible range of beliefs. In order to identify these additional participants, we contacted local leaders from the universities where the email invitation response was low and asked them to identify potential participants. Potential participants identified through this approach were sent a personal email invitation to participate.
Data collection procedure
Consent was obtained before all interviews. Using a SurveyMonkey questionnaire (see Appendix 2, available online at http://dx.doi.org/10.1017/cem.2014.49), information was collected on participant’s age, gender, degree, university affiliation, previous consultation and edition of any Google Docs application or other CWA (e.g., Wikipedia) used. The questionnaire asked if participants had consulted or edited the online slideshow, and if so, how often and what changes they had made (e.g., adding, correcting). Participants were asked if they knew of articles that were missing, if they preferred another CWA, and if they had downloaded the slideshow for personal use. Using a 7-point Likert scale, participants were asked to rate their intention to contribute a new slide summarizing an important missing EM article within the next six months. A few days after the online questionnaire was completed, an interviewer, blinded to the online responses, conducted a telephone interview with each participant. The interviewer read the vignette to each participant and elicited their beliefs regarding the targeted behaviour. Each interview was digitally recorded and anonymously transcribed verbatim. Interviews were conducted in French or English.
Sample size
Godin and KokReference Godin and Kok 38 suggest that a sample size of 25 participants is sufficient to achieve data saturation when conducting a salient beliefs study. To verify that data saturation was reached with our sample, we followed principles described by Francis and colleaguesReference Francis, Johnston and Robertson 43 and used a cumulative frequency graph to identify the data saturation point (i.e., three consecutive interviews without any new reported beliefs after the initial 10 interviews) for each of our three constructs (behavioural, normative, and control beliefs).
Data analysis
For participants’ demographic characteristics and their self-reported contributions to the online presentation, the mean, standard deviation, median, and interquartile range for continuous variables was calculated, as was the percentages for dichotomous variables. Because of limited information about the population of all eligible residents, the study was only able to assess whether the sample differed significantly from all eligible residents in gender distribution and distribution of EM training programs. All calculations, including the χ2 test to calculate p-values for categorical data, were performed with Microsoft® Excel for Macintosh 2011 (Microsoft, Redmond, WA, USA).
For qualitative data, a mix of inductive and deductive thematic analysis was used. Two authors (SG, JT) independently analysed transcripts of the recorded interviews to identify control, normative, and behavioural beliefs. Through discussion, grouped beliefs were inductively placed into themes to produce a list for each construct. In order to compare the study results to those of similar studies, labels from a validated taxonomy of social media adoption determinants were used.Reference Archambault, Van De Belt and Grajales 11 , Reference Gagnon, Desmatis and Labrecque 44 Each belief was ranked from most to least frequently mentioned. The top 75th percentile of most frequently cited beliefs were considered “salient” in keeping with TPB methodology.Reference Francis, Eccles and Johnston 45 In cases of disagreement, a third author (PA) was consulted.
Results
Data saturation
Data saturation was subjectively achieved during the interview with the 17th participant. However, we continued recruitment until our intended sample size of 25 participants was reached. Using the methods proposed by Francis and colleagues,Reference Francis, Johnston and Robertson 43 we retrospectively determined that data saturation had been attained for control, behavioural, and normative beliefs after the 14th, the 13th and the 16th interviews, respectively (Figure 2).
Cumulative frequency of reported beliefs by Canadian emergency medicine residents for the studied behavior between 2010–2012 to determine the data saturation point.
Participant characteristics
Demographic characteristics of the eligible and participating study population are presented in Table 1. Over the three years of this study, a total of 150 residents were eligible to participate (37 in 2010, 49 in 2011, and 64 in 2012). Of these, 77% (n=115) did not respond to our invitation, 4% (n=6) declined, and 3% (n=4) of email addresses were invalid. A total of 17% (n=25) of participants were recruited. The participation rates for each respective cohort were 11% (n=4), 20% (n=10), and 17% (n=11). Demographic information was collected for all participants except one, whose data were lost. Residents were recruited from 10 of the 13 EM programs. Among participants, 38% (n=9) had previously edited another Google Docs document and 8% (n=2) had previously edited a wiki.
Demographic characteristics of eligible and participating study population
* Demographic information is missing for one participant because his/her responses were not recorded in the online survey
Abbreviations: n: number; SD: standard deviation; 25%–75% IQR: 25th to 75th percentile interquartile range; University U1–13: to preserve the confidentiality of each participating university, we named the 13 different universities U1 to U13.
SurveyMonkeyTM on contributions to Google DocsTM
Among the 24 participants with SurveyMonkeyTM data, only 17% (n=4) stated that they had edited the online presentation (Table 2). Although 50% (n=12) of participants had read an article in the past year that they considered should be included in the slideshow, only 8% (n=2) added a new summary to the presentation. One participant had downloaded the slideshow and made offline edits. A higher proportion of participants (75% [n=18]) reported having consulted the slideshow in the last year. Four participants said they would prefer a different collaborative platform (e.g., DropboxTM or a blog). Finally, although the self-reported contribution rate was low, 29% (n=7) of participants stated that they were “quite likely” or “very likely” to contribute a slide to the slideshow within the next six months.
Participants self-reported use of the study Google Docs online slideshow
* One participant’s responses were not recorded in the online survey and thus were lost. This participant's responses to the semi-structured interviews are included.
Residents’ salient beliefs
After interviewing all 25 participants, 14 behavioural, 12 normative, and 30 control beliefs were identified (Table 3). Of all these, seven behavioral, seven normative, and 16 control beliefs were considered salient.
Reported beliefs of study participants regarding contributing to an online Google Docs slideshow
* Beliefs identified with asterisk were considered salient because they represent the top 75 percentile most frequently cited beliefs.
§ The rank number corresponds to the position held in the ranking of all beliefs. The most frequently mentioned belief is ranked first. The ranking numbers do not necessarily follow each other in this table, since we grouped them as advantages, disadvantages, favorable referents, unfavorable referents, barriers, and facilitators. Two beliefs can hold the same rank when they were mentioned at the same frequency by our participants.
† n=the number of participants who reported the belief during their interview, and %=the number of times the belief was reported in all interviews divided by the number of times all beliefs in that category (behavioral, normative, and control beliefs) were reported in all interviews.
‡ The label for this belief was taken from the Gagnon and colleagues <44> framework.
¶ The label for this belief was taken from the Archambault and colleagues <56> framework.)
∆ ICT: Information and communication technologies
Behavioural beliefs
Eleven advantages and three disadvantages were reported by the participants. Interestingly, 76% (n=19) of participants saw no disadvantages to contributing to the slideshow. The three behavioural beliefs mentioned most frequently were that contributing a slide would “consolidate learning” (15% [n=15]), would be “useful” (14% [n=14]), and would allow “mutual aid and collaboration” (13% [n=13]), where the denominator is the total number of beliefs. All salient behavioural beliefs were about advantages. Some non-salient beliefs about disadvantages were reported, such as the belief that contributing to the slideshow added to “information overload” (n=3).
Normative beliefs
The three most frequently mentioned positive referents were “graduating EM residents writing the exam” (26% [n=16]), “all EM residents” (21% [n=13]), and “EM teachers” (13% [n=8]). All salient normative beliefs concerned positive referents; 88% of participants (n=22) did not identify any negative referent. Only two disapproving referents were identified: “competitive people” reluctant to share information with others (3% [n=2]), and regulatory authorities who might disapprove of sharing information (2% [n=1]).
Control beliefs
Sixteen control beliefs were considered salient: eight facilitators and eight barriers. The top three were barriers: “time consuming” (13% [n=22]), “other priorities in the last year” (9% [n=15]), and “online presentation not easy to access” (8% [n=14]). The top three facilitators were “peer-reviewed high quality scientific information” (6% [n=10]) provided by the slideshow, “the use of a template” (5% [n=8]), and “familiarity with information and communication technologies” (5% [n=8]).
Discussion
Summary of findings
This study identifies EM residents’ beliefs about contributing to an online collaborative slideshow via Google Docs applications designed to help them share summaries and critical appraisals of landmark articles in preparation for the RCPSC EM certification exam. The study used TPB methodology to identify the salient beliefs necessary for designing a theory-based intervention to increase contributions to collaborative writing projects in medical education. Overall, participants perceived more advantages than disadvantages to contributing and believed that this initiative would receive wide support. However, participants reported several barriers that need to be addressed to increase contributions.
Among the major themes explaining the low contribution rate were organizational factors (e.g., lack of time, other priorities in the last year of residency), technical factors related to the CWA (e.g., low accessibility, task complexity, lack of usefulness), and individual factors (e.g., fear of making a mistake, lack of motivation, poor computer literacy).
Clinical relevance
These findings are important because CWAs are increasingly used in medical schools and residency programs worldwide. 18 , Reference Moeller, Spitzer and Spreckelsen 25 , Reference Phadtare, Bahmani and Shah 26 , Reference Thompson, Schulz and Terrence 29 , Reference Cheston, Flickinger and Chisolm 46 – 54 Beyond helping learners pass an exam, CWAs have the potential to keep students updated on relevant evidence, train them to quickly summarize it, and teach them to collaborate and share information in time-sensitive clinical contexts (e.g., EM, critical care). These study results are consistent with previous findings indicating that while the intention to contribute is high, contribution rates to collaborative projects remain low.Reference Archambault, Van De Belt and Grajales 11 , Reference Cheston, Flickinger and Chisolm 23 , Reference Johnson, Subak and Brown 28 As highlighted in recent systematic reviews, if these applications are expected to produce convincing and positive results in teaching medical skills and knowledge, their implementation must be optimized.Reference Archambault, Van De Belt and Grajales 11 , Reference Cheston, Flickinger and Chisolm 23 , Reference Hollinderbaumer, Hartz and Uckert 24
The use of CWAs in medical education has been explored using a variety of theory-based approaches. However, few authors have used this approach to understand how learners’ or clinicians’ beliefs influence their likelihood to contribute to collaborative online projects. While McGowan and colleagues used a quantitative approach to study the determinants of the “use of social media applications to share medical knowledge with other physicians,”Reference McGowan, Wasko and Vartabedian 55 this study used TPB-based qualitative methods to generate an exhaustive list of all the salient beliefs that influence contributions to a collaborative writing project. This list could be useful in creating questionnaires to investigate other collaborative writing projects and designing interventions aimed at increasing contributions.Reference Archambault, Bilodeau and Gagnon 56
Time constraints were found to be a major barrier. This is common to all collaborative writing projects, especially among health care professionals and students.Reference Stutsky 27 , Reference Hulbert-Williams Nicholas 57 – Reference Sandars and Morrison 61 Although there is no simple solution, some authors have proposed incentives such as offering continuing medical education or residency training credits for contributions.Reference Bender, O’Grady and Deshpande 62 , Reference Cohen 63 A new set of scholarly impact metrics that could measure the impact of open-source contributions might also provide further motivation.Reference Bender, O’Grady and Deshpande 62 Study participants suggested offering prizes for contributions and making the slideshow available earlier in their residency. Finally, convincing participants of the usefulness of contributing to the CWA is another potentially important solution.Reference McGowan, Wasko and Vartabedian 55 The negative beliefs identified (barriers, negative referents, and perceived disadvantages) are similar to those reported in a recent systematic review: information overload, perceived unequal distribution of work, competitiveness, regulatory authorities preventing access, time constraints, lack of motivation, and lack of computer literacy.Reference Archambault, Van De Belt and Grajales 11 Difficulty accessing the online presentation was a common barrier; 33% of study participants had downloaded the slideshow to their personal files. The only unique barrier identified, although not salient, was the lack of a track changes feature in the Google Docs slideshow platform. Four participants (17%) suggested using other collaborative tools (e.g., Dropbox or a blog) to share summaries, although it should be noted that these do not allow for better tracking of changes and have other disadvantages.Reference Luoma 64 It may be helpful to assess other relevant platforms, such as Google SitesTM, to address these issues.
As reported previously, the top three facilitators identified were the quality of information, the provision of a template for contributions, and training for CWA use.Reference Archambault, Van De Belt and Grajales 11 While the number of perceived advantages and positive referents was notably higher than the disadvantages and negative referents, the top three control beliefs were barriers, and overall, more barriers were identified than facilitators. Future studies measuring the importance of each of these barriers would be necessary to identify which contribute the most to low contribution rates. Interventions to increase contribution rates could then better target these barriers.
Strengths
To the authors’ knowledge, this study is the first to use a qualitative theory-based approach to identify residents’ salient beliefs concerning their intention to contribute to an online collaborative writing project. These study results provide baseline data for comparison with future studies. Furthermore, the detailed and rigorous description of qualitative content analysis provided in this study will enable other researchers to reproduce this study’s approach. The planned sample size (n=25) was based on Godin and Kok’s work,Reference Godin and Kok 38 even though the subjective data saturation was reached at the 17th interview. This allowed verification of the principles of data saturation proposed by Francis and colleagues.Reference Francis, Johnston and Robertson 43 This method could help future qualitative researchers operationalize data saturation determination and avoid unnecessary patient recruitment and resulting resource usage. Finally, this study explored three distinct cohorts of participating residents over the course of three years, thus allowing the presentation of data on a broader and more generalizable description of salient beliefs.
Limitations
This study had limitations that should be considered. First, we did not perform member checking. However, two independent research professionals experienced in TPB methodology carefully analysed the transcripts and resolved disagreements through a rigorous approach. The inclusion of verbatim quotes for each salient belief was intended to enable readers to judge the interpretation of the results. Second, our sample size of 25 participants represents only 17% of all eligible EM residents and failed to include all universities, potentially limiting the generalizability of the results. Moreover, the results may be affected by a positive response bias (with one university, representing 5% of the sample, providing 25% of the responses) and a negative non-response bias (with 25% of the sample providing 0% of the responses). The participating residents could have been more likely to have used and/or felt favorably about the product than those that did not. Since we did reach data saturation, we feel it is unlikely we would have identified new beliefs even if more residents and all universities had been represented. Among the elicited salient control beliefs, the top three were barriers indicating that the current participants shared many negative views about this project. Third, the measurement of future intention to contribute a slide to the collaborative writing project is only a proxy of the actual behaviour and may also be influenced by a “social desirability” bias, with participants responding with what they believe the authors wanted to hear. Alternatively, the survey participants may represent a small group of active contributors that represent the “long-tail” distribution described in other collaborative writing projects.Reference Anderson 32 , Reference Benkler 33 Since the Google Docs slideshow application did not allow precise measurement of how many new slides were added, the study could not verify whether the measured intention to contribute resulted in the completion of the actual behaviour. Future studies could utilize web analytics to investigate this relationship. Finally, the studied behavior was complex and some participants may not have understood the difference between “contributing to” and “using” the slideshow, although efforts were made to minimize this problem (e.g., presenting a clinical vignette, conducting interviews after participants had the opportunity to contribute).
Conclusion
EM residents share different salient beliefs regarding their intention to contribute content to an online collaborative writing project using a Google Docs slideshow. Overall, participants perceived more advantages than disadvantages to contributing and believed that this initiative would receive wide support. However, participants reported several barriers that need to be addressed to increase contributions. The researchers’ intention in the next phase of this initiative will be to construct a questionnaire to measure the quantitative importance of the salient beliefs identified in this study and to conduct another survey with a larger and more representative sample. This will allow the researchers to prioritize the most important beliefs that should be targeted by a theory-based intervention aiming to increase contributions to the collaborative writing project and will help develop implementation strategies for increasing contributions to other CWAs in medical education.
Acknowledgements
The authors extend their gratitude to all the participating emergency medicine residents, to Mary Lee for her logistical support, and to Simon Rioux and Catherine Nadeau for their participation as interviewers. We also thank Cynthia Fournier, Simon Rioux, and Claudie-Anne Giasson for transcribing the interviews, and Tina Marie Lalonde for reading and analysing some of the verbatim transcripts. Finally, we thank Louisa Blair for editing the manuscript.
Competing Interests: The principal author (PA) has received honoraria for presentations at the National Emergency Medicine Review course at Queen’s University. All other authors declare that they have no competing interests. None of the authors have a financial interest in the free online collaborative tool discussed, and no patents are pending for this tool. This study was supported by a grant from the Gilles Cormier Fund (Fonds Gilles Cormier reference number: FO099395). This fund provided by the Faculty of Medicine at Université Laval supports research projects in medical education. The funding organization did not influence the design of the study or content of the manuscript. Patrick Archambault holds a career scientist award from the Fonds de Recherche du Québec – Santé (reference number: FQ102051). France Légaré holds a Canada Research Chair in Implementation of Shared Decision Making in Primary Care.
Supplementary material
To view supplementary material for this article, please visit http://dx.doi.org/10.1017/cem.2014.49
