Medical education has a long-established interest in using virtual reality simulators to train doctors. Reference Merril, Notaroberto, Laby, Rabinowitz and Piemme1 Such technology is now replicated on the internet in the form of virtual worlds; this software enables users to navigate ‘avatars’ through virtual environments and communicate with other avatars online. Although a number of medical educational and psychoeducational applications of virtual worlds have been developed, Reference Gorini, Gaggioli, Vigna and Riva2 there remains a paucity of research. Reference Peachey, Gillen, Livingstone and Smith-Robbins3 Moreover, like virtual reality simulators, current virtual world medical applications generally involve learners interacting with the virtual environment (e.g. using objects, accessing text or audio-visual media through embedded pop-up windows) rather than with people. Research shows however, that without the interpersonal interaction, the application struggles to capture the complexity of clinical interaction; Reference Rampling, O'Brien, Hindhaugh, Woodham and Kavia4-Reference Toro-Troconis, Roberts, Smith, Partridge, Zagalo, Zagalo, Morgado and Boa-Ventura6 Rampling et al Reference Rampling, O'Brien, Hindhaugh, Woodham and Kavia4 conclude that conversational interaction between avatars would significantly improve the learning experience.
Our study aimed to harness interpersonal aspects of Second Life technology (a free-to-access online virtual world) to develop and evaluate a real-time clinical role play simulation. We are unaware of research evaluating Second Life for teaching child psychiatry, a specialty embracing interpersonal dynamics and complex decision-making. Reference Dave7-Reference Nestel and Tierney9 Simulation can standardise learning, particularly important given medical students' variable exposure to child psychiatry. Reference Sawyer, Giesen and Walter10 The specialty also involves risk management in patients vulnerable by virtue of age and mental disorder; simulation could provide a standardised learning environment to safely explore risk. Reference Ziv, Wolpe, Small and Glick11
Kneebone et al argue that simulation involves the ‘suspension of disbelief and then “buy-in” to the simulated experience’. Reference Kneebone, Nestel, Wetzel, Black, Jacklin and Aggarwal12 Virtual worlds could enhance role play simulation fidelity, helping students suspend belief that they are interacting with a teacher, while engaging them deeper with the clinical narrative. Reference Slater, Antley, Davison, Swapp, Guger and Barker13 Few medical education studies have previously explored the relationship between simulation fidelity and educational utility. This study asks whether medical students perceive a Second Life role play simulation as useful. Secondary questions explore: emotional aspects of learning, technology's ease of use, whether the simulation delivers ‘good enough’ fidelity and its relationship to perceived utility.
Method
The study, approved by Imperial College Research Ethics Committee, comprised part of a Jisc multisite project. 14
Participants
We recruited ten Imperial College penultimate-year medical undergraduates post-psychiatry or paediatric attachments. Sample size matched the methodologies of small-group teaching and qualitative research. We advertised internally and emailed information sheets to responding students, accepting participants consecutively. The research session immediately followed the teaching session. The session began with participants completing consent forms, receiving role play tasks and 30 minutes' training on Second Life. Participants received £45 retail vouchers to reimburse time.
Teaching session
Two consecutive 90-minute teaching sessions were conducted in computer labs, each with a randomly assigned group of five participants. The teacher operated from a different lab. All teaching occurred on Second Life Viewer (version v.3.3.4), within a virtual prefabricated clinic designed from a real clinic for authenticity (Fig.1; colour version given in online Fig. DS1). Reference Kneebone, Nestel, Wetzel, Black, Jacklin and Aggarwal12 Users only interacted through their avatars, communicating via audio-microphone headsets (technical details available from the authors on request).
Learning outcomes covered psychiatric assessment and management, communication and professional skills. The session was divided into briefing, role play, reflective and debriefing stages (Table 1). Reference Jarvis, Holford and Griffin15-Reference Laurillard17 The role play involved students using their avatars to role-play with the teacher playing a ‘depressed teenager’ avatar.
The role play session in the virtual clinic. The teacher plays an adolescent avatar (avatar on right-hand side, facing forwards, with an overhead icon). Role-playing participants are in the clinic room; observers are watching through the ‘virtual’ one-way screen.
Research methods
Research was primarily qualitative, since the goal was to explore in-depth attitudes towards a novel application of learning technology which had hitherto been minimally researched. Furthermore, the teaching was small-group by nature, which particularly lent itself to qualitative research. We included an additional quantitative method to evaluate the degree of consensus and ambivalence across the group relating to the various research questions. The mixed method enables triangulation of data interpretation.
Questionnaire
A literature search showed no evidence of an accepted standard validated questionnaire on clinical simulation. We developed and piloted an internet-based questionnaire encompassing Likert-style questions and open-box comments, in accordance with our research questions and respective theoretical foundation (online Table DS1). Reference Jarvis, Holford and Griffin15-Reference Lee, Chung and Chen18 We incorporated a questionnaire from Bonanno & Kommers, Reference Bonanno and Kommers19 chosen for its relevance but adapted to make explicit reference to Second Life rather than ‘computer gaming’ more generically. Additional questions explored usability, utility and prior experience. Reference Jarvis, Holford and Griffin15,Reference Enochsson, Isaksson, Tour, Kjellin, Hedman and Wredmark20 Analyses of demographic variable influence and fidelity-utility relationship were undertaken with SPSS Version 18, Windows XP, although the small sample size was expected to lack power for statistical significance. Open-box data fed into the thematic analysis.
The stages of the 90-minute teaching session
| Teaching session stage | Process | Educational modelsReference Jarvis, Holford and Griffin 15 , Reference Laurillard 17 |
|---|---|---|
| Briefing session | As ‘teacher avatar’, teacher reviews the principles of child psychiatric assessment, depression and self-harm; participants adopt ‘clinician avatars’. |
Discursive dialogue enables the teacher to elicit and consolidate students' prior understanding. Students enter learning armed with a biography of experience, expectation and knowledge. |
| Role play | The ‘teacher avatar’ leaves the virtual clinic; the teacher returns as a ‘patient avatar’ presenting ‘post-overdose’. The teacher acts flexibly, following a narrative crib sheet. Three participants per group respectively assess: presenting complaint, mental state and psychosocial history. The two remaining participants retire to an adjacent ‘virtual’ observation room, observing the role play through a ‘virtual’ one-way screen. |
The role play represents an episodic learning experience with cognitive, physical, attitudinal, emotive and sensory components. Learners receive real-time intrinsic feedback through the responses of the ‘patient avatar’, enabling them to experiment with different strategies and learn from others' responses. |
| Student reflection | Students write reflective notes on an electronic form, accessed from an icon embedded in the Second Life environment. Using SLOODLE software (www.sloodle.org), these forms are then submitted electronically to the teacher. |
Written reflection aims to facilitate learning through appraisal of one's experience and trialling in future situations. |
| Debriefing session | The tutor facilitates a reflective discussion, guided by students' submitted reflections, and delivers general and individual feedback. |
Learning is further facilitated through the teacher's extrinsic feedback. |
Nominal group and thematic analysis
Compared with conventional focus groups, nominal groups minimise individual dominance and encourage theme breadth. Reference Van de Ven and Delbecq21 The process began with participants writing their own responses to the question: ‘What are the advantages and the disadvantages of learning in Second Life (compared with other methods)?’ Views were then shared one by one in round-robin fashion and recorded and coded on a whiteboard. A discussion clarified and rationalised single points into larger or smaller themes where appropriate. Each participant then anonymously ranked their top ten themes. Each rank related to a score (first place 10 points, second place 9 points, etc.), which enabled generation of an overall group ranking based on collective scores. Further group discussion allowed final reordering of the top ten themes according to consensus.
A.K.V. moderated, recorded, transcribed and thematically analysed the nominal group. Themes generated by the nominal group guided the first round of coding in the thematic analysis, whereas further rounds of analysis interrogated the data. We are unaware of this approach being used in other research studies, but applied it to achieve a balance between the nominal group procdures democracy and breadth of participant-derived themes with a more in-depth thematic analysis. Themes were explicitly identified and analysed by a realist perspective. Reference Braun and Clarke22
Results
Participants were aged 22-24 years; six were female, four were male. Seven had heard of Second Life, but none had experience of it; all regularly played computer games; nine had role play experience.
Evaluation data from the questionnaire
Quantitative data from the questionnaire are shown in Table DS1; statements in the adapted Bonanno & Kommers' questionnaire Reference Bonanno and Kommers19 are grouped into four domains: ‘affective’, ‘behaviour’, ‘perceived control’ and ‘perceived usefulness’. As per the original study, adapted Bonanno & Kommers' statements were each scored on a 1-5 scale, with positive attitudes scoring higher. Statement scores are combined to provide domain scores and a composite ‘general attitude’ score. These scores were evaluated for differences with respect to gender, ethnicity, prior gaming use and experience of using role play in training. The only analysis meeting statistical significance was that more males (v. females) and those with experience in online multiplayer games (v. those with no experience) agreed that ‘I can make the computer do what I want it to do while learning using Second Life’ (both P=0.048, two-tailed Fisher's exact test). This result needs to be interpreted cautiously given the multiple statistical analyses performed.
To explore the relationship between simulation fidelity and perceived usefulness, a specific analysis compared participants who agreed that ‘the graphics/visuals were sufficient to make the role play appear realistic enough’ with those disagreeing or uncertain (Table 2). Those demonstrating a positive attitude about simulation fidelity scored significantly higher across all four domain scores and on the total ‘general attitude’ score. Data consensus within this section supports the likelihood that the significance is genuine.
Nominal group data
Delays following unexpected software maintenance resulted in one participant having to leave before the nominal group started; two more left before voting. Twenty-six themes were identified and ranked by their total score (Table 3). Thematic analysis results are covered in the next section.
Discussion
We developed a novel teaching session using Second Life as a role play simulation to teach child psychiatric assessment to medical students, and evaluated it with a multimodal research approach. This section discusses results, structured according to thematic analysis' domains. The key quotes are reported in online Table DS2.
Utility of the Second Life role play to facilitate learning
Whereas e-learning frequently involves knowledge transmission, participants valued the Second Life simulation for also developing skills (Table DS1); furthermore, all participants thought that making/observing mistakes helps avoid making them with real patients. Reference Ziv, Wolpe, Small and Glick11
Other advantages identified include ‘standardising clinical experiences’, voted for by 3/7 nominal group participants, with two more referencing it in the group discussion. The group also identified a closely related theme: that Second Life simulation can expand clinical exposure, also reported in Toro-Troconis et al. Reference Toro-Troconis, Roberts, Smith, Partridge, Zagalo, Zagalo, Morgado and Boa-Ventura6 Although standardisation could be attained by any role play activity, participants noted the technology's capacity to mimic illnesses, for instance colourful clothes in mania or thinness in anorexia.
Combining several nominal group themes revealed that participants valued virtual worlds' potential to allow acclimatisation to clinical environments before interacting with real patients, replicating other studies. Reference Rampling, O'Brien, Hindhaugh, Woodham and Kavia4,Reference Toro-Troconis, Roberts, Smith, Partridge, Zagalo, Zagalo, Morgado and Boa-Ventura6 Acclimatisation also encompasses anxiety reduction. One participant argued that simulation may, however, lull students into a ‘virtual comfort world’, preventing graduation to real clinical environments.
As in other studies, Reference Rampling, O'Brien, Hindhaugh, Woodham and Kavia4,Reference Toro-Troconis, Roberts, Smith, Partridge, Zagalo, Zagalo, Morgado and Boa-Ventura6 our participants identified ‘distance learning’ as an added value of Second Life; the nominal group ranked it third. Moreover, 5/7 participants voted for a closely related issue: Second Life can save time, effort and money on commuting or venue-booking. Furthermore, it may allow learning by anyone at any place, even beyond a single institution. It could therefore be applied to help doctors learn in countries where specific services (e.g. child psychiatry) or clinicians are limited.
Limitations of the Second Life role play
Like other learning technology studies, Reference Conole, de Laat, Dillon and Darby23 participants saw the Second Life simulation as a supplement, rather than substitute, for methods such as patient interaction: ‘nothing replaces the “real thing”’. Discourse relating to this theme focused on the general application of the simulation. In contrast, when specifically discussing its application for child psychiatry, participants instead valued the potential to expand their clinical experience to situations they would not normally encounter. Indeed, the simulation was partly conceived to compensate for the limited and variable exposure to child psychiatry across medical school curricula. Reference Sawyer, Giesen and Walter10 Although access to real-life scenarios would arguably be the ideal learning experience, in practice curricular time pressures and ethical issues challenge such access.
Comparing participants' attitudes of fidelity with other attitude domain scores
| Median scores (IQR)Footnote a | |||
|---|---|---|---|
| Domain scores | Uncertain/disagree/strongly disagree | Agree/strongly agree | P Footnote b |
| Usefulness | 13.50 (10.75-14.50) | 16.50 (15.25-19.25) | 0.02 |
| Behaviour | 11.50 (8.75-13.25) | 14.50 (14.00-18.00) | 0.02 |
| Affective | 20.00 (15.75-23.50) | 25.50 (24.00-27.00) | 0.03 |
| Control | 17.50 (16.00-19.00) | 24.50 (21.50-26.75) | 0.01 |
| General attitude | 62.00 (55.75-68.50) | 80.00 (76.25-90.50) | 0.01 |
IQR, interquartile range.
a. ‘The graphics/visuals were sufficient to make the role play appear realistic enough’.
b. In comparison between the two groups, using two-tailed Mann-Whitney statistical analysis.
The sixth-ranked nominal group theme comprised the limited ‘added value’ compared with real-life role play or other learning opportunities; some participants noted that standardisation and acclimatisation could be delivered by real-life or actor role plays. The questionnaire meanwhile revealed discrepant data (Table DS1). On one hand, half the participants agreed they learned nothing more than what is achievable by real-life role play and most agreed that most things learned from role-playing in Second Life is obtainable through other means. In contrast, only two participants would avoid learning using Second Life, whereas only two disagreed that using Second Life justified the effort.
Aspects of simulation
Simulation involves detaching from real life and absorbing oneself in the simulated scenario. Reference Kneebone16 Virtual worlds can engage as simulations: in Slater et al's 21st-century reprise on ‘Milgram's obedience experiments’, Reference Slater, Antley, Davison, Swapp, Guger and Barker13 participants scored highly on subjective and objective measures of stress on seeing an avatar ‘electrocuted’.
The relationship between simulation fidelity and educational utility has rarely been explored in medical education research. In our study, by detaching students from the teacher and by asking them to interact with an adolescent-appearing avatar, we hypothesised that fidelity may be enhanced. Results showed that some participants felt that being detached from the teacher improved fidelity; however, most felt that fidelity was compromised by the limited expression of body language.
Some participants also noted difficulty in concentrating, suggesting several factors: the need to occasionally multitask, overhearing participants in the real world, and trying to master the technology.
‘Top ten’ themes - positive (coloured) and negative - generated, voted, ranked and ordered by the nominal group
| Rank | Themes generated | DomainFootnote a | Total score | Participants rating theme in ‘top ten’ n |
|---|---|---|---|---|
| 1 | Lack of avatar realism, especially limited/stereotyped body language, impeded psychiatric assessment and group interaction, both of which depend on non-verbal communication |
Sim | 57 | 7 |
| 2 | Conducting the role play in Second Life reduced stress/worry, which eased engagement |
Affect | 35 | 5 |
| 3 | Second Life could facilitate distance learning - students and (expert) teachers can access it from different geographical locations |
Utility | 32 | 4 |
| 4 | Audio delay on Second Life resulted in participants talking over each other | Op | 23 | 4 |
| 4 | Second Life increases potential to act disinhibited (e.g. changing avatar's appearance, walking around), which distracts from learning |
Sim | 23 | 4 |
| 6 | Role-playing on Second Life does not add value compared with existing learning opportunities |
Limitation | 19 | 2 |
| 7 | Second Life provided a useful conceptual barrier between teacher and student, which facilitated role play but still allowed teacher to effectively highlight clinical aspects |
Sim | 24 | 6 |
| 8 | Second Life would allow teaching to be flexible in terms of space (allows for a private/convenient location; does not require booking rooms) and time (can be done at any hour) |
Utility | 21 | 5 |
| 9 | Role-playing in Second Life was interactive and more fun than role-playing in real life |
Affect | 21 | 4 |
| 10 | Compared with interacting in real life, it was harder to maintain concentration throughout the session on Second Life (e.g. as less varied non-verbal communication) |
Sim | 16 | 3 |
| 10 | Conducting the session in the same physical room meant you could hear people in both real life and in Second Life; the audio delay was distracting |
Sim | 16 | 2 |
Op, operational issues; Sim, aspects of simulation.
a. Domains were defined retrospectively by the subsequent thematic analysis.
Detachment from the ‘real’
The seventh-ranked nominal group theme defined Second Life as a useful ‘conceptual barrier’, weakening the sense of interacting with a teacher; 6/7 participants voted for it. However, questionnaire data revealed that most still struggled to ‘switch off’ from thinking the patient was actually the teacher. Furthermore, no role players found that Second Life helped get into role compared with real-life role-playing, implying that any advantage derived from the conceptual barrier may be offset by the simulation's low-fidelity body language. Intriguingly, the fourth-ranked nominal group theme identified the ‘conceptual barrier’ as leading to disinhibited behaviour. Perhaps virtual reality leads users to behave to different norms, particularly as they explore its limits and rules.
Avatars' low-fidelity expression of body language
Questionnaire data showed that most participants expressed difficulty in feeling connected with the patient. The simulation's low-fidelity expression of body language was probably a primary contributing factor: the nominal group identified and ranked this theme highest, and related this to hindering patient engagement and group interaction. Many noted the importance of non-verbal communication to psychiatric practice, and this is reflected in empirical research showing how psychiatric disorders may influence expression of non-verbal communication. Reference Ellgring24
Factors contributing to the low-fidelity body language include limited graphics, although questionnaire data showed variation as to whether this seriously compromised fidelity: four participants thought graphics were sufficient to make the role play appear realistic enough, whereas four disagreed. Another factor is the teacher's inexperience with Second Life; observers noted that gestures occurred only sporadically. Greater teacher experience may have improved fidelity: half the participants agreed that realism would be enhanced with more gestures.
Questionnaire data suggest a positive relationship between simulation fidelity and perceived utility. First, six participants agreed that if the simulation appeared exactly like real life, then the session would have been significantly more useful. Second, participants who agreed that ‘the graphics/visuals were sufficient to make the role play appear realistic enough’ had significantly higher computed scores on all four Bonanno & Kommers'domains, including ‘perceived usefulness’ (Table 2). Reference Bonanno and Kommers19
Given that participants rated an improvement in knowledge and skills, by implication the audio-visuals were arguably at least ‘good enough’. However, some participants experienced the limited representation of body language as significantly detracting from utility, if not compromising it altogether. Surgical simulation research shows examples where low-fidelity simulators still compete favourably with high-fidelity counterparts. Reference Grober, Hamstra, Wanzel, Reznick, Matsumoto and Sidhu25
Some participants reported that low-fidelity body language impeded group interaction. Nevertheless, a reasonable flow of dialogue was observed.
Affect
Various educational models emphasise emotional aspects to learning. Reference Jarvis, Holford and Griffin15,Reference Kneebone16,Reference Lee, Chung and Chen18 Indeed, the nominal group identified two distinct ‘top ten’ themes relating to positive affect, although the thematic analysis revealed two participants reporting stress or boredom, both relating to the limited visual cues.
Whereas interacting with patients and role-playing can unsettle students, Reference Nestel and Tierney9,Reference Moss and McManus26 the second-highest nominal group theme indicated that Second Life attenuated anxiety, thus improving role play engagement. The simulation's conceptual barrier may have reduced anxiety by facilitating detachment from the teacher. Several participants also described feeling less anxious compared with other small-group settings, feeling more confident asking questions or voicing thoughts. The relationship between anxiety and performance is, however, more equivocal: four participants agreed that being more relaxed led to better performance, whereas three disagreed. Some participants noted that anxiety may enhance performance.
The ninth-ranked nominal group theme was that role-playing in Second Life was interactive and more fun than role-playing in real life. The thematic analysis identified six participants describing the session as ‘interesting’ or ‘fun’.
Overall, questionnaire data support the qualitative data (Table DS1). Moreover, the ‘affect domain’ score was high, exceeding comparative scores in a solo game-based Second Life application. Reference Toro Troconis, Meeran, Higham, Mellstrom, Partridge, Peachey, Gillen, Livingstone and Smith-Robbins27 Perhaps enjoyment was enhanced by group interaction.
Many participants noted the novelty of the teaching, but would students enjoy it long term? Lee et al Reference Lee, Chung and Chen18 showed that students' perceived enjoyment of a novel internet-based learning medium was significantly associated with future intention.
Operational and practical issues
Despite this being their first experience of Second Life, and with only 30 minutes' training, difficulties navigating were not raised in the nominal group; questionnaire data showed that most participants felt comfortable with the technology, with some exceptions. This contrasts with the difficulty experienced by students in Rampling et al's Second Life study Reference Rampling, O'Brien, Hindhaugh, Woodham and Kavia4 who received only online guidance. This may reflect the value of face-to-face training, however brief. Although all participants play computer games for leisure, some reported that older, ‘digitally naive’ teachers may have more difficulty. Reference Prensky28
Although fractional, audio delay (the fourth-ranked nominal group theme) was experienced as a challenge to group interaction, particularly as participants could also overhear each other in the ‘real world’ room.
Finally, the Second Life role play requires logistics (e.g. setting up log-on times) and initial resources (e.g. computers, headsets). Although designing the virtual child and adolescent mental health service required initial investment, interested readers are welcome to use it (register with Second Life, then visit http://slurl.com/secondlife/Imperial%20College%20London/150/86/27/).
Participants identified ways that Second Life could actually represent cost-effective learning, relating to the benefits of distance learning or saving on actors.
Study limitations
Recruitment bias may have occurred if particularly enthusiastic students (for psychiatry, technology or education) applied. Furthermore, students at Imperial College - a predominantly science- and technology-oriented institution - may differ to those elsewhere, in that they may be interested particularly in science and technology and thus more likely to be enthusiastic towards innovative learning technology.
Although the nominal group process helps generate breadth of data, the structured approach may restrict in-depth exploration of themes. Admittedly, the thematic analysis aimed to compensate for this, but dialogue still proceeded in accordance with the nominal group process. Thematic analysis is open to subjective interpretation, particularly as only one researcher was involved.
The teacher had limited experience of Second Life; participants' concerns about body-language expression may therefore reflect the teacher's lack of utilisation rather than an intrinsic software issue.
It is important to emphasise that this study evaluated the student experience; although participants generally perceived an improvement in their skills and knowledge, this does not necessarily translate into objective changes in academic performance.
Finally, although some questions explicitly incorporated a comparison (e.g. to real-life role-playing), most were open. Likewise, the nominal group question was open-ended. Students may have made implicit responses with a comparative teaching method in mind. A future study may benefit by making comparisons more explicit or by directly incorporating a control group (e.g. real-life role-playing). Another direction could involve more in-depth qualitative analysis.
Funding
Jisc provided funding (e.g. technological resources).
Acknowledgements
We thank Bang Nong and Julie Edwards for their technical backing, and Sue Smith and Sue English for their support. The study formed part of A.K.V.'s Masters in University Learning and Teaching (Medicine) at Imperial College London. A.K.V. thanks his supervisors Martyn Kingsbury, Annette Mahon, Dave Riley and Jo Horsburgh. We also thank Dr Lorraine Als and Imperial College's Academic Unit of Child and Adolescent Psychiatry.
Open access
eLetters
No eLetters have been published for this article.