Skip to main content

Managing multimodal data in virtual world research for language learning

  • Cristina Palomeque (a1) and Joan-Tomàs Pujolà (a2)

The study of multimodality in communication has attracted the attention of researchers studying online multimodal environments such as virtual worlds. Specifically, 3D virtual worlds have especially attracted the interest of educators and academics due to the multiplicity of verbal channels, which are often comprised of text and voice channels, as well as their 3D graphical interface, allowing for the study of non-verbal modes. This study offers a multilayered transcription method called the Multi-Modal MUVE Method or 3M Method (Palomeque, 2016; Pujolà & Palomeque, 2010) to account for the different modes present in the 3D virtual world of Second Life. This method works at two levels: the macro and the micro level. The macro level is a bird’s-eye view representation of the whole session as it fits into one page. This enables the researcher to grasp the essence of the class and to identify interesting sequences for analysis. The micro level consists of three transcripts to account for the different communication modes as well as the interface activity that occurs in the virtual world of Second Life. This paper will review the challenges when dealing with multimodal analysis in virtual worlds and how the multimodal data were analyzed and interpreted by using a multilayered multimodal method of analysis (3M transcription). Examples will be provided in the study to show how different modes of communication were used by participants in the virtual world of Second Life to create meaning or to avoid communication breakdowns.

Hide All
Anderson, T. L. (2009) Online instructor immediacy and instructor-student relationships in second life. In Wankel, C. & Kingsley, J. (eds.), Higher education in virtual worlds: Teaching and learning in Second Life. Warrington: Emerald Group Publishing, 101114.
Atkins, C. and Caukill, M. (2009) Serious fun and serious learning: The challenge of Second Life. In Molka-Danielsen, J. & Deutschmann, M. (eds.), Learning and teaching in the virtual world of Second Life. Trondheim: Tapir Academic Press, 7989.
Baldry, A. (2000) Introduction. In Baldry, A. (ed.), Multimodality and multimediality in the distance learning age: Papers in English linguistics. Campobasso: Palladino Editore, 1139.
Baldry, A. and Thibault, P. J. (2001) Towards multimodal corpora. In Aston, G. & Burnard, L. (eds.), Corpora in the description and teaching of English. Bologna: CLUEB, 87102.
Baldry, A. P. (2004) Phase and transition, type and instance: Patterns in media texts as seen through a multimodal concordancer. In O’Halloran, K. L. (ed.), Multimodal discourse analysis: Systemic functional perspectives. London: Continuum, 83108.
Bearne, E. (2009) Multimodality, literacy and texts: Developing a discourse. Journal of Early Childhood Literacy, 9(2): 156187.
Bezemer, J. and Jewitt, C. (2010) Multimodal analysis: Key issues. In Litosseliti, L. (ed.), Research methods in linguistics. London: Continuum, 180197.
Book, B. (2004) Moving beyond the game: Social virtual worlds. Proceedings of State of Play 2 Conference. New York: New York Law School, 6–8.
Cambra Giné, M. (2003) Une approche ethnographique de la classe de langue. Paris: Didier.
Cunningham, U., Beers Fägersten, K. and Holmsten, E. (2010) “Can you hear me, Hanoi?” Compensatory mechanisms in synchronous net-based English language learning. The International Review of Research in Open and Distance Learning, 11(1): 161177.
Deutschmann, M. and Panichi, L. (2009) Instructional design, teacher practice and learner autonomy. In Molka-Danielsen, J. & Deutschmann, M. (eds.), Learning and teaching in the virtual world of Second Life. Trondheim: Tapir Academic Press, 2744.
Domingo, M. (2011) Analyzing layering in textual design: A multimodal approach for examining cultural, linguistic, and social migrations in digital video. International Journal of Social Research Methodology, 14(3): 219230.
Downey, S. (2014) History of the (virtual) worlds. The Journal of Technology Studies, 40(2): 5466.
Friedman, D., Steed, A. and Slater, M. (2007) Spatial social behavior in Second Life. In Pelachaud, C., Martin, J.-C., André, E., Chollet, G., Karpouzis, K. & Pelé, D. (eds.), Intelligent virtual agents: 7th International Conference, IVA 2007, Paris, France, September 2007 Proceedings. Berlin Heidelberg: Springer, 252–263.
Hampel, R. and Hauck, M. (2006) Computer-mediated language learning: Making meaning in multimodal virtual learning spaces. The JALT CALL Journal, 2(2): 318.
Hampel, R. and Stickler, U. (2012) The use of videoconferencing to support multimodal interaction in an online language classroom. ReCALL, 24(2): 116137. doi:
Healey, D. (2016) Language learning and technology: Past, present and future. In Farr, F. & Murray, L. (eds.), The Routledge handbook of language learning and technology. Abingdon: Routledge, 922.
Herring, S. C. (2002) Computer-mediated communication on the internet. Annual Review of Information Science and Technology, 36(1): 109168.
Herring, S. C. (2004) Computer-mediated discourse analysis: An approach to researching online behavior. In Barab, S. A., Kling, R. & Gray, J. H. (eds.), Designing for virtual communication in the service of learning. Cambridge: Cambridge University Press, 338376.
Herring, S. C. (2007) A faceted classification scheme for computer-mediated discourse. Language@Internet, 4: 137.
Herring, S. C. (2015) New frontiers in interactive multimodal communication. In A. Georgopoulou & T. Spilloti (eds.), The Routledge handbook of language and digital communication. London: Routledge, 398402.
Hubbard, P. (2009) General introduction. In Hubbard, P. (ed.), Computer assisted language learning: Critical concepts in linguistics. London: Routledge, 120.
Kress, G. (1998) Visual and verbal modes of representation in electronically mediated communication: the potentials of new forms of text. In I. Snyder & M. Joyce (Eds.), Page To Screen: taking literacy into the electronic era (pp. 53–79). London: Routledge.
Kress, G. and van Leeuwen, T. (2001) Multimodal discourse: The modes and media of contemporary communication. London: Arnold.
Lamy, M.-N. (2006) Multimodality in online language learning environment: Looking for a methodology. In Baldry, A. and Montagna, E. (eds.), Third International Conference on Multimodality, Pavia: Palladino, 237–254.
Lee, L. (2002) Synchronous online exchanges: A study of modification devices on non-native discourse. System, 30(3): 275288.
Naper, I. (2011) Conversation in a multimodal 3D virtual environment. Language@Internet, 8(7). http://urn:nbn:de:0009-7-32225
Norris, S. (2004) Analyzing multimodal interaction. A methodological framework. New York: Routledge.
Nunan, D. (1989) Designing tasks for the communicative classroom. Cambridge: Cambridge University Press.
O’Halloran, K. (2004) Visual semiosis in film. In O’Halloran, K. (ed.), Multimodal discourse analysis: Systemic functional perspective. London: Continuum, 109130.
O’Halloran, K. (2011) Multimodal discourse analysis. In K. Hayland & B. Paltridge (eds.), Companion to discourse analysis. London & New York: Continuum, 120137.
Örnberg, T. (2005) Multimodality in a three-dimensional voice chat. In Allwood, J., Dorriots, B. & Nicholson, S. (eds.), Proceedings of the 3rd Conference on Multimodal Communication, Papers in Theoretical Linguistics. Göteborg: Göteborg University, 303–316.
Palomeque, C. E. (2016) Communication in a MUVE: An exploratory case study of teacher interactional devices in Second Life. University of Barcelona, unpublished PhD.
Pellettieri, J. (2000) Negotiation in cyberspace: The role of chatting in the development of grammatical competence. In Warschauer, M. & Kern, R. (eds.), Network-based language teaching: Concepts and practice. Cambridge: Cambridge University Press, 5986.
Peterson, M. (2008) An investigation of learner interaction in a MOO-based virtual environment. University of Edinburgh, unpublished PhD.
Pujolà, J.-T. (2002) CALLing for help: Researching language learning strategies using help facilities in a web-based multimedia program. ReCALL, 14(2): 235262.
Pujolà, J.-T. and Palomeque, C. (2010) Developing a multimodal transcription to account for interaction in 3D virtual worlds: The 3M method. In Bueno Alonso, J. L. (ed.), Analizar Datos. Describir Variación. Vigo: University of Vigo, Servizo de Publicacións, 134145.
Salmon, G. (2009) The future for (second) life and learning. British Journal of Educational Technology, 40(3): 526538.
Sweeney, P., Palomeque, C., González, D., Speck, C., Canfield, D. W, Guerrero, S. and MacKichan, P. (2011) Task design for language learning in an embodied environment. In Vincenti, G. & Braman, J. (eds.), Teaching through multi-user virtual environments: Applying dynamic elements to the modern classroom. Hershey, PA: Information Science Reference, 259282.
Tan, S., O’Halloran, K. L. and Wignell, P. (2016) Multimodal research: Addressing the complexity of multimodal environments and the challenges for CALL. ReCALL, 28(3): 253273.
Tudini, V. (2003) Using native speakers in chat. Language Learning & Technology, 7(3): 141159.
Wang, A. (2015) Facilitating participation: Teacher roles in a multiuser virtual learning environment. Language Learning & Technology, 19(2): 156176.
Warburton, S. (2009) Second Life in higher education: Assessing the potential for and the barriers to deploying virtual worlds in learning and teaching. British Journal of Educational and Technology, 40(3): 414426.
Warschauer, M. (ed.) (1995) Telecollaboration in foreign language learning. Honolulu, HI: Second Language Teaching & Curriculum Center, University of Hawaii at Manoa.
Werry, C. C. (1996) Linguistic and interactional features of Internet relay chat. In S. C. Herring (ed.), Computer-mediated communication: Linguistic, social and cross-cultural perspectives. Amsterdam: John Benjamins 4764.
Wigham, C. R. (2012) The interplay between nonverbal and verbal interaction in synthetic worlds which supports verbal participation and production in a foreign language. Université Blaise Pascal, unpublished PhD.
Williamson, R. (2007) El diseño de un corpus multimodal. Estudios de Lingüística Aplicada, 25(46): 207231.
Yee, N., Bailenson, J. N., Urbanek, M., Chang, F. and Merget, D. (2007) The unbearable likeness of being digital: The persistence of nonverbal social norms in online virtual environments. CyberPsychology & Behavior, 10(1): 115121.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

  • ISSN: 0958-3440
  • EISSN: 1474-0109
  • URL: /core/journals/recall
Please enter your name
Please enter a valid email address
Who would you like to send this to? *



Full text views

Total number of HTML views: 4
Total number of PDF views: 146 *
Loading metrics...

Abstract views

Total abstract views: 328 *
Loading metrics...

* Views captured on Cambridge Core between 24th January 2018 - 23rd July 2018. This data will be updated every 24 hours.