1.1 A Brief Overview of Research on VMC

The literature on VMC in the social sciences with a focus on language and meaning making is practically endless and cannot be summarized in a few pages, owing to the vast range of disciplines (e.g., applied linguistics, pragmatics, experimental psychology, interactional sociolinguistics, linguistic anthropology, media studies, digital ethnography), theories and empirical methodologies (e.g., conversation analysis, mediated discourse theory, multimodal discourse analysis, translanguaging, etc.) and scenarios (e.g., educational, professional, clinical, legal, personal) involved. This section will thus only mention a limited number of studies that, far from fully mapping the extremely wide area of VMC as a meaning-making event and as a practice, will help readers pinpoint areas of special interest for this domain, as it has been evolving in the last two decades.

From a theoretical standpoint, several disciplines and approaches have tackled VMC since its emergence in the electronic arena of the Internet. Early research on VMC was focused on comparisons between face-to-face communication and VMC in synchronous or asynchronous contexts, for example in experimental psychology settings, by exploring dialogue organization and task performance (Reference O’Conaill, Whittaker and WilburO’Conaill, Whittaker & Wilbur, 1993; Reference Doherty-Sneddon, Anderson and O’MalleyDoherty-Sneddon et al., 1997). These concerns were likewise relevant to conversation analysis (CA), which developed three main areas of enquiry, namely language-based, social-ethno-anthropological and mixed approaches to the study of ‘online talk’ (Reference Paulus, Warren and LesterPaulus, Warren & Lester, 2016). Studies within the tradition of interactional sociology have shown the productivity of the application of participatory frameworks for the comprehension of media affordances (Reference HutchbyHutchby, 2014), as derived from Goffman’s model (Reference GoffmanGoffman, 1971, Reference Goffman1981). Among other pertinent interactional and sociolinguistic models, Goodwin’s cooperation model illustrates the laminated organization of human action, whereby individuals cooperatively build action by putting together different semiotic signs that are part of a medium and, at the same time, shape how the medium itself works (Reference GoodwinGoodwin, 1989, Reference Goodwin2000, Reference Goodwin2013). Another take on the question is Scollon’s mediated discourse theory (Reference Scollon2001), which studies how practices are learnt and socialized by individuals and communities, with a focus on the mediated nature of communication (particularly so in the case of VMC), on background sites of engagement as entry points of observation in space and time, and nexus of practice as the node of social practices and mediated actions. Other examples from the social sciences include visual ethnography (Reference PinkPink, 2007), pragmatics of computer-mediated communication (Reference Herring, Stein and VirtanenHerring, Stein & Virtanen, 2013), ethnomethodology and CA (Reference RintelRintel, 2015), and other multidisciplinary work, such as the development of the visual data framework for the analysis of diverse video datasets, including mobile phone footage, body-worn cameras and video surveillance (Reference Legewie and NassauerLegewie & Nassauer, 2022).

Multimodal, sociosemiotic and translanguaging approaches have also developed systems to tackle the richly semiotic nature of VMC. The most current and relevant approaches here include multimodal visual analysis (Reference van Leeuwen and Jewittvan Leeuwen & Jewitt, 2001), multimodal interaction analysis (Reference NorrisNorris, 2004), multimodal transcription and text analysis (Reference BaldryBaldry, 2005; Reference Baldry and ThibaultBaldry & Thibault, 2006, Reference Baldry, Thibault, Vasta and Baldry2020) and systemic functional-multimodal discourse analysis (Reference O’HalloranO’Halloran, 2004, Reference O’Halloran2008). A core methodological concern in this regard is the transcription and annotation of multimodal data to capture and reproduce the contribution of each semiotic resource, including, but not limited to, language to make meanings (see Reference Flewitt, Hampel, Hauck, Lancaster and JewittFlewitt et al., 2009).

In a recent chapter, Reference Sindoni, Moschini, Moschini and SindoniSindoni and Moschini (2021b: 205–6) summarized four major areas of interest connected with VMC in its synchronous and asynchronous formats, namely (1) the relationship between configurations of interpersonal communication; (2) affordances and constraints of technologies; (3) organization and distribution of turn-taking; and (4) their multimodal organization, not limited to language.

Thematically, task-oriented studies have been crucial for this domain’s development, most specifically when investigating how tasks are interactionally organized in the workplace (Reference Heath and LuffHeath & Luff, 1992) and in educational contexts (Reference Swan, Richardson, Ice, Garrison, Cleveland-Innes and ArbaughSwan et al., 2008). In pedagogical scenarios, for example, Reference Ho and TaiHo and Tai (2021) applied a multimodal and translanguaging lens to investigate how teachers use YouTube videos for role play in English-language online classes by multimodally drawing on different styles, registers and experiences and how educational materials were designed by teachers of English by orchestrating their multimodal and translingual repertoires. More recently, a systematic attempt to bring together multimodality and translanguaging in the context of asynchronous language learning on YouTube has been devised and applied to the ‘street language learning’ context by analysing how the vlogger Anming draws on her plurilinguistic, multisemiotic and multimodal repertoires to promote language learning ‘on the move’ (Sindoni & Ho, unpublished data).

Regardless of the theories and methods adopted, thematic studies have recurrently placed a focus on educational aspects, with reference to language learning and translingual practices, such as exploring the use of videos for English-language teaching (Reference Kessler, Loewen and TregoKessler, Loewen & Trego, 2021), as well as on the multimodality of video-mediated interaction (VMI), for example analysing the influence of tasks on discourse strategies in virtual 3D settings (Reference ShihShih, 2014) and on students’ engagement with 360-degree videos through multimodal data analytics (Reference Tan, Wiebrands, O’Halloran and WignellTan et al., 2020). Other studies have examined the management of personal media spaces in one-to-one and multiparty interactions as they emerge in specific media platforms, such as young people’s use and understanding of spoken–written variation in Skype calls (Reference SindoniSindoni, 2011), affordances of online video game live streaming (Reference RecktenwaldRecktenwald, 2017), cross-modal engagements in Google Hangouts (Reference Rosenbaun, Rafaeli and KurzonRosenbaun et al., 2016a, Reference Rosenbaun, Rafaeli and Kurzon2016b) and media ideologies in relation to FaceTime and WhatsApp (Reference Busch, Sindoni, Groff, Hollington and Hurst-HaroshBusch & Sindoni, 2022).

The intrinsic, context and culture-dependent vitality of VMC transpires from the multiplicity of theories and methods used to make sense of, and learn from, the interactions conducted online in a wide range of settings well before the Covid-19 pandemic. Within this multiplicity of approaches, this Element specifically combines theories of multimodality and translanguaging. In Section 1.2, the benefits of selectively integrating theories and the empirical methods adopted in these two theoretical models are illustrated to show how their combination mutually invigorates and broadens their agenda.

1.2 Multimodality and Translanguaging: A Common Reformist Agenda

Video-mediated communication is inherently multimodal, like any other communicative event, but it is generally richer and, overall, more complex to follow and make sense of due to the intense and interlaced co-presence of semiotic resources, or modes, that contribute to overall meaning making. Multimodality has been defined as both a theory and a method to unravel such complexities by unpacking the resources that contribute to meaning making (Reference JewittJewitt, 2014). Multimodality derives from the sociosemiotic model developed by Michael A. K. Halliday in the 1970s, who expanded the notion of text to include semiotic resources other than language:

This idea was not certainly new, as it can be dated back to the well-known Saussurean conception of semiology as the science of signs (Reference de Saussure, Meisel and Saussyde Saussure, [1916] 2011). Nevertheless, Halliday accelerated an intellectual movement of semioticians and critical linguists, including Gunther Kress, Theo van Leeuwen, Bob Hodge and the New London Group, who set out to account for the systems of regularities of semiotic resources and modes (the so-called grammars) other than language (see, for example, Reference O’TooleO’Toole, 1994; Reference Kress and van LeeuwenKress & van Leeuwen, [1996] 2020). Particularly relevant insights for our context were presented in the manifesto by the Reference GroupNew London Group (1996), who heralded the notion of multiliteracies. These were meant to overcome ‘traditional language-based approaches’ to learning, so as to challenge orthodox literacy pedagogy, which was firmly grounded on reading and writing practices connected to ‘page-bound, official and standard forms of the national language’ (Reference Group1996: 61, my emphasis). They made the important point that literacy pedagogy had been designed as a project, ‘restricted to formalized, monolingual, monocultural, and rule-governed forms of language’ (Reference Group1996: 61, my emphasis). The concept of national language, which had prevailed at that time in curriculum design, content and forms of assessment, needed to be challenged in that it projected a monolingual and monocultural context that was a myth rather than a reality. In response to the global communicative phenomena that had already started to gain momentum in the 1990s, their plan for pedagogical reform was proposed to account for the diversity of texts, discourses, languages and cultures that were circulating at the time.

An emphasis on visual ‘language/s’ amidst the rapid cha(lle)nges of modernity and an increased need to raise awareness about the multicultural and plurilinguistic repertoires of speakers was felt more acutely in the geographical areas where the New London Group was joining forces, namely the United States, Europe and Australia. They anticipated the development of a metalanguage of multiliteracies based on the concept of ‘design’, framed within a theory of discourse in which semiotic activities ‘are creative application and combination of conventions (resources – Available Designs) that, in the process of Design, transforms at the same time it reproduces these conventions’ (Reference GroupNew London Group, 1996: 74, my emphasis).

In the New London Group authors’ view, the Available Designs include the grammars of the other semiotic systems, such as language, film, gesture and photography. Hence, within the order of discourses, Available Designs take on different shapes, becoming styles, genres, voices, and so on. In other words, when speakers design texts, events and interactions, they draw on systems of sociolinguistic practice as well as on grammars. Crucially, the authors argued that ‘the process of shaping emergent meaning involves re-presentation and recontextualization. This is never simply a repetition of Available Designs. Every moment of meaning involves the transformation of the available resources of meaning. Reading, seeing, and listening are all instances of Designing’ (Reference Group1996: 75, my emphasis). Hence, in their interpretation (and subsequent ones), meaning making can emerge out of Available Designs, or resources, that are never reproduced automatically by sign makers, but imply an act of creative transformation on their part.

Production of ‘grammars’ ensued. These described systems of meaning making as part of this much-needed treasure trove of multiliteracies. These grammars were intended to add a breath of epistemological fresh air to ‘language-alone’ models of literacy, for example with descriptions of regularities of different systems of meaning making, such as the built environment of displayed art (Reference O’TooleO’Toole, 1994), (mostly) still images (Reference Kress and van LeeuwenKress & van Leeuwen, 2020), music (Reference van Leeuwenvan Leeuwen, 1999), short films (Reference Baldry and ThibaultBaldry & Thibault, 2006), mathematics (Reference O’HalloranO’Halloran, 2006), document layout (Reference BatemanBateman, 2008) and three-dimensional texts (Reference Ravelli and McMurtrieRavelli & McMurtrie, 2015), to name but a few. Along with the development of these ‘grammars’ as models to act upon resources and empower the agency of the sign maker, the manifesto prompted the emergence of several pedagogical reformist agendas grounded on the intrinsic transformative nature of meaning making (Reference Kress, Cope and KalantzisKress, 2000, Reference Kress2003). Additionally, by resisting approaches that were exclusively focused on language as the sole resource to make meanings and to understand communication, multimodally oriented studies started to examine texts, events, performances or any other kind of artefacts (such as objects, toys and buildings) that could even not include any language at all. In this sense, the reformist project of multimodality was concerned with proving that communication went well beyond language and that linguistics needed to embrace a more far-reaching view and, at the same time, be informed by sociosemiotics.

The reformist agenda that lies at the core of the historical origins of multimodal and sociosemiotic studies can likewise be traced back to the earliest conceptualizations of the translanguaging project. Within one tradition, the term derives from the Welsh trawsieithu coined by Reference WilliamsCen Williams (1994) and translated into English by Colin Baker in 2011 (see Reference García and LiGarcía & Li, 2014; Reference García, Lin, García, Lin and MayGarcía & Lin, 2016). It originally referred to bilingual students involved in the pedagogical practice of alternating English and Welsh in class activities, for example speaking in one language and writing in the other. Thanks to movements from various parts of the world advocating the pursuit of linguistic rights in the second half of the twentieth century, bilingualism and multilingualism started to be seen as political constructs. The development of translanguaging theories has been paramount in this paradigm shift. García explained this shift by contending that translanguaging is ‘an approach to bilingualism that is centred not on languages as has been often the case, but on the practices of bilinguals that are readily observable’ (Reference García2009: 44). In her view, translanguaging is a set of ‘multiple discursive practices in which bilinguals engage in order to make sense of their bilingual worlds’ (Reference García2009: 45, emphasis in the original). In a similar vein, Reference Creese and BlackledgeCreese and Blackledge (2010) challenge the ‘two solitudes’ assumption of bilingual education (see Reference Cummins and HornbergerCummins, 2008) and propose a model of ‘flexible bilingualism’ that places speakers at its centre, but which is not idealized and does not hold monoglossic beliefs about language. Other significant lines of enquiry are pursued by Canagarajah, who has highlighted the dual nature of translanguaging as a theory and as a practice (Reference Canagarajah2013, Reference Canagarajah2020, Reference Canagarajah2022). Taking monolingual orientation (Reference CanagarajahCanagarajah, 2013) as his point of departure, he developed the notion of codemeshing. This recognizes the fluid nature of communication and the hybridization of languages and varieties that become meshed with other symbolic systems, such as icons, emoticons or other graphical resources and modalities, including images, audio and video.

The reformist agenda is clearly foregrounded in Li’s work. The dimensions of ‘creativity’ and ‘transformation’ resonate in Li’s theoretical design of translanguaging as the act of seamlessly navigating through linguistic structures and systems across modalities, such as speaking, writing, signing and remembering, but at the same time transcending them (Reference LiLi, 2011). Li draws attention to the transformation implicit in what he defines as a ‘translanguaging space’:

Li’s approach mirrors the general translanguaging project to ‘decolonize’ linguistics from the ‘lingual bias’ found in much second language acquisition research, including that concerned with bilingualism and multilingualism (Reference Block and MayBlock, 2014). The ‘lingual bias’ equates overall communication with language properties (e.g., phonology, morphology, syntax, lexis), thus disregarding questions of pragmatic, intercultural and learning dimensions and not addressing ‘embodiment and multimodality as a broadened semiotically based way of looking at what people do when they interact’ (Reference Block and MayBlock, 2014: 56). This decolonizing project, however, is organized around a focus other than the one identified in multimodal studies. Translanguaging has been broadly designed as the attempt to resist monoglossic views that posit the existence of abstract linguistic structures in separate areas of bilingual and multilingual speakers’ brains. On the theoretical level, the different languages and varieties started to be conceptualized, and consequently described, as multifaceted linguistic repertoires that had only been heuristically and ideologically named as ‘distinct’ languages. The clear-cut boundaries that distinguished one language or variety from another in fact reflected normative, prescriptive and top-down approaches within an underlying political agenda. The different linguistic repertoires could, and indeed should, be represented as plurilinguistic continua interlaced with the personal, individual, social and cultural life trajectories of language users. The names of languages are therefore political constructs rather than intrinsically linguistic descriptive labels. The names of languages are outward oriented rather than inward oriented.

Against these deterministic views, Li proposed to repurpose the theoretical agenda of (applied) linguistics by formulating the paradigm-shifting claim that language is ‘a multilingual, multisemiotic, multisensory, and multimodal resource for sense- and meaning-making’ (Reference LiLi, 2018: 22). This intellectual endeavour was framed around one focus that still today foregrounds language analyses (mostly, but not exclusively within educational scenarios), with a sensitivity to multilingual and plurilingual contexts, as well as pragmatic, intercultural and sociosemiotic realities (Reference Zhu, Li and Jankowicz-PytelZhu, Li & Jankowicz-Pytel, 2020). Meaning making, within translanguaging perspectives, is understood as a semiotic act and choice in ways that run parallel to those theorized in multimodality. However, while translanguaging theories are primarily focused on examining how language users draw on their plurilinguistic and multisemiotic repertoires to make and exchange meanings, multimodality may analyse events, practices and artefacts that have no language whatsoever included.

Even though multimodal and translanguaging theories come from different pedagogical, sociosemiotic and linguistic research traditions and directions, they share a profoundly reformist agenda that puts speakers – wherever their life worlds and trajectories are – at the core of meaning making, as they are immersed in ever-evolving creative and transformative practices. If the New London Group manifesto and ensuing multimodal and pedagogical theories contested the automatic primacy assigned to monolithic literacy pedagogies (Reference Cope, Kalantzis, Sindoni and MoschiniCope & Kalantzis, 2021), translanguaging, as a ‘practical theory of language’ (Reference LiLi, 2018), reconceptualizes language as broader than a phonetic, lexical, grammatical and syntactical complex of structures so as to incorporate its multilingual, multisemiotic, multisensory and multimodal components (Reference LiLi, 2018: 22).

While resisting approaches that prioritize language and adopt monoglossic views, both multimodality and translanguaging theories, in their multiple strands, recognize that ‘human languaging activity is radically heterogeneous and involves the interaction of processes on many different time-scales, including neural, bodily, situational, social, and cultural processes and events’ (Reference ThibaultThibault, 2017: 76). These combined approaches to communication provide a fully fledged model that reflects the need to incorporate the multilinguistic and multisemiotic repertoires of speakers, in order to capture the complexities of contemporary communication in the classroom and beyond (Reference Zhu, Li and Jankowicz-PytelZhu, Li & Jankowicz-Pytel, 2020) in its analysis and description. Repertoires are richly interlaced ensembles of available resources and modes that speakers orchestrate to communicate in a variety of contexts, practices and across a range of forms of embodied and/or disembodied configurations that make meanings and cause them to be circulated. By way of summary, multimodality and translanguaging have distinct theories, methodologies, research foci and different preferential objects of study. Consistent with their historical (and parallel) developments that addressed two major concerns in linguistics – logocentrism in multimodality versus monoglossic ethnocentrism in translanguaging – these two theories are highly compatible and can contribute to increasing awareness about how communication works in plurilingual, multisemiotic and multimodal scenarios, with both embodied and disembodied resources for meaning making.

1.3 An Overview of This Element

Building on the arguments provided in the previous sections, this Element begins by assuming that multimodality and translanguaging can reciprocally benefit from each other. In what follows, the resources and modes of VMIs will be analysed by combining the insights and methodologies of multimodality and translanguaging. The overarching aim of this Element is therefore to show how these two theories and methodologies can be complementary to unpack communicative events, be they mediated or not. In subsequent sections, multimodality and translanguaging will not be merged or incorporated into one another. They will be selectively used to illustrate the two models in action in the context of VMC instead.

The broad theoretical question that will be addressed in this Element deals with how multimodality and translanguaging can work in partnership to understand the meaning making in VMC and beyond. The underlying assumption is that both share a common reformist agenda that underpins their models. This reformist agenda is neither abstract nor vague and encapsulates two principles that will be shown in action. The first multimodal principle is that communication goes beyond the verbal and that the verbal can in fact be missing without compromising successful communication. The second translanguaging principle is that the labels of languages and varieties (e.g., English, Tagalog, Spanish, Sicilian or Mandarin Chinese) are political constructs and that what language users do with languages (and/or with any other semiotic resource or mode) cannot be pinned down by clear-cut linguistic taxonomies that do not account for personal and community life trajectories or for specific pragmatic or functional goals.

This Element will systematically map the resources that come into play in VMC by accounting for the translinguistic/plurilinguistic, multisemiotic and multicultural repertoires of speakers as they perceive and make sense of them. The embodied resources include speech, gaze, proxemics and kinesics patterns, while disembodied resources incorporate writing and images, among others (Reference SindoniSindoni, 2013), in ways that are made salient and meaningful by participants in situated contexts.

In Section 2, the resource of language will be unpacked in the typical multimodal conceptualization of mode (or resource), that is, more specifically, in the configuration of speech and writing rather than that of language tout court. Kress defined mode as a ‘socially shaped and culturally given resource for making meaning’ (Reference Kress and JewittKress, 2009: 54), thus carrying ‘the discernible regularities of social occasions, event and hence a certain stability’ (Reference KressKress, 2010: 8). He argued that common instances of modes are ‘speech; still image; moving image; writing; gesture; music; 3D models; action; colour’ (Reference KressKress, 2010: 28, emphasis in the original). Kress himself problematized the speech/writing labelling that, in his view, was not so clear-cut if placed in a cultural context. To corroborate his problematization of the speech/writing categorization, he mentioned as examples the different script systems for writing and the distinction between tone languages and intonation languages for speech (Reference KressKress, 2010). However, even within these caveats, language cannot be semiotically unpacked into the categories of speech and writing alone. This semiotic categorization is mostly unsatisfactory, and for a number of reasons. Viewing language as a resource only in its spoken and written components is in fact certainly highly reductionist. The question of how different languages and varieties can semiotically influence and shape communication is mostly left underexplored in multimodal research (Reference Kress and van LeeuwenKress and van Leeuwen, 2001; Reference Kress and JewittKress, 2009, Reference Kress2010). Even though several multimodal studies deal with translingual and intercultural case studies, those languages and varieties are not conceptualized as semiotically different, but rather as intrinsically similar languaging variants. This Element wishes to draw attention to the distinct ontologies of languages and varieties in the moment they are used in context. These distinct ontologies incorporate distinct but compatible forms of language variation (be they from speech to writing or from one language to another one) as triggers of creativity and transformation, along the lines indicated by Reference LiLi (2011).

The case studies in Section 2 will illustrate how users make sense of alternations between speech and writing, defined as mode-switching (i.e., alternation of speech and writing within the same communicative event). These will be used as a frame to introduce Section 3, which presents other forms of alternations, namely those between and across different languages and varieties, to examine similar patterns of meaning making, as well as differences. Rather than forcing interpretative evaluations on the patterns emerging from the video samples and interviews, the analyses will be more focused on showing participants’ reflections on their and others’ video interactions, along with their diverse degrees of metapragmatic awareness.

In Section 4 and 5, other patterns of meaning making in VMC will be illustrated, namely the use of gaze in its manifold configurations in video-mediated environments (e.g., lack of mutual gaze, practice of self-looking), as well as rearrangements of personal space and social distance (what will be defined as ‘staged proxemics’ in Section 5) and bodily presence, as mainly instantiated in head, hand and torso movements in the video frame.

Within these scenarios, the complex notion of ‘intercultural communication’ will be tackled as perceived and made sense of by individuals. It is a wide, interdisciplinary and multifaceted area and a discussion of its many interlaced components goes beyond the scope of this Element. When explicitly mentioned, intercultural communication will be identified here as ‘concerned with how individuals, in order to achieve their communication goals, negotiate cultural or linguistic differences which may be perceived relevant by at least one party in the interaction’ (Reference ZhuZhu, 2019: 2010). The suggestion to problematize ‘the notion of cultural identities’ and emphasize ‘the emergent, discursive and inter nature of interactions’ (Reference ZhuZhu, 2013: 9, emphasis in the original) will be taken up in the analyses of discourse practices of learners and research participants.

1.4 A Methodological Note on Datasets

The subsequent sections selectively draw on several datasets used to illustrate examples of how resources are used in VMIs that were collected over a ten-year time span. This video corpus comprises over 900 hours of recorded interactions in a wide variety of contexts, such as educational settings, leisure and affinity spaces, and taken from a range of platforms, including one-to-one (e.g., Skype and FaceTime) and multiparty (e.g., Camfrog). All the data has been collected ethically, that is, always informing participants of research purposes and distributing informed consent forms. Names have been changed when possible, that is, when it did not involve changing the original students’ transcriptions, which are all reproduced here unedited.

A small sample of this larger dataset includes recorded video interactions of university learners in two research projects, namely: (1) twenty-one video interactions from the European-funded project EU-MaDE4LL, European Multimodal and Digital Education for Language Learning (2016–19), (2) twenty-four from the Italian national project MoM, Multimodality on the Move (2014–15) and (3) thirty-two VMIs from the VIDEO dataset (‘Video Interactions in Digital EnvirOnments’) that I obtained from my research on these topics and from a variety of participants, giving a total of seventy-seven VMIs in one-to-one contexts.

The video data from the first two research projects was obtained in a controlled environment and under similar conditions. The VMIs were recorded for research purposes in university contexts and by students who had been informed beforehand of the projects’ learning objectives. Both projects had common core classes taught by five researchers on five different digital texts,Footnote ¹ with multimodality, digital literacies and English as the language of intercultural communication as the three pedagogical milestones. Core courses for undergraduate and postgraduate students were delivered by the researchers in their home universities. All the researchers then visited the other universities to deliver their workshops on the digital text in which they were experts. I taught the workshop on VMIs in all the classes involved in both projects (nine altogether, four undergraduate and five postgraduate), in relation to personal and professional contexts (e.g., video conversations and job interviews).

The final assignment for the students who had chosen VMIs as the digital text for end-of-course formal submission consisted of four tasks: (1) a recording of a video call with a friend or with another student; (2) a multimodal transcription of a maximum of three minutes of interaction on a pre-filled grid, meant to unpack all the multimodal resources involved; (3) a written analysis of c. 3,000 words that explained how learners had gone about the multimodal transcription and how it had helped them to make sense of the overall interaction; (4) a peer-assessment grid about another student’s transcription and written analysis. All peer-assessment tasks involved evaluation of the same digital text type, so that learners had the chance to assess the same tasks they had already carried out at an earlier stage of the project. These materials amount to a total of 192 university students’ digital and multimodal productions only for VMIs.Footnote ²

The Peer Assessment Form and the Teacher Assessment Form, which have been developed within the EU-MaDE4LL project, are supplementary materials to this Element and can be freely downloaded, adapted and reused in the classroom. The Guidelines for Classroom Activities at university level for the module on VMIs are also provided in this Element.Footnote ³

Excerpts of short video conversations between university students complemented by their own multimodal transcriptions and annotations are selectively inserted in the sections to show how conversations can be linearly analysed and how the students’ media, multilingual and multimodal ideologies (Reference GershonGershon, 2010a and Reference Gershon2010b) emerge from transcription and annotation tasks. These provide insights into their perceptions, understanding and sense making of translinguistic and semiotic resources (Reference BuschBusch, 2018; Reference Busch, Sindoni, Groff, Hollington and Hurst-HaroshBusch & Sindoni, 2022). All the EU-MaDE4LL teaching and learning materials and content have been used to develop the Common Framework of Reference for Intercultural Digital Literacies (Reference Sindoni, Adami, Karatza, Marenzi, Petroni and RoccaSindoni et al., 2019).

2.1 Speech and Writing: Is This Still a Useful Distinction?

Whatever their communicative function may be, synchronous and spontaneous spoken face-to-face interaction displays recurring conversational planning and structuring patterns that can be broadly summarized as: (1) shared co-presence of participants; (2) common situational context; (3) involvement strategies; (4) other-paced dynamics, and (5) features typical of spontaneous interaction, such as repetition, pauses, hesitations, self-corrections, use of discourse markers (Reference SchiffrinSchiffrin, 1986), rephrasing, and so on (Reference SindoniSindoni, 2013, see also Reference Laver, Kendon, Harris and KeyLaver, 1975), along with other non-verbal features, such as gaze distribution, social distance (Reference HallHall, 1966) and body movement organization (i.e., kinesics, see, e.g., Reference ScheflenScheflen, 1964 and Reference Ekman and FriesenEkman & Friesen, 1969), or halfway strategies spanning verbal and non-verbal features, such as prosodic and intonational modelling.

As argued by Reference KressKress (2003), the association between mode and medium used to be more straightforward before the advent of the digital era, for example writing used to be immediately associated with a book or with book-like features, but today this association no longer holds. Writing displays the features of oral language in a number of different contexts, being used for spontaneous, non-editable, redundant and non-standard communication in digital settings (Reference BaronBaron, 2000). VMIs are one interactional digital scenario where the distinction between spoken and written discourse becomes blurred and fuzzy (Reference SindoniSindoni, 2013). The daily use of social media, email writing, smartphone use, and interconnected devices and platforms displays how has this distinction been further blurred. Speech is still interactional and makes use of involving and cooperative strategies, but writing has been consistently used in ways that can be compared to speech. The advent of digital writing has normalized informality, repetitions, spontaneity and the quick decay of anything written. Writing is used for social media posts and comments and in instant messaging apps, and has taken many unprecedented forms. Against the backdrop of the digital conversational world, writing has perhaps never been less permanent than before in the history of human communication. Hence the question as to whether the distinction between speech and writing still needs to be made has been raised.

Several multimodal researchers prefer the distinction of speech and writing rather than the more abstract definition of language. As made explicit in their textbook, Reference Jewitt, Bezemer and O’HalloranJewitt, Bezemer and O’Halloran (2016: 15) claim that ‘by “language” we mean speech and writing’. Kress provides a clear explanation for this preference. He argues that it would be a hard task to find principles of coherence for different features in speech (e.g., what pitch variation has in common with lexis, or what connects levels of energy, such as loudness and softness, with syntax). It would be likewise hard to find principles of coherence for disparate sets of features in writing, such as what fonts have in common with lexis or orthography with syntax (Reference KressKress, 2010). He then provides the following argument:

More recently, Diamantopoulou and Ørevik published a volume on multimodality in English-language learning scenarios, adopting a transnational and ‘international outlook’, as they put it (Reference Diamantopoulou and Ørevik2022: 4). Their description of ‘mode’ follows the tradition of multimodal social semiotics: ‘a culturally shaped semiotic resource, such as writing, speech, movement, gesture, and gaze’ (Reference Diamantopoulou and ØrevikDiamantopoulou & Ørevik, 2022: 9), hence with no explicit reference to if and how diverse translingual practices may impact on meaning making. In their attempt to delineate the principles of a general semiotics, Reference Bezemer and KressBezemer and Kress (2016) had made the point that this theory should identify which semiotic principles are shared and how these principles are differently realized in each mode (as explained in Reference Jewitt, Bezemer and O’HalloranJewitt, Bezemer & O’Halloran, 2016). This interest in what is common and what is different, what the gains and losses are in the use of each mode, and which affordances can activate or silence meaning potential is a central concern for multimodal studies (Reference KressKress, 2005). However, the question of whether drawing from plurilinguistic repertoires can be considered only in terms of speech and writing is left underexplored.

Translanguaging theories likewise show interest in the concepts of the modes of speech and writing as descriptive for language use. For example, Baynham and Lee revisit the work of the linguist Whorf and argue that ‘every language has its affordances, enabling some ways of thinking and disabling others. … It is perhaps even more obvious that different modes (spoken/written/visual/gestural/embodied) have different affordances’ (Reference Baynam and LeeBaynham & Lee, 2019: 29). With this claim, they connect modes with affordances and seem to adhere to the general semiotic principle of the speech/writing dyad as accounting for language affordances (or materiality).

If it is certainly true that all languages and varieties are sign systems (i.e., made up of modes to make meanings), it is questionable whether the dyad of speech and writing is satisfactory to describe all forms of alternation, meshing, mixing across modes, languages and varieties. This question will be taken up further in Section 3.

In this section, speech and writing will be used as heuristic categories suitable for describing the distinct ways of making meaning by means of their different affordances (i.e., what one can do and cannot do with them) and by means of the meaning making of their medium-specific alternations in VMC. Speech and writing in fact follow different coherence principles; they activate and silence different affordances. In VMI, the readily available affordance of writing a message in the text chat box makes its use far more frequent than could be found in face-to-face encounters. The use of writing in offline contexts cannot be excluded a priori, in cases such as passing a written note to another person or sending a private text message by WhatsApp or any other instant message platform, but the exchange of written messages in VMI is much more frequent and normalized (Bateman et al., unpublished data; Reference SindoniSindoni, 2011, Reference Sindoni2013, Reference Sindoni2019).

Section 2.2 explores these VMI-specific patterns in more detail.

2.2 Speech and Writing in VMC

Unlike face-to-face conversations, which typically rely heavily on speech to make and exchange meanings, video conversations allow for the incorporation of written turns that can be inserted in the chat box of the most common video platforms (Reference Sindoni, Adami, Karatza, Marenzi, Petroni and RoccaSindoni et al., 2019; Reference Sindoni, Adami, Karatza, Moschini, Ørevik and DiamantopoulouSindoni, Adami, Karatza, Moschini et al., 2021; Reference Sindoni, Adami, Moschini, Petroni, Karatza, Sindoni and MoschiniSindoni, Adami, Moschini, Petroni & Karatza, 2021). These written comments can be employed for a variety of reasons, ranging from technical issues, for example when speakers cannot hear each other or need to fill in gaps over some time lag, to pragmatic and cultural uses, such as communicating more personal contents (e.g., in one-to-one contexts) or as a face-saving strategy, with writing perceived as a ‘less embarrassing’ and ‘less compromising’ resource when potentially sensitive subjects are touched on (Reference Sindoni, Cambria, Arizzi and CoccettaSindoni, 2012, Reference Sindoni2014). I have labelled this medium-specific pattern as mode-switching, alluding to the spontaneous alternation of speech and writing within the same communicative video exchange (Reference SindoniSindoni, 2011). Examples taken from video-based interactions document different pragmatic and cultural practices, useful for unveiling speakers’ language and media ideologies (Reference BuschBusch, 2018; Reference GershonGershon, 2010b; Reference ZhuLi & Zhu, 2013), as well as participatory engagements, cross-model exchanges (Reference Rosenbaun, Rafaeli and KurzonRosenbaun et al., 2016a, Reference Rosenbaun, Rafaeli and Kurzon2016b) and media asymmetries (Reference Heath and LuffHeath & Luff, 1992). Starting from heuristically categorizing several forms of mode-switching (e.g., self-initiated versus other-initiated), other patterns of speech-writing variation are mentioned in this Element, with reference to culture, context, setting and media affordances (Reference GibsonGibson, 1986 [1979]). These patterns are analysed to explore how these variables influence the way in which speech and writing are alternated in video-mediated environments.

This toolkit is meant to encourage awareness about how modes and media facilitate communication in some circumstances (for example, the possibility of alternating speech and writing by keeping the text chat box open to allow for multiple conversation floors), but hinder mutual intelligibility if resources are not distributed meaningfully and/or evenly (for example, if teachers leave these channels open haphazardly or if learners do not know how to avoid overlapping in spoken/written modes). The two main research questions dealt with in Section 2.3 are:

• RQ1: What are the main communicative functions of mode-switching in video conversations?

• RQ2: How do speakers take advantage of media affordances in video-conferencing platforms? How do they switch from speech to writing and vice versa and to what end?

2.3 Speech, Writing and Mode-Switching in VMC

The communicative feature that will be the focus of this section is specific to VMC, but is not related to any particular platform as all client systems supporting synchronous and video-based exchanges allow users to decide which channel can be used. As already anticipated, using writing in spoken interaction is certainly feasible in face-to-face scenarios, such as giving someone a slip of paper while talking, or even writing on the blackboard while speaking in a classroom context. However, the patterns of regularities that characterize the alternation between speech and writing in VMIs allow systematic and empirical descriptions.

A first basic difference has to do with media affordances, and in particular with the number of people involved in communication. One-to-many means that one participant broadcasts live, while the others are not connected with each other. This type of VMI will not be dealt with in this Element, as no direct and ongoing interaction is entailed. However, as both involve a live exchange between users, the distinctions between many-to-many and one-to-one interactions are dealt with in the following two sub-sections.

2.3.1 Many-to-Many VMIs, or Multiuser Video Chat

Many-to-many VMIs, as the label suggests, are meetings that include multiple users found in professional contexts, such as business meetings and educational settings, including online video classes, which, despite their widespread use during the Covid-19 pandemic, were already a well-established aspect of remote learning contexts, whose features are defined and investigated in this section with reference to a multiparty video platform.

From 2011 to 2013, I analysed several hours of video interactions in a Camfrog room named SpeakEnglish for people interested in conversations in English. Camfrog is a video-based online community for up to one thousand users in a single room. Media affordances of this platform allow the visualization of participants’ screens, their roles and status, and of a foregrounded text chat box used to post comments and add multimodal materials, such as emoticons, emojis, virtual gifts, and much more besides. The platform’s configuration clearly foregrounds the text chat box’s central position. This has a highly salient visual status, as only one person can speak at a time.

Over a two-year time span, I interviewed about one hundred participants. My interviews were based on the observation of their uses of the two channels of communication, namely how they modulated their uses of speech and writing and their degree of awareness in reflecting on their use with guided questions (see Reference SindoniSindoni, 2011, Reference Sindoni2013). Some of the questions are reported in the infographic reproduced in Figure 1.

Figure 1 Results of interviews (2011–13).

I conducted another set of unstructured interviews in 2020–22 with a much smaller number of users (twenty-five), including, among others, Camfrog users, to acquire a broader understanding of how the communicative scenario had been changing, if at all. I repeated the same questions, but with different and fewer interviewees, as reproduced in the infographic in Figure 2, during the Covid-19 outbreak and beyond (years 2020–22).

Figure 2 Results of interviews (2020–22).

All interviewees, both from the first (2011–13) and second batch (2020–22) of interviews, claimed that the primary reason for using the text chat box was to hold parallel floors of conversations. The wording of their responses varied greatly, but this mode-switching, in both self-initiated and other-initiated (i.e., prompted by other users) forms, was clearly identified and motivated by participants. Some excerpts of my more recent interviews are reproduced here:

Excerpt 1: I use it to send my comments. Yes, these comments. When other people are speaking, yes, I want just to say something or comment with some little emoji or something, yes, I write there.

Excerpt 2: I don’t speak often. Yes, I prefer the. Erm … yes, I send IM [instant messages]. Sometimes I IM only one user, if I don’t want all to see. Yeah, I can IM from my list. But I prefer pause my video and see what’s going on there. And IM now and then.

Parallel conversation floors are also kept in the public chat and in the private instant message channel, as specified by one user in Excerpt 2. An external user would encounter problems to know how these multiple floors of conversations develop across the spoken and written channels, and this is especially true when translanguaging occurs. In these platforms, translingual practices are interspersed within these multi-layered conversations where interlocutors come and go very quickly. An example is reproduced in Figure 3.

Figure 3 A snapshot of a conversation on a multiparty video room (date: 3 December 2022).

As Figure 3 shows, a decontextualized instance of the written conversation is extremely complex to decode – not so much because of the absence of audio at this point,Footnote ⁴ but rather because several interlaced conversations were being conducted in the written channel. No cooperative conversation principle seems to be at work, at least not when reading the messages sequentially. Semiotic resources, such as colour, font, hyperlinks and platform codes (such as stars and crowns) are all employed to convey additional meanings. With reference to language, the multiple channels, in speech and in writing, are employed by users to enhance and maximize their participation and engagement with the others. When asked about their translingual practices, one user replied:

Excerpt 3: I use Thai when I want that only [another user’s nickname] understands.

When questioned about how he could be sure that no one in the chatroom spoke Thai, considering the ease of access and constant fluctuation of people in the room, he replied:

Excerpt 4: Ahah. Right. Ah. But [another user’s nickname] knows that … and she knows that I’m talking to her. Or. I don’t care, I can also IM her.

In Excerpt 4, the informant recognizes the fact that Camfrog has not been designed to have full and continuous knowledge of who is in the room at any moment in time. Nonetheless, he prefers to switch to Thai when addressing another user so that he is sure that she will appreciate the specific nuances of meaning and the forms of direct address. This excerpt provides evidence that translingual practices are not mere functional activities, but are rather interpersonal and performative. As indicated by the informant, he could have sent an instant message (IM) and have a private conversation in Thai, but he chose not to do so. In this light, translingual performances are fully meaning making in that they show intentionality, as well as semiotic (implicit) awareness. More details on these translingual practices will be further provided in Section 3.

As these examples show, multiuser video-conferencing software is likely to be perceived by external users as highly chaotic. Any stretch of conversation taken from the rooms that I have studied over the years is complex to decode, irrespective of its written, audio or shared video materials. The participants, however, seem to be at ease with this format and when asked whether they have ever experienced problems in making sense of what was happening in their video chats, they were almost unanimous in confirming their comprehension, as shown in Excerpts 5 and 6:

Excerpt 5: Yes … erm … yeah, you know, maybe it’s you, no, we all see what’s going on in here. You follow your conversation, right? And yes, you don’t pay attention to what other people are saying, you only follow what your friends say, or other people …

Excerpt 6: Yes, I always understand what’s going on. It’s easier when you get the hang of it. Not such a big deal.

These examples show that the participants identified how expert users can manage multiple conversations by simply ignoring what is not relevant to them. It is as if a sort of ‘selective attention’ is activated to silence all the conversational noise. The participants generally agreed that platform newbies may experience serious miscommunication, but that as soon as time spent on the platform increases, the sense of conversational chaos rapidly disappears. The degree of confidence rises with experience and time spent on the platform, as self-assessed by participants. All of them claimed that being able to flip between spoken and written channels was key to successful communication. As one put it:

Excerpt 7: All you need to know is when you can talk and use the audio, when you can IM, when you can DM someone. Here we know which we can use, don’t we?

However, not every multiparty video conversation needs a degree of expert knowledge as does Camfrog or similar video-based platforms. In educational contexts, for example, platforms, such as Microsoft Teams, Zoom or Google Meet, allow a more patterned and systematic organization of turn-taking across the spoken and written channels. The chat can be managed with options such as snoozing or turning off chat bubbles. These settings require a more disciplined and coordinated way of organizing VMIs. Communication is organized hierarchically, for example, when a teacher uses the video-mediated platform as a virtual classroom, thus keeping the floor for longer periods, and being the administrator of all students’ turns.

The participants taking part in professional and educational VMIs and interviewed between 2020 and 2022 claimed that if there were a ‘main’ speaker,Footnote ⁵ such as a teacher, a lecturer, a team leader or any other person in charge of conveying the main content, or to satisfy the meeting’s communicative purpose, they would use chats only to greet the others, to ask questions and to thank the ‘main’ speaker and/or the others. One informant remarked that the written chat is most likely to be used for greeting and thanking other people, and in much more explicit ways than is the case in face-to-face encounters.

In these contexts, speech is still the most foregrounded resource, the one that allows a face-to-face situation to be replicated. Writing is mostly considered as a corollary resource, used to greet the other discourse partners, ask questions without interrupting, thank and congratulate the speaker, and to signal technical issues. When asked about if and why they felt the need to greet and thank, most admitted that they tended to greet and thank verbally more than they would have done in presence because:

Excerpt 8: I want that the others feel my presence somehow. Also because I don’t want to turn my webcam on, so if I don’t say hi no-one will know I am there. I say that I’m there. And I like when other people reply or put a like on my post.

Excerpt 9: I write ‘thank you’ because I can’t give a round of applause. I know we can click on the ‘clapping hands’ icon, but it doesn’t work. I miss the cheering and involvement that we used to have in person. So I write these things in the chat. Like ‘thank you,’ ‘hi everyone’, ‘lovely to see you all’ and stuff like this.

In other cases, such as those exemplified in Excerpt 10, participants explained that the written channel was used to address technical issues and signal media-related actions, such as logging in and out. These written ‘justifications’ are forms of polite addresses to the other participants afforded by the medium that allows for a secondary floor of communication (for example, when the video or audio channel does not work) or for the avoidance of unwanted interruptions.

Excerpt 10: Excuse me, I can’t hear you, I have to log out and then get in again. Sorry.

In classroom contexts, students are likely to use the written chat as a substitute for speaking or for all other non-verbal behaviour indicating comprehension, agreement or following the arguments (such as nodding or looking at the teacher in real-life settings). When asked about their written chat use, all the students interviewed confirmed that their use of the written chat was a substitute for ‘raising hands’ in the classroom, or any other visual interactional cue asking for the floor without interrupting the teacher. I interviewed twenty-five students who used emojis to accompany their written comments. They argued that their function was mostly to express engagement and a constructive attitude. An example of this answer can be found in Excerpt 11:

Excerpt 11: Ok, I’ve never thought about it, but I guess I use them to be friendly. Yes, mm mm, to add some extra meaning, meanings, yes, to say that I was happy with it, yes, to be friendly.

They also argued that they merely replicated their use of emojis from other instant messaging apps, such as WhatsApp, and emails. However, they understood that the communication was asymmetrical because it occurred within a university context. They thus limited their emojis to only using the most neutral, such as the smiling face or the ‘thumbs up’ emoji. Coincidentally, all the students who used emojis engaged in the highest number of interactions, both in spoken and written channels. When questioned about the use of this emoji, students generally identified it as a resource to confirm attention or signal comprehension:

Excerpt 12: I used thumbs up on Teams basically to say ‘yes, I got it’.

In both multiparty interactions for leisure time and for educational and/or professional settings, participants consistently use the written channel. In the former context, the communicative functions include holding parallel floors of conversation when multiple floors are activated by many speakers in symmetrical situations (that is, when there is no ‘main’ speaker, but all have equal status for claiming and keeping the floor). Additionally, the channel may be kept open for one specific speaker or a small group of speakers. In the latter context, the written channel is recognized as a supplementary feed for extra input.

As is to be expected, the use of multimodal resources is richer and freer in symmetrical communicative events and with no professional or educational agenda involved. Formal situations, conversely, prompt a more restrained use of multimodal contents, such as using ‘neutral’ emojis, uploading files and sending educational or professional materials.

2.3.2 One-to-One VMIs, or Personal Video Interactions

Video-conferencing systems usually allow chatting with one or more people, even though, as discussed in Section 2.3.1, some platforms were specifically designed for leisure group activities, whereas others, such as Zoom, Teams and Meet, are corporate oriented and adopted by several types of institutions, including schools, colleges and universities. The nature of one-to-one meetings can be either professional (e.g., one-to-one business meetings, remote office hours between students and teachers/professors, job interviews) or personal.

The MoM and EU-MaDE4LL datasets contain comparable data but with the important caveat that students (or at least approximately half of the people involved)Footnote ⁶ were a priori aware of the communicative events they were asked to analyse, such as mode-switching. Participants involved by the students in the recorded video calls, however, only had information about the research purposes, but were not given specific details as they were not relevant to them. The Information Sheet for this task can be downloaded here www.cambridge.org/Sindoni for inspection and re-use.

The transcriptions revealed many insights into their language and media ideologies, detailed in this and following sections. The VIDEO dataset, conversely, can be defined as completely spontaneous as it only includes videos submitted by voluntary members (with whom I have no teacher–student relationship) and it is only ex post (i.e., recorded for other unspecified reasons and then submitted for analysis).

Figure 4 and Table 2, respectively, summarize reasons given by a group of participants on why they mode-switched and explanations for these reasons (years 2011–13).

Figure 4 Percentage of reasons provided by users for mode-switching.

Table 2 Reasons why users mode-switched (adapted from Reference SindoniSindoni, 2013).

Secrecy	Do not want to be heard/tell a secret, especially if someone who is not part of the conversation is in the room.
Intimacy	Communicate intimate feelings (e.g., love, friendship, affiliation, etc.) or express face-threatening feelings (embarrassment, shame, taboos, etc.).
Fun/Kidding	Add visual (e.g., through the use of emoticons) or expressive (e.g., through swear words) content.
Referential	Give precise information (e.g., addresses, telephone numbers, names, references, etc.).
Other	Not well aware about when and why they resort to written comments.

Despite variations relating to context and communicative purposes, this data may still prove useful to explain these occurrences. One strikingly absent category is what may be defined as ‘Technical’, that is, resorting to writing when some technical issue emerged. The ‘Referential’ label incorporates a large number of cases embracing the need to provide information, sometimes in the form of hyperlinks or files, as mentioned in Section 2.3.1. Instead ‘Secrecy’, ‘Intimacy’ and ‘Fun/Kidding’ require more detailed analysis. In the VIDEO dataset, ‘Secrecy’ is the second most frequent reason, as users often switch to writing when another person comes into the room or is within earshot, as emerges from the following:

Excerpt 13: I use the chat when my roommate is home and I don’t want her to hear what I’m saying.

Excerpt 14: When I have to log out as I don’t want my parents know that I’m still on FaceTime, I write down a quick message, so they don’t know.

In theory, the ‘Secrecy’ category might be thought to overlap with ‘Intimacy’, but one way to distinguish them is the presence of an intruder, or ‘unratified’ participant in Goffman’s participatory framework (Reference GoffmanGoffman, 1981). More specifically, ‘Secrecy’-activated mode-switching has been classified as such only when meeting two criteria: (1) it is only from speech to writing and never the other way around; (2) the two main (ratified) participants want to prevent other unratified participants from hearing what they are saying. When these conditions are not met, speakers typically resort to writing to build up their rapport, in which case mode-switching cannot be explained as a ‘secret’ strategy, but as an ‘intimacy-building’ resource.

The ‘Intimacy’ category always involves two people but is hard to demonstrate as the users are, obviously, looking for privacy in their conversations. As Figure 5 shows, one notable exception comes from the MoM dataset, where, as reported elsewhere (Reference SindoniSindoni, 2019), an Italian student, Irene, included a personal video call with her English boyfriend Fred, based in London. The clip submitted as an end-of-course assignment was complemented by another two tasks: her written analysis and her short transcription of a relevant segment. Interestingly, and in keeping with many other examples in the MoM and EU-MaDE4LL datasets, she inserted a column to describe code-switching, in analogy with mode-switching, that signals the alternation between Italian and English, even though this column was not part of the pre-filled transcription grid provided to the students. Her code-switching category is discussed in greater detail in Section 3, as a translanguaging practice. Of the seventy-one transcribed turns (approximately 4’:07’’), a total of eleven code-switching moves,Footnote ⁷ and seven mode-switching moves, were transcribed by Irene, making this interaction rather rich for the purposes of this analysis.

Figure 5 Part of Irene’s transcription grid.

An intimate situation is displayed both verbally and visually in this example as mode-switching reveals no explicit secret or excessively personal content. While Fred is more explicit in expressing his struggles of coping with a long-distance relationship, Irene seems to be more embarrassed and thus resorts to the written channel when Fred openly talks about him missing her. She turns to Italian and self-initiates mode-switching from speech, writing ‘povero’ (turn 46), meaning ‘poor thing’ in an affectionate way. As Fred replies saying ‘zee’, probably a way of imitating the Italian affirmative particle ‘yes’, Irene sticks to the written channel and replies with ‘I’ll be there so sooooooon’, emphasizing with the number of ‘o’s in ‘soon’ the elongated sound and suggesting her anxiety to be there with him. After a further question from Irene, this time spoken, it is Fred’s turn to self-initiate mode-switching, writing ‘zee’ and ‘zoon’, perhaps to mock the Italian voiceless alveolar sibilant /s/. Their creative and personal use of sounds is most likely a sign of intimacy that the written channel facilitates, as is recognized by Irene in her written assignment:

Excerpt 15: In turn 47, 50, 51, till turn 54, they both use the words ‘zee’ and ‘zoon’ that do not belong to any language and just represents their particular use of language.

Irene interprets mode-switching in the following terms:

Excerpt 16: Mode-switching is less frequent than code-switching and the participants only change mode at the end of the conversation. Irene mode-switches in turn 46 at her own initiative, she starts writing and code-switches at the same time, because she writes something in Italian. The other participant continues in spoken mode and eventually mode-switches in turn 50 and 51 where he writes and speaks at the same time. Irene does exactly the same in turn 53 where she uses two modes to say the same thing. This means that they are not mode-switching for a specific reason, such as bad connection, or because there are other people in the room, but simply to bring the other participant’s attention to the screen and to their conversation by making them read instead of only listening. However, by analysing the entire conversation it is clear that both participants’ preferred mode of communication is speaking and that they only write to catch the other’s attention.

(emphases mine)

Evidently, Irene recognizes the difference between ‘specific reasons’ (i.e., technical reasons) and other mode-switching functions ‘to bring the other participant’s attention to the screen’. While Irene does not explain this any further, this example shows how mode-switching can be used to increase intimacy and for building rapport rather than communicating referential content, such as questions or expressing agreement, for example with the use of emojis discussed above. Another American informant from the VIDEO dataset, Gloria, engaged in a long-distance relationship with Japanese Haru, confirmed the following:

Excerpt 17: Sometimes I felt awkward when it came to saying what I really felt. So I wrote down something nice, it feels less committing, you know what I mean? And especially when we were in the early stages of our relationship. I can’t tell you why, but we used a lot of text chat.

In the interview, Gloria recalled her struggle to manage a long-distance relationship with a person from a different cultural context, mentioning several times that writing ‘personal stuff’ in the written chat alleviated her stress. As the relationship progressed and they got more used to each other’s communicative styles, they started talking more openly using the spoken channel, but after two years, Gloria claims she still uses the written channel when she has trouble with expressing her feelings more straightforwardly.

A different form of mode-switching is displayed in the following example from the MoM dataset. Consistent with what was discussed in Section 2.3.1, the written channel is used here for reasons other than referential content and can be loosely grouped in the category of ‘Fun/Kidding’. An Italian-American student, Ray, is talking to his Canadian cousin, Amanda, about her parents’ refusal to allow her to move in with her boyfriend before marriage. The exchange is reproduced in Figure 6 from my own abridged transcription. An inline video of the overall interaction is available by clicking the link in the Video 1 caption.

Figure 6 Author’s simplified transcription of VMI between Ray and Amanda.

Video 1

Full interaction between Ray and Amanda. Video file available at www.cambridge.org/Sindoni

In this exchange, Amanda creatively employs mode-switching to add extra communicative power to her arguments. She talks about her parents’ different reactions to cohabitation before marriage; hence she self-initiates mode-switching by writing down her mother’s ‘No.’, adding ‘with a period’ (Turn 8) to signal the definitiveness of her mother’s position. As Ray giggles and enquires about Amanda’s father, she again mode-switches and writes ‘No.!’ with a period followed by an exclamation mark, making it clear that her father’s reaction was ‘the worst’ (Turn 14). In this case, punctuation allows her to complement her spoken turns (6, 8, 10, 12, 14) with extra meanings. Interestingly, VMI further contributes to the hybridization between speech and writing as stereotypical language modes (see Section 2.1). Gestures can reproduce punctuation, such as air quotes to signal that what is being discussed should not be taken at face value (Reference CirilloCirillo, 2019), or air brackets to indicate that what is being said is incidental. In this example, the conventions of writing are used within spoken exchanges, but are reintegrated into writing. In other words, what were originally speech-specific features (e.g., prosodic patterns, such as intonation) have been ‘resemiotized’ into written typographic devices (e.g., punctuation meant to replicate prosodic patterns). Reference Li and ZhuLi and Zhu (2019) have discussed the practice of what they label as tranßcripting, which is a form of writing, designing and playfully generating new scripts by means of graphical and typographical devices by Chinese users in digitally mediated social interactions. Even though this example could not be described as tranßcripting, as it does not display excessively extreme forms of script manipulation, it does nonetheless follow the same direction. VMIs allow a further stage in resemiotization, that is, another semiotic transformation, as Amanda’s ‘No.’ or ‘No.!’ chat box input is part of spoken interaction that uses writing to reinforce the spoken meaning. Writing in the chat box allows the boundaries of the two language modes to be pushed to the limit, illustrating how new and creative ways of making meanings are being developed by users.

3.1 Translanguaging in Action

The superdiverse and fluid nature of languages in context has been increasingly reflected in research in recent years (Reference BlommaertBlommaert, 2010; Reference Pennycook and OtsujiPennycook & Otsuji, 2015). Variation takes several semiotic forms in computer-mediated environments that involve heteroglossia and translanguaging practices (Reference Androutsopoulos, Thurlow and MroczekAndroutsopoulos, 2011). As argued by Reference Mora, Tian and HarmanMora and colleagues (2022b), the combination of frameworks from translanguaging and multimodality is not new. Several studies have recognized the common identification of resources and modalities to make meanings, thereby acknowledging that speakers draw on a wide repertoire of resources that go beyond one language and one single modality for communication.

A notion common to both theories is that of semiotic choice, deriving from Halliday’s model of social semiotics (Reference Halliday1978). In order to make meanings, speakers make choices and do so by designing and constructing communication in different configurations that respond to their communicative needs and social expectations. Li directly refers to the tradition of multimodal social semiotics as shaped by the work of Gunther Kress (among others). Crucially, Li describes this integration as forms of resemiotization across diverse signs (Reference Lin2018: 22, my emphases):

Translanguaging is implemented within this ongoing cycle of resemiotization (Reference IedemaIedema, 2003). The arguments put forward by Li index three aspects motivating the integration of translanguaging with multimodality which help gain knowledge of what language users do to communicate. According to Li, (1) linguistic signs are only one part of the repertoire of the semiotic resources that speakers have at their disposal. This means that language is not given priority over other resources and modalities. (2) Language users can adopt different strategic configurations of resources to make meanings: how these will be combined is a matter of choice, but is influenced by socio-historical contingencies, habits and conventions. And (3) this process is not linear and straightforward: it implies a tension between what is conventional and what is disruptive and, hence, is quite often creative. In a similar vein, Reference Lin, Wayne, Sovicheth and GarcíaLin (2015, Reference Li2018) speaks of ‘trans-semiotizing’ practices, where languages and varieties cross and are crossed by other semiotic modalities and resources. The following research questions will be addressed in the following sections:

• RQ1: How are interlaced multilinguistic repertoires represented multimodally by participants in transcription, annotation tasks and interviews?

• RQ2: How do language users make sense of their translanguaging practices in their VMIs?

3.2 Translanguaging in VMI

Translanguaging has been observed in VMIs in institutional and informal teaching and learning settings. Within the former, negotiation of meaning has been explored in tandem virtual exchange classes in Spain and Canada among Spanish-speaking and English-speaking students (Reference CanalsCanals, 2021) and in the role of translanguaging to get feedback (Reference CanalsCanals, 2022). Other studies have investigated online classes by video recording the embodied interactions using screen recording and multiple cameras as part of efforts to rethink static notions of language as an ‘a-historical entity’ and instead to navigate the ecosocial systems of learning across different timescales (Reference Chen and LinChen & Lin, 2022). Reference Ho and FengHo and Feng (2022) combine a translanguaging and multimodal model to scrutinize how an online English teaching video makes use of several semiotic resources, including translingual practices.

Studies have also examined multilingual identities beyond the classroom. For example, Reference SchreiberSchreiber (2015) discusses how a hip-hop artist mixes and mashes his Serbian and English identities in video communication in local and global communities. Adopting the perspective of digitally mediated communication and digital translanguaging space, Reference ClabbyClabby (2021) analyses how individuals build their digital personas on an instant messaging platform. Other studies map evidence of learning based on comments on YouTube videos (Reference BensonBenson, 2015), as well as the impact of online learning and peer interaction after the Covid-19 pandemic, considering how students harnessed their resources to improve intercultural communication and how they recognized the empowering value of translocal identities (Reference Ou, Gu and LeeOu, Gu & Lee, 2022). In the latter study, developing translocal identities can be achieved by giving pedagogical value to translanguaging practices.

An important focus that some research directions are taking is the recognition that learning is a transformative practice. These directions have been advocated by the reformist agendas in education discussed in Section 1. These can be pursued by means of alternative forms of learning, not necessarily confined to institutional contexts and traditional curriculum design. In this sense, Reference Tai and LiTai and Li (2021) highlight the role played by translanguaging as a pedagogical practice to motivate learners to exploit the full range of their multilinguistic and multimodal resources. To do so, they analyse the role of playful talk in an EMI (English Medium Instruction) mathematics class in Hong Kong (Reference Tai and LiTai & Li, 2021).

In the next section, some examples from the datasets used in this Element are discussed to see how individuals make sense of translingual practices as a metareflective process that mobilizes their cognitive, social and interpersonal skills, as evidenced in transcription tasks and interviews.

3.3 Translanguaging in Many-to-Many or Multiuser Interactions

As anticipated in Section 2.3.1, translanguaging practices are extremely common in platforms for instant video communication, where users come from diverse cultural and linguistic backgrounds. The ‘Main Camfrog room’, the first and only clickable one, is constantly populated with multiple language conversations, thanks to its permanently activated channels of communication, that is, the speaking floor and the chat box for written comments. These produce multiple overlapping conversations, rich in multimodal elements, such as distinct colours, fonts, emojis, GIFs and other multimodal ‘gifts’ that users can send to each other, increasing their status within the platform while at the same time receiving help themselves.

In the short extract reproduced in Figure 7, the languages used (in the order in which they appear) are Thai, English, Tagalog, Uzbek and Gujarati. The conversation, however, is hard to comprehend, as turns are clearly not sequential. A host of multiple micro-conversations co-occur, both in the spoken channel (Turkish and English when the screenshot in Figure 7 was taken) and in the written channel. A guide to follow the flow of conversation is the name of the speaker, who addresses one or more people using their mutual language of understanding.

Figure 7 An example of translanguaging practices on multiuser video platforms.

Figure 8 illustrates a similar translingual situation and shows the alternation of Tagalog, Turkish and English. This second case illustrates how users expand the nature and extent of their dialogues by switching from one language to another and mode-switching from the spoken to the written channel. English generally appears to be preferred as the most ‘open’ and most intelligible language, whereas translingual practices are most frequently observed in the switch from one’s ‘first’ language to English. The network of addressees, in this scenario, may be expanded or restricted according to language choice, as exemplified in Excerpts 18 and 19:

Figure 9 quantifies the different types of reasons given for the use of translanguaging practices in the interviews, while Table 3 briefly characterizes the explanations given.

Figure 8 An example of translanguaging practices on multiuser video platforms.

Figure 9 Percentage of reasons for translanguaging provided by users (2020–22).

Even though some major differences between Table 2 (on reasons for mode-switching) and Table 3 are to be noted, some striking similarities emerge. Percentages of reasons in replies cannot be compared, as, between the first and second interviewee cohorts, the figures involved vary greatly, quite apart from their sociodemographic background, which could not always be collected or organized systematically.

The reasons for mode-switching (Table 2) show some correspondence with those provided by multiparty video platform users (Table 3). The most salient is VMI participants’ willingness to fully exploit the playfulness and humorous potentialities of variation. The ‘Fun/Kidding’ category in the mode-switching domain mainly regarded the use of the written chat to increase the expressive power of their messages. This means that the written chat was used to add humorous visual or other multimodal content, or to add jokes, commenting or overlapping with the main spoken floor of interaction. This use may be compared to what has been defined by Li and Zhu as Tranßcripting (Reference Li and ZhuLi & Zhu, 2019). The same category in the translanguaging domain plays a similar function: changing language or variety within the same communicative event is perceived to be ‘fun’ and therefore strategic when building a positively engaging digital persona. The ‘Secrecy’ category is likewise partially overlapping, as mode-switchers used the video affordances of writing content that had to be kept hidden from overhearers or ‘unratified participants’, to draw on Goffman’s participatory framework (Reference GoffmanGoffman, 1981). In an equivalent manner, individuals who mixed languages during their VMIs claimed that this behaviour may be due to the need to restrict the wider number of potential addressees to a single individual or a small group in the multiuser room. Seen in this light, using a different language or variety marks in-group from out-group conversational dynamics. The ‘Personal’ category refers to one-to-one communication (most likely by means of direct messaging) and differs from the ‘Intimacy’ reason that mode-switchers resorted to when intense or intimate conversational moments arose. However, both the ‘Personal’ and the ‘Intimacy’ explanations indicate the intention, on the part of users, to build rapport within the Available Designs and within the modalities afforded by the medium.

The ‘Referential’ reason differs slightly in the two discourse practices, but is nonetheless comparable. Mode-switchers mostly claim that they use the written chat to give the discourse partners specific information, such as website addresses, names or to upload a file. Camfrog users explain that one of the reasons for language mixing is when local information needs to be provided or if such information is available only in one language that is different from the others previously used. Within the ‘Secrecy’ domain, both spoken/written variation and translanguaging practices serve the purposes of alignment and/or disalignment in conversations. Users imaginatively combine media affordances and linguistic repertoires to satisfy their communicative needs, build rapport with the other users and discipline conversational chaos.

Even though the reasons and explanations for these forms of alternation vary, such as speech/writing and alternation/mixing of several languages and varieties, some similarities can be observed in the users’ perceptions.

In the next section, some examples of one-to-one conversations will be presented to further examine other occurrences. These will be complemented by adding a sample of users’ own perceptions and sense making in tune with multimodal and translanguaging theories of semiosis that highlight the agency and interest of the sign makers.

3.4 Translanguaging in One-to-One Interactions

The datasets used in this section have already been presented in Section 2.3.2. Hence all case studies introduced here draw on them for the sake of consistency. Returning to one example discussed in Section 2.3.2 and partially reproduced in Figure 5, Irene understood her and her boyfriend’s translingual practices as ‘code-switching’. It needs to be noted at this stage that not all students participating in the project (and who had chosen video chat as the preferred digital text for analysis as their final assignment) had had formal instruction in sociolinguistics. They had diverse backgrounds and only a few small groups had studied general linguistics. None of them were familiar with theories of translanguaging. Furthermore, the transcription grid with the relevant categories of analysis did not feature either ‘code-switching’ or ‘translanguaging practices’.Footnote ⁸ However, the students were invited to edit the transcription grid if they felt that some categories needed to be added or if others were not relevant in their video clip and could be merged or deleted. As a result, students from both the MoM and the EU-MaDE4LL projects had a certain degree of freedom as regards adapting the transcription table to their video materials and indeed proceeded to do so.

Since all video chats were in English, but with at least one participant speaking English as an additional language, some 70 per cent of all the video data included in MoM and EU-MaDE4LL datasets presents examples of translanguaging practices. A large number of students decided to add the category labelled as ‘code-switching’ in the transcription grid to represent the occurrence in question. This modification of the original transcription grid reveals students’ language ideologies (see Reference Li and ZhuLi & Zhu, 2013), as they showed that they understood translingual practices as meaning-making systems worthy of separate discussion. When reading their written analyses, the reason they gave to explain these practices emerges as being mostly attributed to facilitating communication, as illustrated in the following example:

In this example, Tiziano, an Italian postgraduate political sciences student, motivates his use of code-switching as a way ‘to explain the meaning of a word’. Code-switching is defined as a ‘technique’. It is unclear why he defines ‘Duomo’ as one ‘particular Italian word’; it is probably because he considers ‘Duomo’ as untranslatable and therefore such a term cannot be explained in terms of code-switching or as a form of translanguaging. Other students with a background in linguistics show a much more refined ability in unpacking the interactional functions of translanguaging, even when the latter is not explicitly identified.

In this scenario described between S (the secondary participant) and T (the author of the excerpt, an English ‘native’ speaker), use of mode-switching once again runs parallel to the combined use of two languages (in this case, Italian and English). The written channel is used for informal teaching purposes and T, the English speaker, implicitly recognizes that changing languages is part of the communicative repertoire that usefully facilitates intercultural communication. Even when using this basic and improvised teaching strategy, T mobilizes all the resources at her disposal, such as ‘superior language knowledge’ and media affordances to facilitate communication. Translanguaging and mode-switching are equally identified as viable solutions that improve intercultural communication, as made explicit in the following (my emphases):

As Excerpt 22 shows, the use of the written and oral channel is interlaced with translanguaging practices and is not directly addressed in the discussion. The video interaction displays that mode-switching is finely interlaced with translingual practices. The learner’s comments are also indicative of her perception of symbolic power coming from her position of ‘native speaker’ (see Reference Zhu and KramschZhu & Kramsch, 2016). Her commitment to facilitating interaction resonates with the argument made by Zhu: ‘conversation involvement refers to the psychological solidarity and connectedness individuals show to each other through active participation in interaction’ (Reference ZhuZhu, 2019: 106). Figure 10 shows a portion of the transcription grid produced by Veronica in conversation with her German friend Barbara.

Figure 10 Transcription grid as modified by Veronica.

In this example, the column labelled ‘Code-switching’ is inserted by Veronica to show the change from English to German as taking place in her interaction with her German friend. However, no mention of this event is made in her written assignment. The positioning of the column alongside the column on ‘Mode-switching’ reveals how the two forms of language alternation can work in partnership in VMIs. The conversation is about Christmas presents. Veronica is the first to start using German in this conversation section, by saying ‘Weihnachtsmärkte!’ (a turn not reproduced in Figure 10). Then Barbara repeats ‘Weihnachtsmärkte! Ya, jawohl!’, thus confirming in her language the typical folkloristic venue evoked by her friend. She confirms that these places are similar everywhere as they imitate the German Christmas markets and then writes down the city in question, that is, Leipzig. Veronica initially confirms the name, but then she corrects her friend and tells her the exact place she had in mind in the first place, that is, Stuttgart. What is to be noted here is not so much the change in language (as naming cities in their original language is not a noteworthy translanguaging practice in itself), but how Veronica makes sense of the overall exchange. Unlike Tiziano, Veronica considers naming German cities as ‘code-switching’ and, as such, feels the need to update the transcription table. This means that she values these examples as language changes, from English to German, even though she does not elaborate on these occurrences in her written assignment and only says:

In fact, the two occurrences of mode-switching are paired with two occurrences of the use of German. Whether interpreted as language change or as mere reproduction of the name of the two German cities, this is yet another example of how these practices are intermingled, and not by chance, with the creative use of the spoken and written channels in video chats.

Another example of translanguaging can be found in the case exemplified in Figure 11.

Figure 11 Transcription of interaction between Mariangela and Tonino (part 1).

The whole interaction can be viewed by clicking the link to inline Video 2 in the footnote.

This transcription grid comes from the EU-MaDE4LL dataset. The interaction is between a student participating in the project and her Canadian-Italian cousin Tonino. As a Canadian-Italian, Tonino quite often switches from Canadian English to Italian, as he knows Mariangela will understand him effortlessly. More importantly, these translanguaging strategies increase solidarity between participants. The conversation is about the different degrees of confidence in face-to-face versus video-recorded interviews. Both discourse partners recognize that being recorded adds a sense of awkward self-awareness that impinges on overall performance. As shown in Figure 11, Mariangela identifies what she labels as ‘code-switching’ as a significant resource in video interaction and, as such, adds a column to transcribe all its occurrences. Later on in the interaction, Tonino laces his Italian with the use of a typical idiomatic expression, that is, ‘al volo’, which means ‘without prior notice, quickly, without preparation’. This short exchange is reproduced in Figure 12.

Figure 12 Transcription of interaction between Mariangela and Tonino (part 2).

In this second part of the transcription in Figure 12, Mariangela reports on the comment made by Tonino, who was forced to give an interview ‘al volo’, therefore switching to Italian and taking advantage of the expressive power of this Italian idiom. ‘Al volo’ captures the improvised nature of the social request imposed on Tonino and his unwillingness to comply with it. This language choice is taken up in Mariangela’s written assignments. She explains this choice in the following terms:

In this example, language choice functions to express the exact feeling that Tonino wishes to convey, and Mariangela captures his translanguaging practice to increase his rapport with her, as she shares the same language knowledge. An analogous situation can be observed in another VMI from the EU-MaDE4LL dataset. The Italian student Francesca is interacting with her British friend Jake. Drawing on her basic knowledge of linguistics, Francesca equates all translanguaging practices as ‘code-switching’ and makes sense of these practices as follows:

In her attempt to be scientific in her assignment, she unproblematically defines any form of language mixing as ‘code-switching’, drawing on classic literature in the field of sociolinguistics and refers to variation. She then makes it clear that the use of more than one language has to do with how speakers draw on their linguistic repertoires to achieve communicative goals, which could be building rapport or mutual intelligibility. These functions of language change are explored in Excerpt 26:

In the first reported example, ‘ciao’ is the almost universally acknowledged Italian informal greeting, which Jake uses to signal ‘sympathy’ in Francesca’s words, by recalling his past experience of being an Erasmus student in Italy. He then uses the word ‘teatro’ when attempting to recall another past experience, namely visiting the ancient Greek theatre in Taormina with her. Figure 13 shows how the exchange is reproduced in the transcription grid filled in by Francesca.

Figure 13 Francesca’s transcription grid.

Jake here brings into play his linguistic repertoire (and reminisces on his basic knowledge of the Italian language) to recall past memories that he shares with Francesca. The latter, as other students had done in both the MoM and EU-MaDE4LL datasets, makes sense of the interaction by adding another column in the transcription grid, labelled as ‘code-switching’, perhaps in analogy with the ‘mode-switching’ column. In this case, even though all the communication is brought forward in English, fragments in Italian contribute to building common ground for intercultural communication.

These examples of translanguaging arise and are made sense of outside the classroom context. This means that they are not pedagogical or designed by a teacher; they were neither used as specific class strategies by the researchers participating in the project, nor explicitly taught in the syllabus. They were defined and described by students as ‘code-switching’, in keeping with a classic and popular definition readily available to university students with no background in sociolinguistics, and even less knowledge of translanguaging theories. However, these examples have been presented and discussed in this section to show how translanguaging practices emerge spontaneously when people from distinct cultural and linguistic contexts interact and are also recognized as meaning-making ‘resources’, as explicitly defined by the student Francesca.

This section has made use of the same datasets but reflected on these practices a posteriori, as they spontaneously surfaced in VMIs and were not explicitly taught in class. This a posteriori analysis was not meant to explore language choices, but to primarily assess if, how and to what extent learners recognized and interpreted translingual practices. Occurrences and frequencies, as well as the explicit mentions and explanations given by students, provide insights into how these are perceived and unpacked in terms of language ideologies, especially because students had received no overt teaching on these issues.

4.1 The Lack of Mutual Gaze in VMI

In keeping with the other sections of this Element, participants’ understanding of how gaze is exchanged and distributed in VMI is foregrounded. Individuals, as sign makers, make sense of gaze patterns and comment on the various implications that media affordances have on their interactions and on the meanings exchanged.

One example from the MoM dataset helps gain some insight into learners’ meaning making vis-à-vis gaze behaviour. Figure 14 shows a screenshot of a Sri Lankan travel agent who lived in London at the time when the video call was recorded.

Figure 14 Linda looking at the screen.

Linda is captured in this and other screenshots in very close-ups and her gaze behaviour is described in the following terms by her discourse partner Zoara (my emphases):

Zoara demonstrates a remarkable level of awareness about gaze behaviour and discusses several aspects of it at some length. First, she mentions that Linda is using a smartphone for her video call, thus significantly reducing the size of her image. Zoara realizes that this medium-afforded reduction in visibility may well influence her interpretation of gaze. She points out that gaze directionality can sometimes be hard to detect in VMI. She then describes how Linda manages gaze. In her interpretation, when Linda speaks, she tends to avoid direct gaze, whereas she does the opposite when specific moments arise in the interaction, such as when she wants to ‘underline a word’ (i.e., emphasis), has ‘decided her speaking turn is over’ (i.e., signalling turn completion) or is ‘asking a question’ (i.e., requesting the other to take the floor). All these gaze patterns can be interpreted as accompanying speech in keeping with CA approaches that conceptualize gaze as an important visual cue (or ‘visual method’, as described by Reference Ford, Thompson, Ochs, Schegloff and ThompsonFord & Thompson, 1996). However, Zoara goes beyond this and continues her explanation as follows (my emphases):

What she expresses in Excerpt 28 shows that Zoara comprehends media affordances regarding the problems in reciprocating gaze. She raises the question of intentionality, as she takes for granted that looking at the webcam serves the purpose of ‘giving the impression’ of reciprocating the gaze of the other. Looking straight at the webcam entails a degree of unnatural gaze patterning in that this visual behaviour forces the participant to divert their eyes from centre-screen, where the items discussed in the interaction are displayed. A high degree of awareness is needed to change this behaviour so that the other discourse partner ‘feels’ a more natural interactional situation. In other words, conveying a sense of natural gaze use implies forcing oneself to look at the webcam instead of looking at the other discourse partner’s projected image (Reference SindoniSindoni, 2013). Quite strikingly, Zoara interprets this state of affairs as a ‘psychological’ personality trait rather than attributing such gaze behaviour to inadequate media affordances (i.e., webcam–eye disalignment). Her ‘psychological’ interpretation is, however, moderated by other contextual factors that in her view may well affect the screen-mediated conversation, as reported in Excerpt 29:

Here, Zoara hedges her ‘psychological’ interpretation of Linda’s gaze management by listing some contextual factors that may impact on communication, such as different physical locations where they are placed and the considerable time lag involved, as Zoara is temporarily living in Australia. She is able to explore all these various components by mentioning features that are not exclusively linked to turn-taking management, but which surpass the verbal. These suggest that the ‘interpretation of meaning-making resources’, as she puts it, is much more complex and involves resources that transcend language-only data.

The VIDEO dataset reveals that the difficulties of conveying a sense of natural gaze behaviour was acutely felt by some other participants. An American teacher of English in one high school based in Brazil, Thomas, submitted clips of his online classes after the Covid-19 pandemic for my research project. Screen captures cannot be reproduced here to protect his students’ anonymity, but we viewed the clips together and discussed at length the problems he had regarding the offline/online transition. In one of the post-recording interviews, which took place in May 2020, I asked him what the main challenges he was facing were. Excerpt 30 reproduces part of his answer:Footnote ⁹

Thomas was intensely self-conscious of his gaze behaviour, because his face-to-face teaching methods relied heavily on eye contact to check his students’ understanding. He had been teaching in Brazil for more than twenty years, so his cultural knowledge of the context, his students’ class behaviour and their expectations were high. However, the pandemic turned all his past experiences upside down. He tried to reproduce his usual gaze behaviour in video environments and reported that he had realized from the very beginning that he needed to look at the webcam to convey the feeling of ‘being looked at’ to his students, as a technique to maintain their attention and exercise a form of control. However, he recognized that it had been difficult to keep this up consistently and that such contrived gaze organization (i.e., forcing himself to look regularly into the webcam) led to inconsistencies and ‘erratic’ eye movements.

The VIDEO dataset includes several other participants who reported similar circumstances. When the video conversation took place in personal settings and for mundane activities, efforts that failed to achieve perfect mutual gaze were not perceived as a problem and ‘adaptation to the medium came out naturally’ (as one VIDEO informant put it). Conversely, lack of eye contact and/or eye–webcam misalignment surfaced as one of the major causes of breakdown in communication when checks were made to establish whether communication had been successful.

Another example from the EU-MaDE4LL dataset points to different facets of this visual event being interlaced with mode-switching. A Danish student involved in the project, Freja, set up an experiment with her long-time friend Mariana, a Portuguese girl living and studying zoology at the University of Nottingham in the UK. While Mariana knew in advance that she would be recorded for some project by Freja, she was not aware of the details of the research project. The Facebook video call was therefore quite natural (at least for Mariana) and presented an intimate conversation between the two close friends. However, Freja was quite surprised that Mariana had agreed to being recorded in a restaurant, with considerable background noise and distraction. Freja had decided to see her friend’s reaction to a (fake) revelation over the video chat to see if and how Mariana would react, given the social constraints of being in a public space. In normal and relaxed conversation, Freja had recognized Mariana’s attempts at eye contact as being very low, somehow hinting at little involvement and null emotional investment in the conversation, but things changed abruptly as Freja announced that she was going to make a shocking revelation:Footnote ¹⁰

To provide more visual context, the stills mentioned by Freja are reported in Figures 15 (Still 8) and 16 (Still 13).

Figure 15 Still 8 from Freja’s transcription.

As documented in Excerpt 31, Freja was building up suspense, and so Mariana adjusted herself to receiving the news, both in body movements, moving closer to the camera, and by showing with a highly focused gaze direction that Freja had all her attention. However, instead of revealing the content of her announcement with words, Freja decided to send Mariana a picture via the Facebook chat. The mode-switching from speech to visual content initiated by Freja was followed up by ‘channel-switching’ (as effectively described by Freja herself). This refers to Mariana’s change of device, switching from her laptop, used for the conversation, to her mobile phone, to see the image revealing this surprising content. This channel change, from laptop to mobile screen, most likely due to the pressing need to get immediate access to the visual content sent by file, was defined by Freja as ‘channel-switching’ in the written assignment, once again detailing the participants’ critical and creative skills in construing the meaning making of VMIs.

While Figure 15 reproduces Mariana’s slight change in attitude, less formal and slightly closer to the screen, anticipating Freja’s revelation, Figure 16 neatly shows Mariana’s genuine reaction of great surprise that needs no verbal content to be understood, either by Freja herself during the interaction, or by future viewers of the screenshot. Mariana’s bewilderment is conveyed exclusively by means of gaze, which, regardless of the screen constraints discussed in this and the previous section, visibly reproduces what could have happened in a face-to-face context.

Figure 16 Still 13 from Freja’s transcription showing Mariana’s genuine surprise.

This example is telling as regards both Mariana’s reaction conveyed through gaze alone and despite the media-dependent problems of gaze management in VMI, and Freja’s overall meaning making. In her analysis, Freja in fact commented on Mariana’s extremely restrained reaction when compared with the shocking nature of the revelation. She had expected a more dramatic reaction, such as raising her voice or jumping up from the table. This is particularly true as Mariana came ‘from a very temperamental and exotic culture, Portugal’, but ‘it came across that she tried to adapt to the common rules of British society in a public place’. Interestingly, Freja manifested her cultural assumptions and stereotypes about Mariana by taking for granted that because she comes from an ‘exotic’ culture her reaction would be far more dramatic. In the case study here illustrated, Freja makes cultural assumptions based on Mariana’s country of origin, Portugal. Considering it to be a ‘very temperamental and exotic culture’, Freja expected a different reaction. Mariana’s reaction, interestingly, is not projected exclusively and strictly in linguistic terms, but is framed in broader terms, including observations of her socially inscribed movements, body positioning and eye behaviour. In this sense, Freja’s written analysis helps unearth students’ expectations, cultural biases and hypotheses that surpass the mere organization of gaze patterns in VMIs. This form of understanding can be gained by means of a combined multimodal and translanguaging approach that aims to unveil semiotic, plurilinguistic and cultural ideologies. More specifically, drawing attention to all multimodal non-verbal resources that contribute to meaning making can help students, or any other participant interested in improving communication skills in video-mediated environments, to reflect on the many facets that determine whether, and to what extent, communication is successful. Even though learning and teaching are largely based on language-based contents, these case studies show that multimodal communication, if properly scrutinized, can produce insights on how meanings are assumed, produced, circulated and understood in different cultural contexts. Cultural assumptions, in this specific case, seem to rest more on generalized ideas of what one essentialized culture may be in the eyes of the observer (in this case, Freja’s) more than on specific individual personality traits and behaviour (in this case, Mariana’s).

These observations should guide further reflections on how meaning making transcends not only linguistic and cultural boundaries, but even individual and personal ways of understanding and perceiving other people’s knowledge making.

4.2 Self-Looking and Gaze Sequences

Since the very beginnings of my research into VMI, the media affordance of self-looking has emerged as making video interaction quite different from face-to-face situations (Reference SindoniSindoni, 2011, Reference Sindoni2013, etc.). I defined this media affordance as if we were constantly looking at a mirror placed within close sight while having a face-to-face conversation. This self-conscious feeling would, I predicted, be greater than normal and would cause several self-monitoring adjustments (Reference SindoniSindoni, 2013).

The infographics reproduced in Figures 17 and 18 partially report results from interviews I conducted with separate batches of participants in 2011–13 and 2020–22, respectively. They draw on data already reported in Figures 2 and 3, but focus only on the practice of self-looking. Even though, as mentioned in Section 2, the results are not comparable for several reasons, mostly because the participants were different and because the number of interviewees varied greatly, what remains constant is the high percentage of participants claiming that they look at themselves during a video chat, that is, 74 per cent in the first batch and 92 per cent in the second batch (but with 75 per cent fewer participants).

Figure 17 Self-looking by interviews (years 2011–13).

Figure 18 Self-looking by interviews (years 2020–22).

In the more detailed interviews conducted in 2020–22, all participants claimed that their answer referred to informal, personal, mundane situations in one-to-one contexts that always called for leaving the webcam on. ‘No-one would dream of switching the webcam off’ in one-to-one contexts, as one informant explained. Having the webcam on implies the extremely likely action of self-looking during video calls.

On the contrary, gaze directionality in multiparty video interactions is harder to discern; even though we may infer a certain amount of self-looking, this is unlikely to be visually perceived by others in interactions. Furthermore, building on the consideration reported above on the unlikeliness of turning the cam off in one-to-one scenarios, it can be assumed that this improbability tends to decrease in multiparty environments, as the social pressure to be ‘visible’ is weaker. In post Covid-19 educational scenarios, the question about how much the request to turn on the webcam during classes imposed on learners has been discussed at length elsewhere (Sindoni, unpublished data). What can be deduced from digital field observations is that in multiparty scenarios, gaze directionality – and related questions of self-looking – remain unclear even when all cameras are turned on. This can be seen in Figure 19 (with users cartoonized using the Picsart Editor app to protect anonymity) following the vectoriality indicated with arrows.

Figure 19 Complex gaze directionality in multiparty video platforms signalled by arrows.

In some of the interviews reported above for the second batch, the participants clearly distinguished their gaze behaviour in terms of self-looking by reflecting on the diversity of one-to-one versus multiparty interactions, specifying that when they had the chance to switch off the webcam, the possibility of self-looking automatically dropped out, sometimes to their great relief.

Some deeper insights into learners’ reflections on self-looking can be found in the MoM and EU-MaDE4LL datasets. Yet another example of how learners made sense of this medium-specific affordance comes from Zoara, who explains and interprets the meaning making of this feature, even though she defines this behaviour in ‘psychological terms’, by discussing the possible reasons for self-looking:

Zoara interprets self-looking as a signal of narcissism, making it clear that it means appearing at one’s own best. Self-monitoring, however, can influence the naturalness and spontaneous quality of one-to-one conversations, as Zoara points out, connecting this last point to the general sense of detachment produced by the lack, or limited amount, of meaningful eye contact. Another student from the EU-MaDE4LL dataset describes this event in the following terms (my emphasis):

While the student described how participants in the conversation managed gaze directionality by gazing centrally or downward the screen, she also identified some sequences defined as ‘self-checking gazes’. To illustrate these sequences visually, the student inserted the screenshot with arrows reproduced, unedited, in Figure 20.

Figure 20 Example of self-looking as described by one student (i.e., self-directed gaze).

Another example from the MoM dataset is meant to illustrate how self-looking is finely ingrained in gaze sequences. Figure 21 shows a gaze sequence portraying Barbara, a translator from Luxemburg.

Figure 21 Gaze sequence.

Section A on the left-hand side portrays the beginning of the conversation. Barbara gazes at herself, looking right downwards at her close-up frame. In Section B, Barbara matches her gaze direction with what she is saying: ‘It’s a wonderful day here!’, thus directing her gaze out of the window. In Section C of Figure 21, she looks downwards, as she is silent and listening to (and therefore looking at) her interlocutor. This gaze sequence further illustrates how self-looking and gaze vectoriality and directions are finely interlaced and can be understood in context; only in some cases were the participants explicity asked to recall what was going on at that specific moment in the interaction.

In what follows, movement organization and distribution space will be investigated, as well as the reframing, or resemiotization, of social distance in VMI.

5.1 Kinesic Patterns, or How Body Movements Shape VMIs

The study of kinesics, or of how movements contribute to shaping communication, is linked to language-based theories of communication. Kinesics was named as such by Reference BirdwhistellBirdwhistell (1952) and developed out of the general research area of non-verbal communication. Kendon likewise mapped the domain by identifying movements in all their visible and communicative functions (Reference KendonKendon, 1990b, Reference Kendon1997, Reference Kendon2004).

The ‘non-verbal communication’ label itself reveals questions of language dependencies and forms of hierarchies, as the negative prefix ‘non’ seems to implicitly point to a ‘secondary’ nature of gestures and to derivative, language-dependent ways of organizing space in communication. Halliday, for example, recognizes that gestures play a regulatory function in cooperation with language: ‘to control the behaviour of others, to manipulate the persons in the environment’ (Reference HallidayHalliday, 1973: 31), while the psychologist Reference Goldin-Meadow, Messing and CampbellGoldin-Meadow (1999: 428, my emphasis) argues that ‘gesture provides speakers with another representational format in addition to speech, one that can reduce cognitive effort and serve as a tool for thinking. Gesture also provides listeners with a second representational format, one that allows access to the speaker’s unspoken thoughts and thus enriches communication’. The underlying assumption here is that, yet again, movements are considered derivative and language dependent, as a corollary to the main meaning making provided by language.

Social sciences have developed several models to study and categorize gestures, for example, within domains such as psychology (Reference Goldin-Meadow, Messing and CampbellGoldin-Meadow, 1999; Reference McNeillMcNeill, 2005), anthropology (Reference Haviland and DurantiHaviland, 2004; Reference KendonKendon, 2004) and language-based disciplines, such as CA (Reference Goodwin and PsathasGoodwin, 1979; Reference Schegloff, Maxwell Atkinson and HeritageSchegloff, 1984; Reference Streeck, Goodwin and LeBaronStreeck, Goodwin & LeBaron, 2011).

Multimodal studies consider movements in their contribution to meaning making in partnership with or independently from language, according to the situation and people involved. For example, Reference Kress, Jewitt, Ogborn and TsatsarelisKress and colleagues (2001) have given a detailed account of the fundamental role of gestures in classroom contexts. More recently, multimodal research has significantly advanced knowledge of how movements cooperate with other multimodal resources, such as facial expressions, gaze, posture, social distance, head movements and manipulations of physical objects and surroundings in embodied interactions (Reference MondadaMondada, 2016; Reference Mondada, Oloff, Stam and IshinoMondada & Oloff, 2011). Reference LimLim (2021) makes a strong case for the pedagogical relevance of the use of gestures in classroom practice and develops a systemic functional approach to increase teachers’ awareness of the pedagogical effectiveness of gestures in combination with language and other resources. In VMIs, Reference NorrisNorris (2004) mapped the study of gestures and movements in spatially constrained environments mediated by the screen, by categorizing movements along the continuum from full intentionality to unintentionality. As the definition of continuum suggests, it is not always possible to distinguish between intentional and unintentional movements, especially in in-between situations. If pointing is certainly intentional and posturally preferred positions are surely automatically triggered behaviour, other halfway body movements are not so clear-cut. For example, it would be hard to find agreement about whether the hand position of a MoM participant reproduced in Figure 22 is intentional, half-intentional or unintentional. However, in this specific context, we focus on how meanings are made and received.

Figure 22 Hand position: intentional, half-intentional or unintentional?

Hand gestures, body movements and body positioning have been variously classified. Some distinguish between communicative functions of hand gestures, such as deictic (e.g., pointing and context dependent), motor (e.g., batons and beats, facilitating rhythm and pace), symbolic (e.g., emblems, conventionalized), iconic (e.g., visual representations of a referential meaning) and metaphoric (e.g., visual representation of an abstraction, see Reference McNeillMcNeill, 1992).

As key components in human communication, movements, gestures, positioning and facial expressions all contribute to meaning making in interactions. However, VMIs pose a further challenge to their interpretation. As argued in more detail elsewhere (Reference SindoniSindoni, 2013, Reference Sindoni2014), the medium-specific affordances of video platforms and apps in desktop or mobile devices only allow a small portion of the participants’ bodies to be viewed. This is particularly true in the case of mobile communication, when only head movements and possibly occasional postural shifts can be captured by the other discourse partners on screen (Reference LicoppeLicoppe, 2009). Assuming that the webcam is on and other discourse partners can be seen, this body ‘fragmentation’, or partial reproduction of the whole bodily presence, needs to be taken into account when making meanings and interpreting other people’s meaning making in interaction. Teachers, educators and practitioners in the field of language policies, syllabus planning and remote learning need to be aware of the diverse features facilitating or hindering smooth VMC.

In multiparty interactions from desktop devices, several single hand gestures and sequences can be found in Figure 23, which compiles twelve screen captures of users in different situations.

Figure 23 Hand gestures by multiparty video call participants.

Some hand gestures in Figure 23 are iconic in that they are used to represent a concrete object visually and referentially (Stills 1, 2 and 9), size (Stills in Sequence 3 and 4) and quantities (Still 5 and Stills in Sequence 10–11). Other hand gestures are symbolic, such as the emblems in Still 2 (waving) and Still 6, which playfully mean ‘I’m watching you’. Still 9 represents the deictic gesture of pointing, as does Still 12, where the user Strawberry is pointing at the screen with a sheet of paper. These screenshots are illustrative of several kinds of hand gestures that can reproduce the face-to-face context and even create a sense of proximity to the other speakers. Some gestures can be understood universally, such as waving or pointing, and their use can help build empathy in environments that are, by definition, fragmented and disconnected from each other.

One of the EU-MaDE4LL students highlighted that intercultural communication can be encouraged and facilitated by the use of gestures, revealing her language ideologies as an English ‘native speaker’ (see also Reference Sweeney and ZhuSweeney & Zhu, 2010):

In Excerpt 34, the learner feels the need to specify that smiling and nodding was performed by the English native speaker, whereas the non-native speaker (defined by status negation) uses ‘specific hand movements’. These are further explored in the assignment, partially reproduced in Figure 24.

Figure 24 Kinesic action described by one EU-MaDE4LL learner.

In the assignment, the learner explains that smiling and nodding are used as accommodation strategies (Reference Sweeney and ZhuSweeney & Zhu, 2010) to confirm comprehension and prompt the ‘non-native speaker’ to continue conversing. On the contrary, some hand gestures made by the non-native speaker are employed with the following functions:

The learner conceptualizes two different ‘sets’ of facial expressions and gestures, assigned to native and non-native speakers on the grounds of diverse (symbolic power) positions in asymmetrical conversations (Reference Zhu and KramschZhu & Kramsch, 2016). Even though the native speaker is genuinely keen on engaging the non-native speaker in a smooth conversation by adopting several strategies to save the non-native learner’s face, at the same time, she indirectly points to her own implicit (native) linguistic superiority.

Another instance from the same dataset provides an illustration of reversed roles: the non-native speaker, Sara, teaches some Italian words to the English native speaker. Motor gestures, batons and beats, are used to provide coherence by means of a finely interlaced combination of language, prosody, and rhythmic hand and head movements, as described by Matilda below:

In other cases, gestures serve to contextualize the environment and have a ‘showing’ function, what I have defined elsewhere as displaying hand gestures (Reference Sindoni, Garcés-Conejos Blitvich and BouSindoni, 2018). Displaying hand gestures lead, partially or entirely, to substituting the projected image of the participant with an object or the visual availability of the immediate surroundings. Licoppe and Morel aptly defined the default configuration of VMIs as ‘talking heads’, where participants ‘show as much of their face as possible’ (Reference Licoppe, Morel, Broth, Laurier and MondadaLicoppe & Morel, 2014: 4). However, in the situation described in Excerpt 36, the consequences of movements and gestures are a momentarily entire or partial disappearance of the person described as the ‘show-er’ (contrasted with the ‘viewer’ in Reference Licoppe, Morel, Broth, Laurier and MondadaLicoppe & Morel, 2014). Some examples are shown in Figure 25.

Figure 25 Sequences of ‘show-ers’ in displaying mode.

The stills in Sequence 1 and 2 reproduce a user who is moving an empty bottle of wine close to the screen. This is a displaying gesture. Turning the bottle upside down is a gesture that is used to prove visually what he is saying verbally, that is, that the bottle is empty. In the stills in Sequence 3 and 4, another participant is showing some banknotes he has just earned. Yet again, showing corresponds to adjusting distance between oneself and the object in question vis-à-vis the screen. In turn, these movements equate with providing evidence of what is being said. The noteworthy medium-specific affordance here consists in the entire (Still 2) or partial (Stills 1, 3 and 4), yet momentary, disappearance of the shower. In a face-to-face context, hand movements do not change or obscure the bodily presence of all the involved individuals, however extended these hand movements may be. This paradoxical VMI affordance has been noticed by learners, as can be seen in the following excerpt:

These comments reveal the learner’s awareness about the ‘dual nature’ of this medium-specific affordance. If, on the one hand, the partial (and screen-based) reflection of the other discourse partner helps diminish distraction (for example, when one of the two is typing, thereby detaching him/herself from the other’s direct eye focus), on the other hand, this displacement results in a contrived sense of fragmentation, ‘an obstructed visibility’, as vividly put by the student. Another learner from the EU-MaDE4LL project worded the in/visibility question in the following terms:

In this passage, the learner quite nicely captures the possibilities and limits (or to use Kress’ (Reference Kress2005) words, ‘gains and losses’) of gestures and movements in screen-constrained environments. Despite the unattainability of full access to the other discourse partner’s bodily presence, adaptation to, and metapragmatic awareness of the medium, contribute to mutual intelligibility.

5.2 Proxemics, or Staged Social Distance in VMI

The cultural anthropologist Edward T. Hall coined the term ‘proxemics’, meaning ‘the study of social distance’, as ‘the interrelated observations and theories of humans’ use of space as a specialized elaboration of culture’ (Reference Hall1966: 15). In broad terms, proxemics deals with how people put distance between themselves and others according to cultural, social and personal relations and conventions. Hall described personal space by clear-cut categories according to physical distancing. These are presented in Figure 26 in feet and with metric conversions.

Figure 26 Personal space.

WebHamster, CC BY-SA 3.0, via Wikimedia Commons

Personal space deals with the areas very close to the individual, whereas territory refers to larger surroundings that are controlled by the individual, such as the office or a specific chair in a conference room to gain a sense of privacy. Invasion of personal space or violation of what is considered by the person to be a private territory may lead to serious discomfort and communication breakdown. Goffman defines personal space as the ‘space surrounding an individual, anywhere within which an entering other causes the individual to feel encroached upon, leading him to show displeasure and sometimes to withdraw’ (Reference Goffman1971: 29–30).

However, as can be detected from Figure 26, the concept of proxemics needs to be completely revisited in screen-constrained scenarios, as all forms of interactions, whether personal, professional or public, are structured as if they were taking place in intimate spaces. This medium-specific affordance was identified by Reference Grayson and CoventryGrayson and Coventry (1998) who discussed the effects of visual proxemics in VMC. More recent studies investigate how distance in mediatized communication needs to be reframed vis-à-vis media affordances (Reference DrągDrąg, 2020).

In earlier research (Reference SindoniSindoni, 2011, Reference Sindoni2013, Reference Sindoni2014), I contended that proximity or detachment is fixed and frozen in video calls, because distance is not established between people in interaction, but between one individual and one machine, be it a desktop or mobile device. All forms of culture-dependent and socially constrained measures of distance are reduced to intimate, as the usual arrangement of space is that of ‘talking heads’, as described by Reference Licoppe, Morel, Broth, Laurier and MondadaLicoppe and Morel (2014), with only heads and part of the torso visible. The distance between oneself and others in VMI is therefore not semiotically built and engaged with in social interaction, but rather staged, meaning that it is enacted through technical arrangements, dealing with the participants’ positions in relation to the webcam, such as close, far, frontal, oblique or even absent if the webcam is switched off.

As discussed in Section 5.1, bodily presence and movements, as well as engagement with objects and physical surroundings, is constrained by screen framing. Furthermore, closeness or detachment are determined by webcam positioning that can be self-consciously adjusted by participants or, conversely, disregarded. Hundreds, if not thousands, of YouTube tutorials on video communication surfaced when the Covid-19 pandemic hit, and gave practical tips and suggestions on how to adjust webcam position, framing, lighting and setting to appear in the best possible light, or, to use Zoara’s words from Excerpt 32, ‘to look better’.

Users of multiparty platforms for everyday interactions do not seem to observe the ‘social codes’ of proxemics in VMI, producing feelings of awkwardness, as mentioned by some interviewees. The top ‘discomforting’ positioning was reportedly the oblique one. Similar reactions have been detected in the much smaller sample interviewed in 2020–22. Noticeably, no suggestions in my questions were made with reference to the kind of position that caused negative feelings. The extent of discomfort, combined with a negative evaluation of the other’s digital literacies, is summarized in the following comment:

In semiotic terms, this feeling of scarce attention or detachment articulated in various ways by the interviewees is consistent with theories of visual semiotics. An obvious source is the famous distinction between frontal angle as representing involvement and the oblique angle representing detachment made by Reference Kress and van LeeuwenKress and van Leeuwen (2020), with reference to how interactive meanings are produced visually in terms of attitude. The attitude of the sign maker (i.e., the one who produces the visual sign) can be analysed as subjective and objective. Subjective attitude is summarized in Figure 27.

Figure 27 Subjective attitude in interactive visual meanings (adapted from Reference Kress and van LeeuwenKress and van Leeuwen, 2020: 143).

Even though Kress and van Leeuwen analysed forms of visual design and not of spontaneous face-to-face interaction or VMIs, their categorization applies to all datasets and research on VMI in this Element. Some example screenshots of oblique positioning cartoonized participants with the Picsart Editor app are shown in Figure 28. These are all taken from multiparty video calls.

As Figure 28 shows, oblique angles are extremely frequent positions in which users are (self-)framed. This is due to webcam positioning that takes the participant sideways, albeit with varying degrees of obliqueness. Despite the frequency of this visual social framing, users seem to agree on the sense of detachment that this configuration produces, most likely unwillingly, in all those that ‘receive’ (i.e., see) this image projection. However, all instances in Figure 28 reproduce the ‘talking head’ configuration and, as such, succeed in maintaining contact with the other viewers. When asked about this framing, one user from the Philippines pointed out that he had never paid attention to this feature and explained this behaviour as follows:

Jerry was certainly right when commenting on the diverse framing expectations in multiparty versus one-to-one conversations; however, most users (roughly 72 per cent) reported on a negative feeling if and when the other discourse partners had been viewed sideways, even though this framing did not generally lead to communication breakdown, but only a ‘bad disposition’ towards the others.

Figure 28 Multiparty video chat users in oblique angle.

However, other webcam positionings seem to go even further, as shown in Figure 29.

Figure 29 Staged distance in multiparty video calls.

In this case, bad lighting in Row A, Figure 29, limits what can be seen of the discourse partner. While excessive proximity disturbs viewing in Row B, undue distance as ‘staged’ in Row C equally troubles the discourse partners’ viewing. All these forms of framing contribute to ‘staging’ participants in positions that do not facilitate communication, but rather create a sense of ‘distance’, ‘awkwardness’, ‘weird positions’ and ‘detachment’, as variously phrased by the participants. As these multiparty platforms for VMC are increasingly popular, improving awareness about the meaning making of multimodal resources for communication is essential in personal, professional and educational contexts.

As nicely put by one learner, proxemics should be understood as broader than the mere arrangement of webcam positioning in VMI and should incorporate other contextual features that may facilitate, or, quite the reverse, prevent smooth communication.

Both kinesics and proxemics are closely intertwined in VMIs and produce meanings that may vary across linguistic, social and cultural perceptions. Proximity and distance are made sense of in different ways in diverse contexts. What is shown, what is not shown, as well as how bodies, objects and surroundings impart meanings in screen-constrained environments, are other facets of video communication that alter what is generally possible (and impossible) in face-to-face interactions.