1 Introduction

Proposals that the semiotic system of gesture played a pivotal role in the evolution of language have been, and continue to be, influential (Reference ŻywiczyńskiŻywiczyński, 2018).* This statement, however, illustrates not so much a specific theory but an axis of debate in the field of language origins, along which “gesture-first” proposals traditionally compete with “speech-first” theories (Reference FitchFitch, 2010). Below this general characterization, there are many differences between gestural origin theories, including the understanding of basic notions such as the concept of gesture itself. Thus, we begin this chapter by discussing definitional issues concerning gesture, with differences both within and across fields. In the context of language origins, such differences in defining “gesture” have profound consequences for formulating and evaluating theories. Since our aim is to survey a considerable number of contemporary gestural theories of the origin of language, we do so by using a new typology, developed with the help of cognitive semiotics (Reference Sonesson, Ziemke, Zlatev and FrankSonesson, 2007; Reference Zlatev and TrifonasZlatev, 2015a), and in particular the notions of semiotic system and polysemiotic communication (Reference Zlatev and PetersStampoulidis, Bolognesi, & Zlatev, 2019; Reference Zlatev and PetersZlatev, 2019; Reference Zlatev, Devylder, Defina, Moskaluk and AndersenZlatev et al. 2023). In very brief terms, a semiotic system is a combination of signs or signals of a particular type, defined by characteristic properties, and the interrelations between these signs/signals. Universal human sign systems are language, gesture and depiction (the latter understood as forming marks on a two-dimensional surface that resemble three-dimensional or imaginary referents). Signal systems, for example spontaneous facial expressions and non-linguistic vocalizations, are under less voluntary control than sign systems (Reference Zlatev, Żywiczyński and WacewiczZlatev, Żywiczyński, & Wacewicz, 2020). Combinations of different sign and/or signal systems form the basis for polysemiotic communication or polysemiosis.

Applying this conceptual apparatus to gestural theories of language origins, the basic distinction relates to the question of whether the semiotic system of gesture played an exclusive role in early stages of language evolution or whether other semiotic systems were involved as well. A positive answer to the first question implies monosemiotic theories, which we review in Section 3; a positive answer to the latter implies polysemiotic theories, to which we turn in Sections 4 and 5. The latter are more commonly known as “multimodal theories,” but we avoid this term due to its excessive ambiguity.Footnote ¹ Importantly, we distinguish between two kinds of polysemiotic theories of language evolution: (a) equipollent, where language and gesture are considered equally prominent from the onset, and (b) pantomimic, where gesture played the main but not exclusive role in breaking from predominantly signal-based to sign-based communication.

After reviewing the evidence for each of the three kinds of gestural origin theories, we conclude that the last kind, namely pantomimic theories, appears to offer the most viable account of language origins. Further, it has the benefit of accounting for the evolution of polysemiotic communication as a whole.

2 Different Ways of Understanding Gesture

The work of Adam Kendon is hugely influential within gesture studies, and so is his characterization of gesture as bodily “movements that partake of […] features of manifest deliberate expressiveness to an obvious degree” (Reference Kendon2004, p. 14). “Expressiveness” implies that the communicator intends something by a gesture; “deliberate,” that this is done with a communicative intent; and “manifest,” that these features are to be discernible by an audience. As the notion of deliberateness is controversial, and is denied, for example, by Reference McNeillMcNeill (2012), gestures may be more generally defined as “expressive movements performed by the hands, the head, or any other part of the body, and perceived [predominantly] visually” (Reference Zlatev, MacWhinney and O’GradyZlatev, 2015b, p. 458, emphasis in original). How are gestures to be distinguished from other visually perceived communicative movements, such as adaptors and facial expressions (e.g. Reference Żywiczyński, Wacewicz and OrzechowskiŻywiczyński, Wacewicz, & Orzechowski, 2017)? An intuitive proposal of how to delineate this “lower boundary” of gesture is presented by Reference AndrénAndrén (2010), who argued that two dimensions of gestural meaning need to be distinguished: communicative explicitness (CE) and representational complexity (RC). Within each, there are three different levels and at least one of the two dimensions should be on the third level for the bodily act to count as a gesture. For example, the wave-bye gesture does not represent a goodbye but performs it, so it lacks the highest level of RC. Yet, it is typically produced with the communicative intent to be understood as taking farewell, and thus has the highest level of CE. On the other hand, an act of symbolic play performed in solitude would also qualify as a gesture, since by definition it is on the highest level of RC (if it is to be “symbolic”), even though it lacks a communicative intent completely. Most iconic (i.e. resemblance-based) gestures used in face-to-face communication would have the highest levels of both CE and RC.

Other researchers of human gestural communication adopt a narrower definition of gestures. Kendon’s approach encompasses so-called orofacial gestures: communicative movements of facial muscles and the tongue other than articulatory movements of speech; These would not be regarded as gestural by many (e.g. Reference Orzechowski, Wacewicz, Żywiczyński, Cartmill, Roberts, Lyn and CornishOrzechowski, Wacewicz, & Żywiczyński, 2014), though see Chovil (this volume). Reference McNeillMcNeill (1992, Reference McNeill2012) would further limit (prototypical) gestures to spontaneous and idiosyncratic hand and arm movements that are functionally integrated with speech, which is a very limited definition, based on his theory of the nature of gesture (Section 4.2).

On the other hand, within primatology, gestures are typically understood even more broadly than they are by Kendon, as concerning any communicative behaviors that involve body posture, facial expressions, and manual movements, and are mainly perceived visually (e.g. Reference Pika, Zlatev, Racine, Sinha and ItkonenPika, 2008a; Reference TomaselloTomasello, 2008).Footnote ² Voluntary control (Reference Byrne, Cartmill, Genty, Graham, Hobaiter and TannerByrne et al., 2017) and the related properties of flexibility (Reference Bard, Maguire-Herring, Tomonaga and MatsuzawaBard, Maguire-Herring, Tomonaga, & Matsuzawa, 2019) and plasticity (Reference Pollick and de WaalPollick & de Waal, 2007) are commonly used to differentiate gestures from other behaviors. For example, to initiate grooming, infant chimpanzees indicate the place they want to be groomed at, first by looking at this spot and then touching it (Reference Bard, Maguire-Herring, Tomonaga and MatsuzawaBard et al., 2019). This example also illustrates the receiver-directedness of ape gestures, further underlined by persistence: “[I]f the recipient’s response is not satisfactory, […] the signaller […] repeat[s] the produced signal” (Reference Fröhlich, Sievers, Townsend, Gruber and van SchaikFröhlich, Sievers, Townsend, Gruber, T., & van Schaik, 2019, cf. Reference Leavens, Hopkins and ThomasLeavens, Hopkins, & Thomas, 2004; Reference Tomasello, George, Kruger, Jeffrey and EvansTomasello, George, Kruger, Jeffrey, & Evans, 1985) and elaboration: If the recipient fails to respond in a satisfactory way, a gesture different from the original one may be used.

These findings testify to significant cognitive and semiotic complexity in ape gestures, possibly differentiating them from ape vocalizations. At the same time, this evidence does not amount to full-fledged communicative intent, which implies not just “intention” in the sense of volition, but a Gricean (second-order) intention that the addressee recognize the primary intention (Reference Zlatev, Madsen, Lenninger, Persson, Sayehli, Sonesson and van de WeijerZlatev et al., 2013), and it is not clear if apes can either produce or recognize such gestures. Further, unlike human gestures with the highest form of representational complexity (see above), ape gestures hardly refer to an object that is distinct from the communicator or the addressee, that is, they are “dyadic” (me–you) rather than “triadic” (me–you–referent) (Reference HurfordHurford, 2007), which prevents them from being full-fledged signs (Reference Zlatev, Żywiczyński and WacewiczZlatev et al., 2020).

As pointed out in the introduction to this chapter, these definitional differences imply that any comparison and discussion of gestural origins theories needs to proceed with care, and preferably with the help of a uniform conceptual apparatus, such as the one we propose below. In each of the following three sections, we review and evaluate one of the three basic types of gestural theories: monosemiotic, polysemiotic-equipollent, and polysemiotic-pantomimic.

3 Monosemiotic Gestural Theories

4 Equipollent Polysemiotic Theories

5 Pantomimic Polysemiotic Theories

6 Conclusions

In this chapter we have provided a survey of a number of gestural theories of the origin of language, using a new typology that divides theories first in terms of whether they viewed “gesture alone” as the starting point to the emergence of language (monosemiotic theories) or gesture in combination with other semiotic systems (polysemiotic theories). The latter were then divided based on whether gesture and speech were considered to play a (more or less) equal role from the start to the present (equipollent) or whether gesture first dominated, but the vocalizations that were there from the start eventually gained dominance (pantomimic).

To sum up, while there is a lot of value in monosemiotic gestural theories, in particular in regard to the role of iconicity for “bootstrapping” a shared system of signs, they were all shown to have difficulty in explaining the transition from gesture to speech. In tackling this problem, many proponents of such theories tend to overemphasize the role of speech in modern human communication, at the same time as they tend to overemphasize the role of gesture at the beginning of the evolutionary process. Another problematic move is the stress that they put on the continuity between ape and human gestures, whereby they find themselves hard-pressed to fully account for the differences between these two phylogenetically distinct forms of gesture.

As for equipollent polysemiotic theories, they are at least to some degree able to account for the role of co-speech gesture in modern human communication but tend to disregard other forms of gestural communication (and other semiotic systems, even more). They find it difficult to provide an evolutionarily satisfactory explanation of why gesture and speech differ semiotically (rather than just postulating that they are two sides of a “dialectic”), or indeed why they are so closely connected.

We concluded with pantomimic polysemiotic theories, which in a way capitalize on the strengths of the others: Like monosemiotic theories, they claim an advantage of gesture at an initial evolutionary stage, and like the equipollent theories (perhaps most of all, that of Adam Kendon) they proclaim the advantages of flexible polysemiosis. As such, they appear to be best able to explain modern communication: not only speech, but language in general; not only co-speech gesture, but a wide variety of gestures; not only language and gesture, but also other semiotic systems, like depiction and music. In several cases, they are capable of proposing plausible evolutionary scenarios, informed by research in ethnography and non-human communication. At the same time, they are able to explain the differences between human-specific and animal communication, including ape gestures. Finally, their notion of polysemiosis alleviates the transition problem of monosemiotic theories. However, it does not eliminate it completely, and pantomimic polysemiotic theories still have to propose a mechanism responsible for reconfiguring the original system of pantomime into modern language and modern polysemiotic communication. While there are some attempts in this direction, this is still very much work in progress. We have indicated some promising ideas in this direction, including the role of narrative.

1 Introduction

The purpose of this chapter is to capture the role and use of gesture in first language development and its integration in the child’s multimodal communicative system. It presents an overview of theories and methods that have triggered and facilitated the study of gestures in language development. The chapter is primarily focused on production and on gestures used by neurotypical hearing children who acquire spoken languages.Footnote ¹ The main issues are illustrated with detailed analyses of examples extracted from longitudinal data in English and French(Reference Morgenstern and ParisseMorgenstern & Parisse, 2012).Footnote ²

The human communication system develops in a space of shared meanings in which adults socialize children into language in situated activities; consequently, this overview highlights the crucial role of caregivers in adult–child interactions. We thus first focus on the role of gestures in adults’ communicative input and then follow children’s development into the use of the adult multimodal communicative system. We use the word “multimodal” to refer to a variety of semiotic resources used within the audio-vocal and visual-gestural modalities (such as speech, gesture, gaze, facial expressions). In our perspective, humans have created language by interacting via different types of meaning-making resources which have collectively been sedimented through experience and use into what is called “language.” As speech is the primary mode of communication children are progressively mastering, their use of gesture will be presented before they begin speaking, then as they produce their first words, and finally once speech is mastered. At the end of the developmental process, speech becomes clearly predominant but is both complemented and supplemented by other semiotic resources according to variables such as linguistic context, situation, interlocutor, activity, or discourse genre (Reference Cienki, Badio and KoseckiCienki, 2012).

2 Historical Background and Evolution of Theoretical Approaches and Methods

Child language research is one of the first fields in which spontaneous interaction data were systematically collected, initially through diary studies (Reference IngramIngram, 1989; Reference MorgensternMorgenstern, 2009) and later by audio and video-recordings shared worldwide thanks to the CHILDES project (Reference MacWhinneyMacWhinney, 2000). This data-centered method has allowed many researchers to confirm that, in the course of their development, children make their way through successive transitory multimodal systems, each with their own internal coherence (Reference Cohen and Linguistique de ParisCohen, 1924). This phenomenon can be observed at all levels of linguistic analysis. The starting point of language-acquisition scholars’ interest in gesture or visible bodily action could be summarized in de Laguna’s famous assertion that “in order to understand what the baby is saying you must see what the baby is doing” (Reference De LagunaDe Laguna, 1927, p. 91). Children’s productions are like evanescent sketches of adult language and can only be analyzed in their interactional context, taking into account the interlocutors’ interpretations and reactions, shared knowledge, actions, gestures, facial expressions, postures, and head movements, as well as words produced by the children (Reference Morgenstern and ParisseMorgenstern & Parisse, 2007; Reference Parisse and MorgensternParisse & Morgenstern, 2010).

Children’s language development has long been described by starting with the first vocalizations and the maturation of phonic abilities without paying much attention to motor skills, actions, gaze, and gestures. However, the first diaries of observers of child language had already contained dazzling insights about the multisensory qualities (through hearing, sight, touch, and sometimes through taste and smell) and forms of expression (speech, gesture, gaze, facial expressions) that make language acquisition a multimodal process. During the second half of the nineteenth century, child language development was studied through researchers’ observations of their own children. The detailed follow-ups on children’s language production, anchored in their daily lives, were a source of fascinating links between motor and psychological development, cognition, affectivity, and language. The “founding fathers” of the study of child development and language had great intuitions about gestures and their relation to language. In his notes on his son’s development, Reference DarwinDarwin (1877) highlighted the importance of observing the transition from uncontrolled body movements to intentional gestures. Reference DarwinDarwin (1872) was mainly interested in the expressiveness of the baby, and his diary entries therefore focused on the expression of emotions. He first emphasized the different functions of intonation. In addition, according to him, certain habitual movements become automatic and are associated with communicative functions. This can be illustrated by the first bodily manifestations of negation (gestures of avoidance or rejection which consist of turning the head or the body away, pushing things away with the hand). Darwin’s insights were confirmed by more contemporary research. Mimetic patterns of imitable actions, shared representations of objects that can be manipulated, anchor the acquisition of the child’s first gestures (Reference Zlatev, Persson and GärdenforsZlatev, Persso, & Gärdenfors, 2005). There is now evidence from brain imaging studies that the use of language involves motor representations concerning more than just the movement of the vocal apparatus (Reference ArbibArbib, 2012). It is through subtle shaping of daily actions and practices with objects in the environment that manual-gestural communication in social interactions leads the child to adopt conventional forms of symbolic gestural and verbal language. Reference RomanesRomanes (1889) also provided interesting ideas on gesture in his own diary study of his child. He compared human and animal gestures and mentioned the gestural language of the deaf as a sign of the universality of symbolic gestures.

Despite those early observations, gesture was not in the foreground in studies on child language during the first half of the twentieth century. However, thanks in particular to the work of Reference BrunerBruner (1975, Reference Bruner1983), actions and gestures were gradually considered by researchers who analyzed early development. They saw gesture as a system of communication that precedes the verbal system and then becomes complementary to it. According to Reference Werner and KaplanWerner and Kaplan (1963, p. 66): “Linguistic representation emerges from, and is rooted in, non-linguistic forms of representation.” For Reference BatesBates (1976), children’s first gestures have properties that had been in her time specifically attributed by linguists to speech. Thirteen-month-old children (Reference Bates, Bretherton and SnyderBates, Bretherton, & Snyder, 1988) produce manual gestures that are considered the equivalent of nouns (e.g. brushing their hair when seeing a brush is equivalent to saying “brush”).

Research in language acquisition has now developed tools, methods, and theoretical approaches to analyze children’s situated multimodal productions, as they provide evidence for links between motor and psychological development, cognition, affectivity, and language. Those links can only be established by conducting both quantitative and qualitative analyses through both natural settings and experiments.

As of the end of the twentieth century, thanks to video data linked to transcripts with specialized software (CLAN, ELAN, PHON),Footnote ³ detailed coding and analyses of multimodality have been possible and have opened whole new fields of research. It is especially the case for researchers who study language with a usage-based perspective, in its natural habitat, that is daily discourse: “the prototypical kind of language usage, the form in which we are all first exposed to language – the matrix for language acquisition” (Reference LevinsonLevinson, 1983, p. 284). We are now able to document in detail how the visual and vocal modalities come together in daily interaction and progressively shape children’s language. Video-recording tools have notably advanced the detailed analysis of the organization of human action and interaction (Reference MondadaMondada, 2019). These tools have shaped new avenues of research on language in interaction, as it is deployed in multiple ecologies, both in time (the moment-to-moment unfurling of an interaction) and over time (multiple recordings over several years of the same children in their family environment). Sacks, as he was grounding what became the Conversation Analysis framework, encouraged the use of video-recordings (Reference Sacks, Atkinson and HeritageSacks, 1984, Reference Sacks1992) so as to capture, analyze, and share sequences that unveil the structure of everyday practices.

Building on Conversation Analysis, an integrative, multimodal approach to language was developed thanks to contributions from Reference GoodwinGoodwin (1986, Reference Goodwin2013) and Reference Levinson, Enfield and LevinsonLevinson (2006). The recognition of sign languages also played an important role in considering the gestural dimension. An abundance of work on gesture and on the complementary role of semiotic forms is emerging and enriches the research already carried out on the development of children’s language. Specialists in linguistic anthropology (Reference HavilandHaviland, 1998), confronted with a multiplicity of cultures, many of them with purely oral traditions, have helped linguists become aware that the formal apparatus constructed to describe languages is largely based on written texts (Reference LinellLinell, 2005) whereas the most common uses of language are in face-to-face interactions (Reference GoffmanGoffman, 1963). Semioticians (Reference KressKress, 2010) also insist on the importance of taking into account the different simultaneous channels (auditory, visual, tactile) with which we conceptualize the world around us and express ourselves.

Research on spontaneous data (Reference Morgenstern, Inbal, Estigarribia, Tice and KurumadaMorgenstern, 2014, Reference Morgenstern, Mazur-Palandre and Colon2019) is characterized by the researchers’ attempts to capture how children become competent interlocutors as they learn to deploy the various semiotic resources at their disposal in relevant ways, for example, by coadapting to their conversational partners in various situations and environments. Children’s use of speech and gesture differs from that of adults and dynamically changes over time. Children’s multimodal communicative expression is thus analyzed in longitudinal data with an ethnographic approach in line with Kendon’s call for studies of use in context inspired by Reference EfronDavid Efron (1941/1972) and Reference WundtWilhelm Wundt (1921/1973). Children’s communicative profiles are shaped by their local environment (family home) and their microcultural norms. Through children’s everyday interactions in their ecological circumstances, we can understand both how language is “experienced” (Reference OchsOchs, 2012) and how experience is “languaged” in situated activities. Multimodal approaches include some combination of verbal content and the accompanying prosody, facial expressions, posture, gesture as well as gaze according to a dynamic deployment of the “scope of relevant behaviors” (Reference Cienki, Badio and KoseckiCienki, 2012, Reference Cienki2017), ideally taking into account multiple factors such as age, context, and affordances of the situation or interlocutors. When children’s gestures are analyzed in interaction, we try to capture what the modality of gesture affords its child users as a means of communication among the other semiotic resources they have at their disposal and as their motor, cognitive, and social skills evolve in time.

Video-recording has also had an invaluable impact on experimental methods focused on gesture development (Reference Congdon, Novack and Goldin-MeadowCongdon, Novack, & Goldin-Meadow, 2018). Gesture had often been overlooked in standard psychological observation and research until experimenters started using cameras and making detailed annotations of children’s comprehension and production of gestures during the tasks they were being asked to do. Thanks to a wealth of studies, “gesture research has fundamentally changed the way psychologists think about language, learning, and reasoning” (Reference Congdon, Novack and Goldin-MeadowCongdon et al., 2018, p. 497). With video-recorded data, time is suspended. Researchers cannot only relive/replay the experimental sessions as many times at they wish, but they can also change the annotation schemes, enrich them, and revisit them, and when the data are shared, the studies can be replicated more faithfully.

In both naturalistic and experimental data, gesture types, patterns, variations, and formal components can be coded, quantified, or analyzed in fine detail. Investigative experiments are often informed by naturalistic field methods and enable researchers to test what occurs in ecological settings. Researchers make an indispensable contribution to determining which hypotheses should be tested. Since young children’s productions in experimental settings might be influenced by a variety of performance-related factors (Reference AirentiAirenti, 2015), and since observers have to overcome great challenges when video-recording data in naturalistic or home environments, experimental results and spontaneous speech data must both be collected in order to capture children’s multimodal communication system and its development. New avenues include the use of motion capture (Reference Morgenstern, Mazur-Palandre and ColonDodane, Boutet, Didirkova, Ouni, & Morgenstern, 2019) and also involve computational modeling (Reference Kaplan, Oudeyer and BergenKaplan, Oudeyer, & Bergen, 2008). Those possibilities might further extend our knowledge about gesture–speech relations (Reference Abramov, Kopp, Nemeth, Kern, Mertens and RohlfingAbramov et al., 2018) but have not yet been fully deployed for studying the development of multimodal communication.

3 Gesture and Scaffolding in the Adult Input

Children’s language gradually develops into rich multimodal production using the variety of semiotic resources at their disposal through constant exposure to the adult input. It is thus crucial to account for the multimodal input surrounding children in order to understand how they co-construct their communicative skills thanks to adult scaffolding. Given that parents provide their children with other forms of expressions along with speech, we will refer to “child-directed communication” rather than the usual expression “child-directed speech” to insist on the crucial function of the child’s multimodal input.

Scholars have wondered for centuries about how children construct meaning. In his reconstruction of his developmental process, Saint Augustine, back in the fifth century, stressed the link between surrounding events, adults’ actions, gestures, and words in his own acquisition of language (Reference Augustine1996, I.8). Word learning is facilitated by multimodal cues produced by adults (gazing, pointing, touching, manipulating) (Reference Booth, McGregor and RohlfingBooth, McGregor, & Rohlfing, 2008). Adults provide redundant sensory information and positively affect infants’ attention. Symbolic gestures, especially pointing (Reference Iverson, Capirci, Longobardi and CaselliIverson, Capirci, Longobardi, & Caselli, 1999), have been shown to facilitate children’s comprehension by providing additional cues. The role of pointing and eye gaze in the construction of joint attention has been a key topic of research in child language development (Reference Baldwin, Moore and DunhamBaldwin, 1995; Reference TomaselloTomasello, 2003). As shown in example (1), video (1) (Table 14.1) from our longitudinal data (Reference Morgenstern, Blondel, Beaupoil-Hourdel, Benazzo, Boutet, Kochan, Limousin, Hickman, Veneziano and JisaMorgenstern et al., 2018),Footnote ⁴ the caretaker’s gestural information reinforces the vocal information. In the examples, “nb” indicates the utterance number and the column “Part.” indicates the relevant participant in the interaction.

Figure 14.1 Are you finished + palm down lateral movement (turn 4)

Reference Bahrick, Pickens., Lewkowicz and LickliterBahrick and Pickens (1994) explain that intersensory redundancy grounds how we detect stimuli that belong together and constitute a unitary event. They show that redundancy across the senses foregrounds core information for infants. Ellie’s mother presumably expects the gesture to help her daughter understand the meaning of her utterance. By linking gesture and speech, the mother grounds the word “finished” and facilitates the relation between sign and referent as she relies on both auditory and visual cues. She actually connects her own production of what Reference Bressem, Müller, Müller, Cienki, Fricke, Ladewig, McNeill and Bressem.Bressem and Müller (2014) call a “recurrent gesture” (palm down lateral, arms sweeping toward the exterior) used to express completion (Reference Ladewig, Müller, Cienki, Fricke, Ladewig, McNeill and BressemLadewig, 2014) to Ellie’s behavior. The child’s movement, seems to be addressed to her grandmother who is filming (the child waves her right arm quite excitedly, vocalizes and gazes at the camera [turn 2], and it is interpreted as another type of gesture (an emblem) by her mother [“that’s hello,” turn 3]). The child’s voice and arm movements are synchronous in turn 5: the gestural strokes and prosodic nuclei are perfectly aligned. Not only is Ellie’s body in harmony with her own voice (turn 5), despite a lack of semantic content, but after two repetitions on the part of the mother of the same multimodal production (turns 4 and 6), Ellie seems to echo her mother’s gesture in a more sketchy performance with a reverse movement from periphery to center, probably because of the positioning of her arms at the beginning of the gesture excursion (Figure 14.2, turn 7). In this way, Ellie’s body is in resonance with her mother’s.

Figure 14.2 Ellie’s resonance with her mother’s recurrent gesture (turn 7)

Caregivers, in a variety of cultures, synchronize words, actions, and gestures as they show objects, embody actions, or illustrate properties (Reference Zukow-Goldring, Dent-Read and Zukow-GoldringZukow-Goldring, 1997). These practices scaffold the referential process by linking words and gestures to objects and events (Reference Rader and Zukow-GoldringRader & Zukow-Goldring, 2010). Studies show that the nature and the frequency of maternal gesture influence the development of children’s communicative repertoires (Reference Rowe and Goldin-MeadowRowe & Goldin-Meadow, 2009). Caretakers also use gestures in playful scripts or songs and nursery rhymes, such as “bye bye” (waving hands), “peek-a-boo” (playfully hiding face with hands), “bravo” (clapping hands), and the French “ainsi font font font les marionettes” (a song that is accompanied by hand gestures representing puppets), in which they “teach” their children specific conventional gestures along with words.Footnote ⁶

Parents spontaneously use gestures in everyday communication. All those gestures derive from the culture in which the children are being raised and have very strong social and symbolic values. Caretakers embody their communicational intent more, and rely on redundant multimodal combinations, when the children are younger. Gestures are used to attract the child’s attention to particular events, objects, or words, to highlight, reinforce, and disambiguate the spoken content, and function as a support system secondary to speech (Reference Kelly, Arnon and ClarkKelly, 2011). This could be considered as a particular feature of language addressed to infants and young children. “Gestural motherese” is quite specific in terms of the type of gestures used: It involves more deictic and recurrent (semi-conventional) gestures and fewer representational gestures (Reference Özçalışkan and Goldin-MeadowÖzçalışkan & Goldin-Meadow, 2005).

However, not only do we need to integrate all semiotic resources to capture adult–child communicative behavior, but all actions and manipulations of artifacts could also be included. In real life and real time, humans communicate while they are involved in other activities such as eating, cooking, cleaning, drawing, digging. It is thus crucial to analyze interactions in the context of what could be called “multiactivity” (Reference Haddington, Keisanen, Mondada and NevileHaddington, Keisanen, Mondada, & Nevile, 2014). As we adopt a multimodal, dynamic, and situated approach to language use in interaction, the analysis of both actions and symbolic gestures in their multimodal and interactive context is particularly relevant to apprehend how adults and children distribute meaning. Actions and gestures produced by young children are often subtly integrated into adult–child interaction, interpreted, and reformulated into spoken language by their parents. Indeed, body actions, such as children lifting their arms to be picked up, are repeated sometimes several times a day, are interpreted and reformulated into spoken language by adults and thus become ritualized requests, just as reduplications such as papapa are interpreted as referring to the child’s father in French and reformulated into the conventional word “papa.” As children progressively gain control over their bodies, their behaviors are more and more often interpreted by those around them as meaningful and are linked to specific affordances and contexts: Bringing the palms of the hands together is interpreted as clapping and takes on a praising function or expresses glee (Reference Aronsson and MorgensternAronsson & Morgenstern, 2021); bringing the hand to the mouth or extending the arm toward a piece of fruit is interpreted as a request. Children’s body movements thus take on the status of conventional gestures and become intentional communicative signals (Reference TomaselloTomasello, 2008). Reference Goldin-Meadow, Mylander and FrankGoldin-Meadow, Mylander, and Frank (2007) have shown that when a mother reformulates the meaning of her infant’s gestures into words, those words for referents are more likely to enter the child’s spoken repertoire than words for non-reformulated referents are.

Past studies, our own research (Reference Morgenstern, Inbal, Estigarribia, Tice and KurumadaMorgenstern, 2014, Reference Morgenstern, Mazur-Palandre and Colon2019), and our examples illustrate that as children’s language skills develop, adults tend to give primacy to speech in their own productions but subtly use the visual-gestural modality to supplement and complement the verbal production according to their communicative needs. The use of multimodal child-addressed communication to reinforce verbal productions, along with actions, interpretations of children’s bodily movements into words, and adjustments to the child’s age as well as cognitive and linguistic skills, are crucial features of caretakers’ communication with young children.

4 Children’s Gestures before They Produce Spoken Language

As shown in Section 3, gestures provide a means for infants to enter communication prior to their own production of spoken language. Adults’ use of multimodal utterances seems to enhance children’s early comprehension of what adults are trying to communicate (Reference Pfandler, Lakatos and MiklósiPfandler, Lakatos, & Miklósi, 2013; Reference Wu and Gros-LouisWu & Gros-Louis, 2014). More specifically, young children assign meanings to nouns (Reference Clark and EstigarribiaClark & Estigarribia, 2011), to verbs (Reference Goodrich and Hudson KamGoodrich & Hudson Kam, 2009; Reference Ozçalışkan, Gentner and Goldin-MeadowÖzçaliskan, Gentner, & Goldin-Meadow, 2014), and to prepositions (Reference McGregor, Rohlfing, Bean and MarschnerMcGregor, Rohlfing, Bean, & Marschner, 2009) through the mediation of gesture.

Along with vocal productions, eye-gaze is the first semiotic resource used by children to enter communication. After three months old, infants increasingly look reliably in the direction of another person’s own attention as signaled by the direction of their gaze (Reference BrunerScaife & Bruner, 1975). As of nine months old, they follow the adults’ gaze along with pointing gestures (Reference Brooks and MeltzoffBrooks & Meltzoff, 2005). Young children then progressively learn to use their gaze to monitor and guide others’ attention.

In early childhood vocal-motor coordination develops with neural maturation, shown in the rhythmic qualities shared by early hand-banging movements and canonical babbling (Reference MasatakaMasataka, 2003). Reference Iverson and FaganIverson and Fagan (2004, p. 1063) suggest “a link between more speech-like vocalizations and manual activity that may be a precursor to coordinated manual movement of the sort involved in adult gestures” (see example [1], video [1] for an illustration).

Meaningful manual actions precede and pave the way for the development of language and share a semantic link with gestures and words (Reference Capirci, Contaldo, Caselli and VolterraCapirci, Contaldo, Caselli, & Volterra, 2005) as we have shown in example 1. Between nine and ten months old, children use ritualized requests like open-close grasping motions or pulling an open hand to obtain something (Reference Bates, Benigni, Bretherton, Camaioni, Volterra and BatesBates, Benigni, Bretherton, Camaioni, & Volterra, 1979). If infants’ first vocalizations imitate the prosodic patterns that are most salient and frequent in their environment and that have pragmatic and affective functions (Reference Esteve-Gibert and PrietoEsteve-Gibert & Prieto, 2013), thanks to their finer hand-motoric skills, gesturing also allows children to express more specific semantic functions and thus communicate meanings that they might not be able to express with vocal means (or at least that are not captured in their vocal productions by adults). According to Reference Goldin-MeadowGoldin-Meadow (1999, p. 423), gesture may thus serve as a “way-station on the road to language.” It has also been shown that “late-bloomers” (children who seem to have a delay in their first word production, but catch up by the age of three) can be differentiated from late-talkers thanks to their gestural performance (Reference Capone and McGregorCapone & McGregor, 2004); gesture can compensate for verbal expressive deficits.

Conventional gestures enter the child’s repertoire at around 10–11 months old. The most frequent and usually the first gesture used is pointing accompanied by gaze. It has a variety of functions and uses (Reference Bates, Benigni, Bretherton, Camaioni, Volterra and BatesBates et al., 1979; Reference Liszkowski, Carpenter, Henning, Striano and TomaselloLiszkowski, Carpenter, Henning, Striano, & Tomasello, 2004) and a variety of forms, the most common being index-finger pointing (Reference AndrénAndrén, 2010; Reference BatesBates, 1976; Reference Franco and ButterworthFranco & Butterworth, 1996; Reference Morgenstern, Morgenstern and Goldin-MeadowMorgenstern, 2022). Deictic gesture development indicates how children gradually distance themselves from objects physically, without using touch and manipulation, and enter symbolic communication (Reference Capone and McGregorCapone & McGregor, 2004).

Children’s pointing used in interaction is integrated in the dialogue by adults as (proto)speech acts, and could be treated as requests or declaratives according to context (Reference BatesBates, 1976; Reference MarcosMarcos, 1998). When children can point not only to request an object but also to refer to it, that reveals their ability to enter joint attention, but also to create a disconnection. Pointing at a location is progressively used to refer to an absent entity (Reference Le GuenLe Guen, 2011) and predicts to some degree the ability to use language.

The headshake is another gesture produced in early childhood in a wide range of cultures. It is taken up by children at first to mark refusal and rejection (Reference Beaupoil-Hourdel, Morgenstern, Boutet, Larrivée and LeeBeaupoil-Hourdel, Morgenstern, & Boutet, 2015; see also Reference GuidettiGuidetti, 2005). In example (2), video (2) (Table 14.2), Ellie is one year and two months old. She shakes her head several times and the situation illustrates how the conventional gesture can be seen in a continuum with the action of avoidance.

From her past experiences, Ellie knows that when her mother prepares a wet tissue and moves it toward her face, she is about to wipe her face.

In turn 2, Ellie is already preparing to evade the wiping, but not too vigorously; maybe past experience has shown her that it is an unavoidable ritual and that as unpleasant as it is, it is still bearable. Her avoidance is therefore not a complete rejection. However, in turn 5, she is more adamant in her refusal of more food. Her movement allows her to avoid the food physically but also communicate her negation through a headshake and a vocal production “da” which could be associated to the word “done” as reformulated in turn 6 in her mother’s recast “are you done?”.

Both her grandmother with verbal questions (turns 9 and 11) and her mother with the attempt at giving her another spoonful (turn 13) check that the meal is really over. This is made clear not only through Ellie’s headshake, which avoids and refuses the food once again (turn 14), but also through her bringing both arms up in what is interpreted by her mother as a request to get out of the highchair (turn 17 “do you want to go out then?”).

Figure 14.3 Ellie’s preparation for avoidance

Figure 14.4 Avoidance and refusal

In the same video, when her mother says the word “hungry” (turn 28), Ellie represents the idea with a playful smacking of her lips, which is an action sometimes performed by family members when saying “hungry” (Figure 14.6, turn 29), even though she has demonstrated previously and confirms at the end of this sequence that she wants breakfast to be over, as formulated by her mother (turns 30 and 31).

Figure 14.5 Request to get out of the chair

Figure 14.6 a, b Smacking lips

What have been called “representational gestures” in the literature are used around the age of 12 months, before the 25-word milestone (Reference Capone and McGregorCapone & McGregor, 2004). A child can manually represent the action of holding a glass and drinking, or use her hand to comb her hair. Reference Goodwyn and AcredoloGoodwyn and Acredolo (1993) argue that a gesture or a word is symbolic if it refers to multiple examplars (including pictures or in absence of the referent), if it is produced spontaneously (without following the model of an adult), and if it is not part of a well-rehearsed routine. The status of those gestures often performed in pretend play is, however, very different from coverbal representational gestures used with speech later on.

Within a few months, children’s gestural repertoire is enriched, especially in cultures or families where the input is gesturally varied thanks to both the general multimodal features of face-to-face interaction and to specific use of gestures in infant-directed communication (as shown in Section 2).

Reference Iverson, Capirci and CaselliIverson, Capirci, and Caselli (1994) demonstrate that 16-month-old children show a preference for either words or gestures, but by 20 months, types and tokens of spoken words increase significantly. Reference Butcher, Goldin-Meadow and McNeillButcher and Goldin-Meadow (2000) longitudinally observed three boys and three girls during the transition from one-word to two-word speech. During the first session, most of the subjects (five out of six) produced the majority of their gestures without speech. During the following sessions, there was a decline in the proportion of gestures produced. By the end of the observation period, the children mainly used gesture-speech combinations. This is when the gesture-speech integration begins with the progressive predominance of speech in hearing populations.

5 Children’s Gestures as They Enter Verbal Language

Gestures and speech develop in hearing children as they manipulate more and more words. The child’s multimodal communication skills emerge in their first cross-modal combinations and multimodal constructions.

Speech and gesture together form an integrative system (Reference McNeillMcNeill, 1992). However, multimodal productions are quite rare at first or are mostly gestalt communicative acts in which body movements and vocal productions are not fully controlled. Children first use the audio-vocal and visual-gestural modalities together to communicate about the same element, as in example (3). In example (3), video (3) (Table 14.3), when she is one-and-a-half years old, Ellie waves her right arm, shakes her head and says “no” at the same time, but the headshake is pursued vigorously for a while. The gestural modality is amplified and maybe not as finely controlled as it will be when Ellie gets a little older and uses her semiotic resources with more expertise.

First gestures are tightly connected to first words. Reference Iverson and Goldin-MeadowIverson and Goldin-Meadow (2005) found that pointing precedes and predicts lexical acquisition during the early stages of language learning. The authors have shown that there is an increase of pointing gestures during the period when children’s vocabulary expansion is the largest. At least in Western cultures, parents often respond to children’s pointing by labeling, which in turn helps children integrate those words into their verbal repertoire (Reference Bruner, Sinclair, Jarvelle and LeveltBruner, 1978; Marcos, 2003). Productive use of gesture and speech is linked. Children who produce more meanings in gesture at 14 months showed faster growth in productive vocabulary use (word types) between 14 and 46 months (Reference Rowe, Raudenbush and Goldin-MeadowRowe, Raudenbush, & Goldin-Meadow, 2012).

At that age, children still often use gestures rather than words to express themselves, as Ellie does emphatically in example (4), video 4 (Table 14.4).

When her mother says “too late” after Ellie has poured too much sugar in the bowl to make her cake, Ellie deploys a beautiful shrug, which could be categorized as a recurrent or pragmatic gesture (Reference DebrasDebras 2017), and which is part of the child’s cultural repertoire of gestures: She lifts her shoulders as high as she can, slightly opening her two arms with a radiant smile on her face (Figure 14.7). Her grandmother reformulates the same meaning into speech by saying “she just like gave up.”

Figure 14.7 a, b Smiling, lifting shoulders, and opening arms

After that period, use of two simultaneous modalities for two different elements precedes the onset of two-word speech. This might be linked to children’s cognitive ability to produce simultaneous information before they can express the elements sequentially. Therefore, what Reference Levelt, Bellugi and Studdert-KennedyLevelt (1980) calls the “linearization problem” – in speech, we can only order the information successively in a linear format and we need to organize the elements and select what comes first – does not affect children who, just like adults, have multimodal resources to express themselves. By combining gesture with speech synchronously, they can form a predicative structure. In example (5), video (5) (Table 14.5), Ellie is pointing at various elements in the room and saying a color each time. The combination of gesture (pointing at Bob’s trousers) and word (black) in answer to the question “what color are Bob’s trousers?” could be glossed as “the trousers are black,” a multimodal utterance in which the subject, trousers, is designated through the gesture and the predication, “black,” is expressed with a word. Ellie expertly creates a multimodal construction with pointing + word, even though she does not yet appear to have mastered the association between color adjectives and the actual color of objects she points at, as the extract illustrates (she says black but the trousers are actually blue).

In another session, at one year and 11 months, Ellie and her grandmother are playing with dolls called Maddie and Susie. The grandmother is taking care of Maddie while Ellie is taking care of Susie. The grandmother takes a bottle and pretends to feed Maddie. But Ellie shouts “Susie bottle!” and simultaneously produces a headshake. The headshake is often called a co-verbal gesture, but at least in this case, the spoken words could just as well be called “co-gestural.” The two modalities, verbal and gestural, are integrated to construct the multimodal utterance which could be interpreted as “no Grandma, the bottle is for Susie and not Maddie.” The predominance of the verbal channel characterizing Ellie’s communicative productions at the end of her second year leads us to subordinate gesture to speech from this age on in our terminology. However, in Ellie’s productions, each modality is used with a specific function and they are assembled to form a negative assertion. The two modalities do not seem to be organized into a hierarchy, but her linguistic community lead the child to favor the verbal channel for practical reasons in her daily communicative practices in a world in which we speak as we cook, eat, draw, clean, or drive and can skillfully combine bodily actions and spoken communication.

Thus, around the age of two, speech comes to be used more than gestures to refer to objects (Reference Capirci, Contaldo, Caselli and VolterraCapirci et al., 2005; Reference Iverson, Capirci and CaselliIverson et al., 1994). During their second year, children also start making combinations. They produce many more [gesture+word] and [word+word] combinations than [gesture+gesture] combinations (Reference Capirci, Contaldo, Caselli and VolterraCapirci et al., 2005; Reference Capirci, Iverson, Pizzuto and VolterraCapirci, Iverson, Pizzuto, & Volterra, 1996; Reference Goldin-Meadow, Morford, Volterra and ErtingGoldin-Meadow & Morford, 1990; Reference Stefanini, Bello, Caselli, Iverson and VolterraStefanini, Bello, Caselli, Iverson, & Volterra, 2009). In most cases when gestures are combined, one or both of the gestures are deictic rather than representational or pragmatic (Reference Capirci, Iverson, Pizzuto and VolterraCapirci et al., 1996; Reference Morford and Goldin-MeadowMorford & Goldin-Meadow, 1992; Reference VolterraVolterra, 1981), as in example (4).

Studies have also shown that [gesture+speech] combinations reliably predict the onset of [word+word] combinations (Reference Butcher, Goldin-Meadow and McNeillButcher & Goldin-Meadow, 2000; Reference Capirci, Iverson, Pizzuto and VolterraCapirci et al., 1996; Reference Iverson, Capirci, Volterra and Goldin-MeadowIverson, Capirci, Volterra, & Goldin-Meadow, 2008). Interestingly enough, a majority of all types of gestures (deictic or representational) are coordinated with speech rather than being used in isolation during this transition period (Reference AndrénAndrén, 2010). Even though speech and gesture have coevolved (Reference Levinson and HollerLevinson & Holler, 2014) and are codeveloped through infancy, hearing children who are bathed in multimodal input and capture language with all their senses are progressively directed toward the predominance of vocal communication mediated by adult scaffolding. When children are between 18 and 30 months, there still seems to be a symbiotic relation between speech and gesture. According to Andrén’s analyses of five Swedish children (Reference Andrén2010), a majority of gestures observed were coordinated with speech. Gesture seems to play an important role in the productive use of multiword speech as there is an increase in all types of gestures in association with speech between 24 and 30 months old. But speech becomes more dominant, confirming that it is the “typifying medium par excellence” in hearing dyads (Schutz, 1953, p. 10) as there is a shift to a more productive or more generalized mode of communicating in which multiword utterances become more common than one-word utterances associated with gestures.

6 Children’s Gestures When Verbal Language Is Mastered and Dominant

Once the multimodal communication system has been mastered, gesture and speech work more subtly together in later childhood. As speech develops, gestures become more and more diverse and elaborate, especially in their relation to speech. Representational gestures tend to appear more and more with verbs and adjectives, rather than with nouns (Reference Capone and McGregorCapone & McGregor, 2004) as shown in example (6), video 6, Table 14.6.

Intriguingly, the visual modality is used to represent what cannot be expressed in words as well: the exact dimension (or supposed dimension) of a baby gorilla and her mother. Figures 14.8 and 14.9 illustrate how Ellie gestures to demonstrate sizes. As she is co-narrating the story with her mother for her grandmother, she is constantly visually checking with her mother (who is filming at the time) that her gesture is “correct.”

Figure 14.8 “I think it was this big”

Figure 14.9 a, b Size readjustment: “this big?”

After her mother and grandmother’s comments (turns 5b and 6), Ellie adjusts her gesture Figure 14.9 (turn 7).

Then Ellie spreads her arms much wider and the amplitude of the space between the hands is now accompanied with her arms extending in a V shape and her hands bending wide (Figure 14.12, turn 9) to indicate the “huge” (turns 10a and 10b) size of the mother gorilla.

Figure 14.10 “the Mum is this big”

Communication thus remains multimodal in face-to-face interaction throughout the rest of our life span. After representational gestures have increased and diversified even more, then beats are used (Reference CollettaColletta, 2004) as well as more metaphorical gestures and abstract deictics.

However, in his data, Reference AndrénAndrén (2010) found that as children’s language skills developed, conventional (recurrent) gestures were more likely to be used with speech than representational gestures. This is in line with Kendon’s observations (2008) on speakers with a rich repertoire of conventionalized gestures which are “fully integrated into the flow of everyday discourse” (p. 360). Andrén also found in his spontaneously recorded family interactions that the complexity in speech was symmetrically related to the complexity in gesture.

Studies have shown that, as of five years old, children’s motoric skills become more controlled and thus gesture-speech integration turns out to be more subtle and complex (Reference Alibali, Kita and YoungAlibali, Kita, & Young, 2000). Children’s gestures and speech become more closely aligned and complementary in terms of semantics, syntax, and pragmatics as they produce more complex multimodal utterances (Reference Morgenstern, Inbal, Estigarribia, Tice and KurumadaMorgenstern, 2014, Reference Morgenstern, Mazur-Palandre and Colon2019). Reference CollettaColletta (2004) also showed that multimodal story-telling skills (linguistic, prosodic, and gestural skills in narration) develop together and simultaneously. Once they have mastered the complexity of speech, children can still resort to multiple semiotic resources and conventionalized arbitrary as well as non-arbitrary mappings grounded in their sensory and affective experiences to fully express the various facets of their inner thoughts, their desires, feelings, and to make comments, explanations, or narratives. Reference Colletta, Pellenq, Nippold and ScottColletta and Pellenq (2009) conducted multimodal analyses of explanations produced by French children aged three to 11 years. The authors found an increase in all observed measures: duration, number of syllables, number of clauses, and use of connectives as well as use of co-speech gestures. Reference CollettaColletta (2009) showed that children aged nine years and over relied more on gesture and gaze resources and delivered truly embedded narratives, acting as narrators, rather than only recounting facts and events they witnessed. But multimodal language production is found to be closely tied to context and genre. Reference Alamillo, Colletta and GuidettiAlamillo, Colletta, and Guidetti (2013) compared explanations and narratives produced by the same group of children. The task had effects on the use of both language and gesture: Cohesion markers were more often used in narratives when gestures and subordinate markers were more frequent in explanations. Reference ÖzyürekÖzyürek (2014) also showed how the different multimodal layers of integrated processing depend on context, pragmatic knowledge, and the communicative intent of the speaker.

Though a rich range of facial expressions expressing emotions is discriminated in early childhood, their production, especially those expressing stance in interaction, develops at different rates and continues all the way to adolescence (Reference Odom and LemondOdom & Lemond, 1972). More recent work conducted in different regions of France indicates that the use of facial expressions is a complex developmental process influenced by several factors including age, gender, regional differences, and types of expression (Reference Grossard, Chaby, Hun, Pellerin, Bourgeois, Dapogny and CohenGrossard et al., 2018).

As children learn to modulate their expression with a rich pallet of multimodal tools, they can progressively use the various degrees of iconicity (Reference EmmoreyEmorey, 2014) that our semiotic resources can offer. They can use the most abstract, indirect relationships that have been generalized and conventionalized in language, as well as transparent imitation and embodied direct relationships that support their ability to refer to entities in their absence (displacement) and enable their interlocutors to capture their meaning. But gesture is not solely guided by our visual modality and by imagistic relationships between form and meaning. As we have illustrated in our examples, gestures are very much linked to actions. There is a continuum between actional/manipulative and communicative/symbolic gestures which also explains why blind children use gestures (Reference Iverson and Goldin MeadowIverson & Goldin Meadow, 1998). Proprioception and sensorimotor skills are an integral part of children’s entrance and mastery of multimodal communication.

In example (7), video (7) (Table 14.7), we illustrate Madeleine’s multimodal skills at the end of our longitudinal data.Footnote ⁸ As she is about to be seven years old, the little girl is able to recount both the content of the events and the discourse she has witnessed. In her productions related to the act of portraying (Reference StreeckStreeck, 2008), she has acquired the skills to show the situation reported in both the vocal modality and the gestural modality. She depicts her mother finding out on her phone’s agenda that she has a business meeting around the time of Madeleine’s birthday party.

The first instance of reported speech attributed to Madeleine’s mother (turn 1) is not introduced by a quotative verb (turn 2): She uses non-segmental markers to indicate that the viewpoint has changed. This involves a change in voice and accentuated gesturing with specific facial expressions. The observer understands her perfectly (turn 3). Madeleine’s use of gaze to change perspective is particularly interesting: She stops gazing at the observer as she takes on the role of her mother, in a personal transfer reminiscent of what Reference CuxacCuxac (2000) describes in sign language narratives (turn 6). The alternation of gaze is very consistent throughout the sequence. Her gaze at her interlocutor indicates that she is in the discourse space. When gaze leaves the observer and is either on her hands “holding” the phone (Figure 14.11) as she embodies her mother, or up in the air as she makes exaggerated facial expressions, she expresses that she is entering narration space. But gaze management becomes even more complex. As she plays the role of her mother addressing herself, Madeleine, the little narrator, looks into the observer’s eyes (turns 6, 9, Figure 14.12) and thus makes the observer transfer into her own role when the event took place: The observer becomes Madeleine, while Madeleine acts as her own mother.

Figure 14.11 Narrative space

Figure 14.12 Discourse space

(reported speech inside narration)

Madeleine’s voice has become that of her mother, expressed by the subtle pitch variations in her prosody. Madeleine’s body embodies that of her mother with her gestures and facial expressions.

We also notice in this passage how some multimodal constructions are used automatically by Madeleine, such as the rather sophisticated recurrent gesture involving a hand configuration performed with both hands (index extended), a particular localization and a cyclical movement that accompany the verb “régler” in French (to settle the problem, Figure 14.13).

Figure 14.13 a, b, c [régler + recurrent gesture]

At seven years old, Madeleine has become an expert multimodal communicator who masters the different functions of each modality and handles multimodal constructions to express both her own perspective and the perspectives she is able to attribute to others.

More studies of children across languages and cultures are needed to fully capture in detail the development of the various categories of gesture and find out more about similarities and variations between individual children’s pathways to master the complexity of multimodal communication.

7 Conclusion

From an early age, children have cognitive skills that allow them to analyze the language input that surrounds them and thus structure their own practices. Without mastering the complex uses of each word, each gesture, each intonation pattern, and each multimodal construction, they can still construct meaning.

Children use all the resources provided by their bodies to express themselves when they are brought up in an environment that is favorable to multimodal communication and language development. They construct a shared repertoire of gestures and words with their interlocutors. But they constantly use the multimodal resources at their disposal and progressively enrich the complexity of their production, as the examples from Ellie and Madeleine’s longitudinal data have illustrated.

More specifically, throughout infancy, children learn the meaning of gestures in multimodal utterances produced around them. Child-directed communication is very often more emphatically multimodal than is typical in adult-addressed language. Children are also socialized to the use of gestures through specific routines, songs, and rituals in which gestures are integrated or even focused on. Coordination of gaze, gesture, facial expressions, posture, and speech for communication could already be captured in children’s early productions in gestalt, multimodal communicative acts. However, this orchestration of semiotic resources used for interaction develops steadily throughout childhood and into adulthood. Children learn to dissociate the uses and specific functions of each modality and to master the dynamic multimodal communicative system used around them and with them in their daily interactions.

1 Introduction

Most people in the world speak more than one language – many learning new languages throughout life, inside and outside of classrooms, for reasons of study, work, migration, religion, or pleasure. Studies of second language acquisition (L2 or SLA) or foreign language acquisitionFootnote ¹ (FLA) examine how this comes about, querying how a new language emerges and develops in the mind in the presence of one or several existing ones. SLA studies track developmental trajectories and “outcomes,” and explore the factors assumed to influence acquisition, such as the nature of the languages that come into contact (e.g. the difference between being a Russian, Dutch, or Japanese learner of English), learners’ age (child vs. adult), skill levels or proficiency, individual cognitive capacities such as working memory and language-learning aptitude, the learning situation (classroom vs. naturalistic settings), the type of instruction (form vs. meaning-based), the amount of input/exposure, and patterns of output/use in conversation and interaction (see Reference Gass and MackeyGass & Mackey, 2012 for overviews). SLA is thus a vast field of study with linguistic, psychological and neurocognitive, social, anthropological, and pedagogical subfields. Until quite recently, gestures were not seen as scientifically relevant in any of these subfields. However, as embodied views of cognition and language gain ground (e.g. Reference BarsalouBarsalou, 2008; Reference Glenberg and KaschakGlenberg & Kaschak, 2002), and evidence grows that gestures are an integral part of language production and comprehension (Reference ClarkClark, 1996; Reference Goldin-MeadowGoldin-Meadow, 2003; Reference Kendon, Siegman and PopeKendon, 1972, Reference Kendon2004; Reference McNeillMcNeill, 1992, Reference McNeill2014; Reference Volterra, Beronesi, Massoni, Volterra and ErtingVolterra, Beronesi, & Massoni, 1990), gestures are becoming important to SLA concerns as part of the cross-linguistic, psycholinguistic, and sociolinguistic variation to consider.

Over the past two decades, gesture and SLA has become a thriving field of study in its own right (see Reference GullbergGullberg, 2006b, Reference Gullberg, Robinson and Ellis2008; Reference Gullberg and McCaffertyGullberg & McCafferty, 2008; Reference Stam and ChapelleStam, 2012; Reference Stam, Buescher, Phakiti, De Costa, Plonsky and StarfieldStam & Buescher, 2018 for overviews). The work largely concentrates on two broad domains: gestures as a window onto language acquisition issues; and gestures as a medium of acquisition where the effect of gesture on acquisition is examined. In principle, a third domain could be the study of the acquisition of gestural repertoires in and of themselves (cf. Reference Gullberg, Müller, Cienki, Fricke, Ladewig, McNeill and BressemGullberg, 2014), but very little such work exists.

The current overview will focus on traditional SLA, that is, on speakers whose language skills are still emerging (traditionally “L2 users, L2 learners”) and whose learning is at stake, leaving bilingualism aside (but for a review of gestures and bilingualism, see Reference Gullberg, Bhatia and RitchieGullberg, 2012a). Moreover, the review will focus on manual gestures rather than on the full repertoire of behaviors that are part of multimodal communication (gaze, body orientation, head movements, facial expressions, etc.). This chapter summarizes current research on what gestures reveal about SLA and L2 development, and how gestures affect SLA in interaction and in instruction. It closes with a possible research agenda, outlining further issues to explore. Methodological issues for gestures and SLA will not be discussed (but see Reference GullbergGullberg, 2010, Reference Gullberg and Chapelle2012b for overviews). A final terminological point is that we will follow Reference KendonKendon (2004) and refer to gesture functions (representational/referential, pragmatic) rather than gesture “types” wherever necessary.

2 Gestures in Acquisition

3 Gestures in L2 Interaction and Instruction

4 A Research Agenda for Gestures and SLA?

The study of SLA and gesture has made considerable progress in the past 20 years. However, many things remain unexplored. Across all domains reviewed above, there is an obvious need to move beyond the lexicon (and the domain of motion) and consider the acquisition of morphosyntax, (sustained) discourse, phonology, phonetics, figurative language, idiomatic expressions, and pragmatics both in production and in comprehension, both offline and online. The SLA field is rife with claims built on speech alone, ranging from the effects of the languages in contact, individual differences in cognitive makeup, language use, to type of instruction, and so on. These claims could and should be tested taking gestures into account (and preferably on a wider language sample than that which has been examined so far).

Fundamental questions about the study of SLA and gesture are when, why, and how speech and gestures change in L2; why L2 speech seems to change more readily than L2 gestures; under what conditions gestures do change, whether through imitation, through changes in language use, or both. We clearly need more longitudinal work to improve our understanding of these vital issues. Moreover, while we know very little about whether, when, and how L2 learners produce language-specific co-speech patterns, we know even less about the SLA of conventional, quotable gestures, or emblems. If communicative fluency and cultural appropriateness are seen as important to SLA, then the acquisition of gestural repertoires of linguistic and cultural communities should be included. Although a few studies investigate L2 users’ comprehension of culture-specific quotable gestures (e.g. Reference JungheimJungheim, 2006; Reference Molinsky, Krabbenhoft, Ambady and ChoiMolinsky, Krabbenhoft, Ambady, & Choi, 2005; Reference Wolfgang and WolofskyWolfgang & Wolofsky, 1991), it remains largely unknown whether L2 learners themselves produce them. For example, do L2 speakers learn to respect handedness taboos (e.g. Reference Kita and EssegbeyKita & Essegbey, 2001), or to produce appropriate gestural backchannelling, shifting from head toss to headshake (Reference Morris, Collett, Marsh and O’ShaughnessyMorris, Collett, Marsh, & O’Shaughnessy, 1979)? Since emblems function like idiomatic expressions, they may be subject to the same acquisition difficulties as spoken idiomatic expressions (e.g. Reference IrujoIrujo, 1993). However, since they are often assumed to be inherently “salient” in the absence of co-occurring speech, they may also be easier to acquire than spoken idiomatic expressions. A contrastive study of idiom acquisition in speech vs. gesture could illuminate whether the visual modality enjoys an advantage in learning.

Further, the majority of work on SLA and gesture probes L2 production. We know much less about L2 gesture perception – both about learners’ perception of surrounding gestures in classrooms, as well as in study abroad and immersion contexts, and the actual effect of gestures on L2 comprehension (but see e.g. Reference Sueyoshi and HardisonSueyoshi & Hardison, 2005, and Reference Drijvers and ÖzyürekDrijvers & Özyürek, 2020, for somewhat conflicting findings); and about native speakers’ perceptions of learners’ (foreign) gestures. Both domains need elucidating.

We are beginning to get a handle on the measurable effect of explicit gesture training on SLA and memory, but we still know virtually nothing about implicit effects of gesture processing in SLA, that is, the learning effect of gestures that are not part of explicit training, but just normal language use. We know little about the difference between instructed and uninstructed multimodal SLA, about different kinds of classroom instruction (focus on form vs. on meaning), and the role gestures may play in classrooms with literate vs. illiterate L2 learners (e.g. Reference Tarone and BigelowTarone & Bigelow, 2005).

A final methodological point is worth making. We should all aim to provide much more detail on gestures themselves in studies of gesture and SLA, and on their relationship to co-occurring speech. A surprising number of published papers provide no or only the most minimal information about the articulatory, spatial, and temporal properties of the gestures under study, and their temporal relationship to speech. Moreover, it is often assumed that gestural function (“gesture type”), semantic content, and coexpressivity between speech and gesture are easily established when they are not; or that gestures are monofunctional when in fact that are deeply multifunctional. We need to be more attentive to all these details. Data sharing and the creation of open multimodal learner corpora is obviously challenging in gesture studies, but that is precisely why it behooves us all to be more explicit than we typically are in the interest of replicability.

5. Conclusions

The study of gestures and SLA can now be said to be a field in its own right. This field considers gestures both as a tool to study acquisition, and as a phenomenon to be studied per se. The double nature of gestures as interactive, addressee-directed phenomena on the one hand, and as internal, speaker-directed ones on the other, make them deeply relevant to issues of L2 acquisition. Theoretically, this new field still needs to integrate concerns from SLA and gesture studies, from language studies and cognitive (neuro)science. Currently, cognitive learning science lends more attention to gestures than SLA studies do, but this state of affairs will change as the body of work grows, as terminologies and methods become more unified, and as the multimodal view of language gains further ground. The challenge for us all is to shift theories and models of language acquisition away from monomodal monolingual perspectives toward multimodal multilingual ones. It is high time.

1 Signed Languages

Signed languages are natural human languages used by communities of deaf people as their native or primary language. “Sign language” is a broad category requiring clarification in two ways. First, many linguists prefer the term “signed language” rather than “sign language.” The term “sign language” implies that “sign” itself is a language, leading to confusion between the way a language is produced, its modality or medium of expression, and the name of a language. The term “signed” is best used to describe the modality in which a language is produced, parallel to spoken and written. The terms speaking, writing, and signing describe the ways in which a language can be expressed: We speak a language (such as English), we write a language (such as Chinese), and we sign a language (Argentine Sign Language). To put it another way: “Speech” is not the name of a language (students do not take classes in “speech language,” they take classes in Spanish or Japanese). Likewise, “sign” is not the name of a language, although we persist in the pernicious tradition of using the term “sign language” in this way: We say that students are taking “sign language” classes and assume, incorrectly, that this names the language. American Sign Language (ASL) is the name of a signed language. In fact, ASL and other signed languages can be written; linguists and deaf community members have developed orthographies for this purpose Reference Stokoe, Casterline and CronebergStokoe, Casterline, & Croneberg, 1965). There are many signed languages: British Sign Language, Italian Sign Language, Japanese Sign Language, Iranian Sign Language, Taiwanese Sign Language, and so forth. Although their names contain the word “sign,” they are the names of particular languages in the general category of signed languages. Some deaf communities have adopted names for their languages that do not include the word “sign,” such as Auslan (Australian Sign Language), Libras (Língua Brasileira de Sinasi), and others. In fact, the word “sign” and its translation is not even used in some spoken languages to name the local signed language, but rather the equivalent of “gesture” – which makes understanding the relation between sign and gesture even more challenging in those languages, for example, Dutch Nederlandse Gebarentaal, German Deutsche Gebärdensprache, and Russian Russkij zhestovyj jazyk. In summary, many linguists believe it is more accurate to use the term “signed” for the set of all the world’s languages produced in the signed modality, just as we use the term “spoken” for the set of the world’s spoken languages and “written” for those languages with orthographies.

The second way that the term “sign language” – even if replaced with the term “signed language” – must be clarified is that the term often includes signed languages such as American Sign Language, British Sign Language, Chinese Sign Language, and other natural signed languages which are used by large communities; village sign languages which arise when a number of deaf children are born into an insular indigenous community, such as San Juan Quiahije Chatino Sign Language (Reference Mesh and HouMesh & Hou, 2018); newly emerging languages such as Nicaraguan Sign Language (Reference Senghas and CoppolaSenghas & Coppola, 2001), and Al-Sayyid Bedouin Sign Language (Reference Meir, Sandler, Padden, Aronoff, Marschark and SpencerMeir, Sandler, Padden, & Aronoff, 2010); and International Sign (IS), an emerging pidgin that has arisen among signers from different language communities primarily in Europe and is often used as a lingua franca at international conferences (Reference WhynotWhynot, 2016). As one might expect, these different categories of signed languages have different histories, sociolinguistic characteristics, and potentially quite different stages of development, conventionalization, and recognition in education. Signed languages in developed countries with large communities of signers often have become accepted in educational settings (although, as we will see, historically this has not always been the case). In these situations the signed language may be learned by deaf children as a second language in school. Because the school systems in urban communities can draw from a larger population, deaf children from hearing parents often come into contact with deaf children with deaf parents who have learned the local signed language as their first language. However, even in otherwise developed countries such as Japan or China, lesser studied signed languages often are not adopted by educational systems. Village signed languages often occur in naturally isolated settings and smaller communities. As a result, they are sometimes shared with hearing people living in the same community who come into contact with deaf people. Because of the setting, the probability of acquiring more deaf users is relatively low for these signed languages. If deaf children leave these communities to attend school, they often abandon their village signed language and acquire the urban signed language.

A number of myths and misunderstandings have pervaded our understanding of the first class of signed languages, the natural signed languages of large communities of deaf people. One pervasive misunderstanding, held throughout much of history, is that signed languages are merely depictive gestures and not linguistically structured. Signed languages are not simply holistic gestures. There is, however, a complex relationship between signed languages and gesture that scholars are only now beginning to understand (Reference WilcoxWilcox, 2004, Reference Wilcox, Pizzuto, Pietrandrea and Simone2007, Reference Wilcox2009). This relationship will be discussed in Section 3.

Another common misunderstanding is that signed languages are merely representations of spoken languages – that ASL, for example, is a signed representation of spoken English. Signed languages are independent languages with their own lexicons and grammars. Related to this misconception, many people believe that signed languages are invented languages. They are not. Signed languages, like spoken languages, are naturally developing human languages. There are, however, sign systems created to represent spoken/written language, such as Signing Essential English and Signing Exact English.

Following from the belief that signed languages are invented is the assumption that they are languages with a shallow historical depth. The full story of the age of signed languages is quite complex and depends on the language, the question of emergent village signed languages, and the region of the world in which the language is used. Since signed languages were not regarded as true languages for most of recorded history, it is quite difficult to ascertain their age. We know that signed languages are mentioned in Talmudic law, and in the writings of Aristotle, Quintilian, and many others. In his lessons, for example, Quintilian, taught that not only a movement of the hand, but even a nod, may express our meaning, and he noted that such gestures are used by deaf people instead of speech. Deaf people and signed language are also mentioned in ancient Egyptian writings from the 19th Dynasty, ca. 1350–1200 BCE (Reference ErmanErman, 1971).

We do have a few historical accounts of signed language communities, such as that provided by Pierre Desloges in 1779, as reported in Reference Lane, Lane and GrosjeanLane (1980, pp. 123–124). Desloges became deaf at the age of seven from smallpox. As an adult, he wrote a treatise describing how he learned to sign:

Although very little can be definitely claimed about the history of signed language over the course of centuries, we can certainly say that as long as communities of deaf people have existed, they have used signed languages to communicate with each other.

2 Relation of Sign and Gesture: Historical Background

Our understanding of the relationship between signed languages and gesture has been the subject of debate among scholars of language and philosophers for centuries. One way in which this question has been manifest is in the centuries-long debate about language origins. (See more on this topic in the chapter by Żywiczyński and Zlatev, this volume.) The philosopher Étienne Bonnot de Condillac, for example, suggested that language began as a gesture language or langage d’action. The term “language of action” was even later used to describe signed languages. Jean-Jacques Rousseau, Johann Herder, Wilhelm von Humboldt, and other philosophers of language caricatured and ridiculed this position, arguing instead that language could not have arisen from such natural, animalistic beginnings. Herder, for example, proposed that the fundamental linguistic act was naming; critically, this naming was based on an unemotional sense of curiosity, a desire for pure knowledge. Furthermore, Herder argued that the naming had to have been audible, unaccompanied by any visible movement. The reasoning behind this argument was that “when a man is under the influence of an emotion (such as fear of an enemy) and yet suppresses, from rational grounds, any movement which might reveal it, he is acting from reason, not from passion” (Reference WellsWells, 1987, p. 40). Language thus is essential to reason – “without language man has no reason, and without reason no language” (ibid.). From this perspective, the best evidence for reason is that we ignore our body, our senses, our emotions, and our passions.

Thus, the accepted wisdom was that true language is spoken. Signs were regarded as nothing more than natural gestures evoked by emotion rather than reason. This view came clearly into view during the Milan Conference of 1880. At this time a great debate was taking place between educators who supported the use of signing in the education of deaf children and those who supported speech, the so-called oral method. Supporters of speech, such as Marius Magnat, the director of an oral school in Geneva, maintained that signed languages lacked any features of language and thus were not suited for educating deaf children (Reference LaneLane, 1984, pp. 387–388):

Statements made by Giulio Tarra, the president of the Milan conference, reveal even more starkly the confusion that equated speech with language, and sign with gesture.

The debate between those who supported signed language in the education of the deaf, and those who argued that only speech should be used, provides us with a clear view of how they considered the relationship between sign and gesture. It is no surprise that the relationship was framed in terms of mind–body dualism as espoused by one of the leading philosophers of the time, René Descartes. Summarizing the descriptions of sign and speech from Magnat and Tarra, we see that language is of the mind; it is associated with speech, with the acquisition of ideas and the expression of thought; it elicits reasoning, reflection, abstraction, generalization, and rationality. Speech has precision (perhaps meaning it has grammar); its users exhibit calm, prudence, and truth, such that it humanizes its users. Speech is of the soul, the spirit, because it originated from the breath, the aspiration, of God. Sign, on the other hand, is of the body; it is merely gesture. Signs corrupt deaf children, making them less human, more animalistic. Signs are concrete, and thus they cannot engender thought. Signs lack parts of speech and grammar. Because signs are gesture, they are associated with the corporeal body and with the senses. Signs strike the senses materially. They foment the passions – they are, in both senses of the term, sensual. Signs glorify fantasy and imagination. It is no accident that the root of imagination is image, and the iconic nature of signs, the antithesis of abstraction, is considered to be one of their most damning features. Whereas speech originated as the breath by which God gave humans a soul, sign is of the corruptible body, the flesh, and the material world.

The historical significance for signed language linguistics was tremendous, because it set the worldview for nearly 100 years. Scholars were left with the following assumptions firmly entrenched in our understanding of language, speech, gesture, and sign:

• Speech is of the mind; signs are of the body.

• Language is equivalent to speech; gesture is not language.

• Signs are gesture and therefore are not language.

In one form or another, these assumptions persist today. Much of the early work of sign linguists was motivated by a perceived need to distinguish sign from gesture. Linguists sought to demonstrate, for example, that signed languages have the same design features that spoken languages have (Reference Hockett and WangHockett, 1982). The pioneering work of William C. Stokoe Reference Stokoe(1960) was directed at demonstrating that ASL exhibits duality of patterning – that is, that it has a level equivalent to that of the phonology of spoken languages, with meaningless units that he called cheremes by analogy with phonemes that are combined to form meaningful units, or morphemes. Stokoe initially identified three classes of cheremes: hand shapes, movements, and locations. He later simplified the analysis to two classes – that which acts and its action (Reference StokoeStokoe, 1980).

Other linguists worked to document the complex grammar of signed languages (Reference Klima and BellugiKlima & Bellugi, 1979; Reference SipleSiple, 1978) and the nature of iconicity (Reference Engberg-Pedersen, Michael, Harder, Heltoft and JakobsenEngberg-Pedersen, 1996; Reference FrishbergFrishberg, 1975; Reference Mandel and FriedmanMandel, 1977). Sociolinguists documented the historical relation of ASL to French Sign Language (Reference WoodwardWoodward, 1976b, Reference Woodward and Siple1978) and sociolinguistic characteristics of the deaf community (Reference LucasLucas, 1989; Reference WoodwardWoodward, 1974, Reference Woodward1976a).

3 Sign and Gesture in Acquisition

A substantial body of research has investigated the relation between sign and gesture in first language acquisition. One area of research has examined deaf children who have no exposure to a signed language either at school or in the home. These children often develop a system of idiosyncratic gestures to communicate with parents or siblings (Reference MorfordMorford, 2003). These gestures exhibit many of the same properties seen in signed or spoken languages. Homesigners develop systematic ways of indicating negation and questions. Specific hand shapes and movements become associated with specific meanings. Homesigners use these gestures in a consistent way across settings, rather than creating new gestures for each new setting. Homesigners also use gestures to refer to generic entities and events and not just specific instances. However, homesign also displays important differences when compared to more conventional spoken or signed languages. Homesigners do not appear to ever use complex syntactic structure. Homesign also does not seem to develop phonological structure that is independent of morphological structure (Reference Morford and Hänel‐FaulhaberMorford & Hänel‐Faulhaber, 2011).

Research on homesign appears to contradict strong claims that language cannot be acquired after a critical period. While this may be true for spoken languages, deaf children who use homesign do acquire signed language once they are exposed to signing deaf adults (Reference MorfordMorford, 2003). However, as adults, these signers display deficits when compared to other members of the signed language community. Reference Morford and Hänel‐FaulhaberMorford and Hänel-Faulhaber (2011) propose two possible explanations. One explanation focuses on the differences between homesign and conventional languages, noting that one area of deficit is in acquisition of more native-like phonological structure and complex syntax. The second explanation points out that homesign is not acquired in a shared language community, and so homesigners have reduced exposure to receptive language. As a result, late learners also exhibit receptive processing deficits such as slower sign recognition (Reference Morford and Hänel‐FaulhaberMorford & Hänel-Faulhaber, 2011).

A second area compares the acquisition of gesture in hearing children and sign in deaf children. A broad summary of the results suggests links between early actions, gestures and words, and the importance of multimodal communication and the interplay between gestures and spoken words (Reference Volterra, Capirci, Rinaldi and SparaciVolterra, Capirci, Rinaldi, & Sparaci, 2018). Early work examined the role of gestural performatives and representational gestures (Reference Bates, Camaioni and VolterraBates, Camaioni, & Volterra, 1975). Performatives include ritualized requests, showing off, showing, giving, and pointing. Performatives were typically classified as deictic gestures, the content of which can only be interpreted referring to the extralinguistic context. The second type of gestures, termed representational gestures, are used by children to refer to objects, persons, locations, or events through hand, body, or facial movements (Reference Capirci, Iverson, Montanari and VolterraCapirci, Iverson, Montanari, & Volterra, 2002). Representational gestures differ from deictic gestures by iconically representing attributes or actions of specific referents. The meaning of representational gestures does not change across different contexts.

Early development of these two types of gesture, and of signs, were often attributed to different underlying cognitive systems, one gestural and the other linguistic. A more recent conclusion suggests “not a clear-cut separation, but a continuity between co-speech gestures produced by hearing children and early signs produced by children exposed to a sign language” (Reference Volterra, Capirci, Rinaldi and SparaciVolterra, Capirci, Rinaldi, & Sparaci, 2018, p. 217). These researchers conclude that the traditional dichotomy between gestures as gradient, variable, iconic, and signs as categorical, invariable, arbitrary, should be replaced with a multimodal approach to the study of both spoken and signed languages. This conclusion mirrors recent research questioning any clear-cut distinction between gestural and linguistic systems (Reference TalmyTalmy, 2018; Reference Wilcox and MartínezWilcox & Martínez, 2020; Reference Wilcox and OcchinoWilcox & Occhino, 2016a).

4 The Relation of Sign and Gesture: Current Views

As is so often the case in the history of ideas, the pendulum of science swings between two poles. Space and time were seen as distinct by Newton; Einstein merged them into space-time. Physicists now debate whether time exists at all. Such is also the case with sign and gesture. As we have seen, for centuries signed languages were considered to be gesture and not language. The early work of sign linguists was directed at demonstrating that signed languages are languages and not gesture. In recent years, there has emerged a position which holds that signed languages are integrations of linguistic and non-linguistic or gestural systems. Finally, there are even those who are beginning to question the usefulness of the term “gesture” itself.

One contribution to the paradigm shift that permitted sign and gesture to be reexamined was the publication of Gesture and the Nature of Language (Reference Armstrong, Stokoe and WilcoxArmstrong, Stokoe, & Wilcox, 1995), which made the case that scholars could examine gesture and language as related phenomena. The authors did not simply claim that gesture and signed language might be related. Rather, they put forward the hypothesis that all language is gestural, and that the origins of human language can be traced to visible gestures. The ideas in this were based in part on an understanding of gesture as a functional unit, an equivalence class of coordinated movements that achieve some end (Reference Studdert Kennedy and AllportStuddert Kennedy, 1987). Gesture was seen more broadly as articulatory movements of any part of the body. This concept was derived from the work of the cognitive psychologist Ulrich Reference NeisserNeisser (1967, pp. 156, 161):

The ideas presented in Gesture and the Nature of Language were also informed by cognitive linguistics, and in fact the work was instrumental in bringing together a nexus of gesture-language-sign research informed by cognitive linguistics. Cognitive linguistics dramatically changes the perspective of how we view language, and it does so in a way that allows linguists to learn from gesture researchers, research on animal communication, and the study of general perceptual and cognitive abilities, options that were precluded under prior theories.

Although linguists have demonstrated that signed languages are not simply unanalyzable depictive gestures, recent research has begun to explore the relation between signed language and gesture. This work can be classified into three major themes: (1) the historical development by which gestures used in a hearing community are incorporated into a signed language, thus becoming part of the sign linguistic system; (2) the claim that some signs and sign constructions are fusions of linguistic and gestural material, so-called “sign-gesture fusions”; and (3) an analysis of signs and gestures based on cognitive linguistic theory, and in particular, the theory of cognitive grammar (see below).

The first approach is based on grammaticalization theory (Reference BybeeBybee, 2006; Reference Hopper and TraugottHopper & Traugott, 2003). Grammaticalization is the process by which lexical material becomes grammatical material. Working within this framework, linguists have demonstrated that some of the gestures used in the surrounding language community are incorporated into a signed language as lexical signs. These lexical signs then grammaticize, forming grammatical signs. Grammaticalization of gesture and signs has been described for a number of signed languages Reference Pfau, Steinbach, Heine and NarrogPfau & Steinbach, 2011; Reference Shaffer, Jarque, Wilcox, Nogueira and Lopes.Shaffer, Jarque, & Wilcox, 2011; Reference Wilcox, Rossini, Antinoro Pizzuto and BrentariWilcox, Rossini, & Antinoro Pizzuto, 2010; Reference Xavier and WilcoxXavier & Wilcox, 2014).

An example occurs with the gesture meaning “to leave or depart” (Figure 16.1) common in the Mediterranean region (Reference Morris, Collett, Marsh and O’ShaughnessyMorris, Collett, Marsh, & O’Shaughnessy, 1979). This gesture appears to have been incorporated into old French Sign Language (LSF) as the sign PARTIR “to depart.” ASL is genetically related to French Sign Language (LSF), LSF having been a historical source for ASL since the early 1800s (Reference Wilcox and OcchinoWilcox & Occhino, 2016b; Reference Woodward and SipleWoodward, 1978). An old ASL sign derived from the old LSF sign PARTIR appears in films of older signers from 1913 with both lexical and grammatical meanings. The lexical sign DEPART is used in utterances that can be translated from ASL to English as, “At that time, the president of Gallaudet, Edward Miner Gallaudet, departed. A few days prior, he departed/left (to go) to Philadelphia.” We also find the sign with slightly reduced movement used to mark future (Reference Janzen, Shaffer, Meier, Quinto and CormierJanzen & Shaffer, 2002). In the same series of 1913 films, in giving a lay sermon, a deaf person uses the sign when saying (again translating this from ASL to English), “When you understand the words of our Father, you will do that no more.”

Figure 16.1 Depart gesture

(by permission of Advanced Reasoning Forum)

Reference Wilcox, Wilcox, Bybee and FleischmanWilcox and Wilcox (1995) demonstrated that gestural forms often serve as the source for lexical signs, which then grammaticalize to modals. The ASL modal sign CAN, for example, is derived from the ASL lexical form STRONG. Reference LongLong (1918) pointed out that the sign for “strong” is very similar to CAN, noting that the difference lies in the way the hands are moved. For “strength” they are moved somewhat sidewise with a slight circular motion. The source for the lexical sign STRONG is a gesture indicating strength, commonly expressed by moving the fists in an outward or downward motion so as to indicate upper body strength.

An example demonstrating a longer diachronic and cross-linguistic chain occurs with the modal meaning “necessity” (Figure 16.2). The notion of necessity is expressed by the ASL modal form glossed as MUST. The history of this form is more complex, appearing in modern ASL as a downward moving bent index finger. In LSF the form IL FAUT “it is necessary” is similar, except the index finger is straight and the palm orientation is to the side rather than down. In LSF from the nineteenth century, the straight index finger is used but the entire hand points down. Ultimately, the form appears to have had as its gestural source a downward pointing finger. This gesture was used in classical antiquity to indicate “in this place” and “insistence.” Reference DodwellDodwell (2000, p. 36) discusses this gesture from ancient Roman times, which he calls an imperative, “consists of directing the extended index finger towards the ground.” The gesture was ascribed a modal sense by Quintilian, who noted that “when directed towards the ground, this finger insists” (ibid.). Insistence is semantically related to the modal notion of necessity.

Figure 16.2 LSF sign IL FAUT and ASL modal sign MUST

(LSF image with permission from IVT-International Visual Theatre)

Reference WilcoxWilcox (2004, 2009) proposed two routes by which gesture is incorporated into a signed language. The first route, described above, begins when manual gesture enters a signed language as a lexical sign and develops through grammaticalization into a grammatical morpheme. The second route proceeds along a distinctly different path. The source is not the manual gesture itself; rather, it is the way that a manual gesture is produced, the sign’s manner of movement, as well as various facial, mouth, and eye gestures that may accompany a manual gesture or sign. Here, too, we see gesture as a source. For example, Quintilian observed that when the hand is thrown out gently it promises and declares assent; when it is moved more quickly, it is a gesture of exhortation or sometimes of praise.

Upon entering the linguistic system, these manner of movement and facial gestures follow a developmental path from paralinguistic (prosody or intonation) to grammatical marker. As an example of the grammaticalization of manner of movement, in many signed languages the manner in which a modal sign is produced indicates strength of the modal. Reference JarqueJarque (2006) notes that manner of movement is used to indicate differences in modal strength and to mark deontic versus epistemic function in Catalan Sign Language. Modal strength is also marked in Italian Sign Language by weak or strong articulation of the base movement.

Manner of movement also appears as a marker of verb aspect. Reference Pizzuto and VolterraPizzuto (1987) observed that temporal aspect can be expressed in Italian Sign Language via systematic alterations of the verb’s movement pattern, specifying, for instance, the “suddenness” of an action by means of a tense, fast, short movement (e.g. the distinction between to meet and to suddenly/unexpectedly meet someone). Conversely, a verb produced using an elongated, elliptical, large and slow movement specifies that an action is “repeated over and over in time” or “takes place repeatedly in time (e.g. to constantly telephone or to always be on the telephone). Similarly, in ASL, manner of movement marks verb aspect (Reference Klima and BellugiKlima & Bellugi, 1979).

The second route also characterizes the grammaticalization of facial displays. Facial displays play a significant role cross-linguistically in signed languages. In addition to expressing emotion, facial gestures mark a variety of grammatical functions such as interrogatives, topics, adverbials, conditionals, imperatives, and more. These facial displays often begin as gestural expressions. For example, brow furrow is well documented as a display of physical or mental exertion. Reference DarwinDarwin (1872, p. 221) noted that brow furrow marks “the perception of something difficult or disagreeable, either in thought or action.” Brow furrow marks a number of grammatical meanings across several signed languages, including wh-questions, imperatives, and root or deontic modality.

Recently, some sign linguists have proposed that certain signs are not purely linguistic; rather, they claim that these signs and sign constructions are better described as “language-gesture fusions” (Reference Fenlon, Cooperrider, Keane, Brentari and Goldin-MeadowFenlon, Cooperrider, Keane, Brentari, & Goldin-Meadow, 2019; Reference Hodge and JohnstonHodge & Johnston, 2014). One such claim concerns pointing signs. Pointing signs are quite common across signed languages, functioning as deictic and anaphoric pronouns, possessive and reflexive pronouns, demonstratives, locatives, body part signs, and indicating verb agreement. The gesture-language fusion claim is most clearly made in the case of personal pronouns. Reference Meier and Lillo-MartinMeier and Lillo-Martin (2013) observe that the first-person pronoun in ASL is fully specified phonologically: a point to the center of the signer’s chest. However, they claim that the locations to which non-first person signs point cannot be enumerated in a listing of sublexical phonological units: Signers point to an open-ended set of locations. Since location is a phonological prime, having an open-ended phonological inventory of locations is not possible. Thus, they conclude (Reference Meier and Lillo-MartinMeier & Lillo-Martin, 2013, p. 163) that “first-person points, but not non-first points, can be specified entirely in terms of the phonological units that form lexical signs. […] In contrast the locations in space of non-first person points appear to be gestural inasmuch as the direction of pointing is – when the signer is referring to an individual who is present at the conversation – determined by the referent’s physical location in the environment.” This same argument is extended to the marking of verb agreement, in which agreement is marked by location in space.

The gesture-fusion claim requires that linguists identify criteria by which gesture can be reliably and objectively distinguished from language. One proposal that has been offered (Reference SandlerSandler, 2009) is that gestures are holistic and synthetic; they are lacking in hierarchical and combinatoric properties; they are idiosyncratic – different speakers or even the same speaker may use different gestures to represent the same image; and gestures are context-sensitive, their interpretation dependent on the linguistic context. Linguistic structure, on the other hand, is componential, combinatoric, and hierarchically organized. Linguistic signs such as words are highly conventionalized in form, meaning, and distribution. These criteria, however, pose problems. Language is both conventional and unconventional – innovation by definition establishes new, unconventional expressions. So is gesture: Emblems or recurrent gestures are conventional, while idiosyncratic or innovative gestures are, by definition, not conventional. Language is gradient. As Reference BybeeBybee (2010, p. 2) observes, “All types of units proposed by linguists show gradience, in the sense that there is a lot of variation within the domain of the unit (different types of words, morphemes, syllables) and difficulty in setting the boundaries of the unit.”

Another option has been to propose a modality-free definition of gesture (Reference Okrent, Meier, Cormier and Quinto-PozosOkrent, 2002), based on degree of conventionalization (How conventionalized must something be in order to be considered linguistic?), site of conventionalization (What kinds of conventions are linguistic conventions?), and restrictions on combination (What kinds of conditions on the combination of semiotic codes are linguistic conditions?). Answers to these questions are, however, not offered, nor is it clear how they could be objectively answered. For these and other reasons, not all sign linguists accept the language-gesture fusion claim (Reference QuerQuer, 2011; Reference WilburWilbur, 2013). Reference DotterDotter (2018) presented a strong rejection of the claim that gestural components are combined with language elements in essential areas of signed language grammar.

As we have seen, the relation between gesture and sign has a complex history, informed by different perspectives on sign itself and on gesture. Reference MüllerMüller (2018) offers a comprehensive overview of the history of gesture studies and sign linguistics with special attention to examining the relation between gesture and sign. First, Müller reconstructs the history of gesture studies, focusing on the seminal work of Kendon, McNeill, and Goldin-Meadow. She traces Kendon’s view that gesture and sign are one gestural medium of expression, or utterance visible actions, detailing the functional similarities between gesture and sign. Müller points out that McNeill, on the other hand, claims that gesture and sign exhibit sharp discontinuities. Müller attributes this position to McNeill’s decision to restrict the concept of gesture to spontaneously used gestures. A third position described by Müller is that presented by Reference Goldin-Meadow and BrentariGoldin-Meadow and Brentari (2017), in which they strengthen the discontinuity view, framing the distinction such that the essential features distinguishing the two are that sign and speech are categorical, while gesture is imagistic. Müller rejects the assumptions that gestural equals imagistic and that there is a clear-cut boundary between categorical and gestural. Müller concludes by suggesting the need to view gesture and sign dynamically, both in historical relation and in multimodal interaction. Although Müller essentially aligns with Kendon’s continuity position, she rejects Kendon’s proposal of the term “utterance visible actions,” preferring instead to retain the term “gesture,” advocating an understanding of gestures as “deliberate expressive movements.” Ultimately, she suggests that “the question of how gesture and sign relate critically depends on the notion of ‘gesture’ employed” (Reference MüllerMüller, 2018, p. 16). Concerning the historical relation between sign and gesture, Müller concludes that research such as that discussed above suggests a “dynamic, continuous and ongoing process of historical change, where no cataclysmic break is involved, and no sudden rupture transforms gesture into sign from one moment to another” (Reference MüllerMüller, 2018).

Recently, Wilcox and his colleagues (Reference Wilcox and OcchinoWilcox & Occhino, 2016a) have proposed an account of pointing that does not require evoking gesture. Their approach uses a dynamic usage-based model, specifically cognitive grammar (Reference LangackerLangacker, 1987, Reference Langacker, Barlow and Kemmer2000, Reference Langacker2008). Cognitive grammar claims that grammar and lexicon form a continuum of symbolic assemblies composed of phonological structures, semantic structures, and the symbolic links between the two. Phonological, semantic, and symbolic structures are abstracted from usage events: “instances of language use in all their complexity and specificity” (Reference LangackerLangacker, 2008, p. 547). Symbolic assemblies vary along two dimensions: schematicity and complexity. Schematicity pertains to level of detail or precision. Schematic elements are elaborated or instantiated by more specific elements. Symbolic structures combine to form higher-level symbolic structures, or symbolic assemblies. Through repeated combination, symbolic assemblies of high complexity may be formed.

In this analysis, pointing is a complex symbolic assembly, a construction consisting of two component structures: a pointing device and a Place (Figure 16.3). Both of these are symbolic structures: They consist of a semantic pole and a phonological pole. The pointing device serves to direct attention; this is its schematic meaning. The schematic phonological pole of a pointing device is any articulator capable of directing attention. This may be a pointing finger, but eye gaze and even body torso orientation may serve as the phonological pole of a pointing device. The Place (the term appears with initial capital letter to indicate that it names the entire symbolic structure) is the entity at which attention is directed. The schematic semantic pole of a Place structure, which must be specified in an actually occurring usage event, is the referent. The schematic phonological pole of a Place structure is a location in the spatial surroundings. Places appear in a number of constructions, including both deictic and anaphoric expressions (Reference Wilcox and OcchinoWilcox & Occhino, 2016a), as well as reported dialogue and so-called “agreement” constructions (Reference Wilcox, Martínez, Morales, Jucker and HausendorfWilcox, Martínez, & Morales, 2022). Place structures thus unify two distinct types of phonological locations: those internal to language, and those external locations which in other analyses are regarded as gestural. Places also subsume the traditional distinction between deixis and anaphor. Reference TalmyTalmy (2018, p. 1) writes that, “Broadly, an anaphoric referent is an element of the current discourse, whereas a deictic referent is outside the discourse in the spatiotemporal surroundings. This is a distinction made between the lexical and the physical, one that has traditionally led to distinct theoretical treatments of the corresponding referents.” Talmy offers an account in which language engages the same cognitive system for both speech-internal and speech-external referents. Places provide the comparable symbolic resource for languages produced in visual space. For signed languages, a referent at a location in the spatiotemporal surroundings is not outside of the discourse; rather, both deictic and discourse referents occupy phonological locations in the spatiotemporal surroundings.

Figure 16.3 Pointing construction

Reference Ruth-Hirrel and WilcoxRuth-Hirrel and Wilcox (2018) apply the pointing device and Place analysis to speech-gesture constructions. They focus primarily on complex symbolic assemblies consisting of pointing constructions and beats that accompany speech. Beat gestures have been formally characterized as “biphasic movements of the hands” (Reference Biau and Soto-FaracoBiau & Soto-Faraco, 2015). These movements typically involve a “simple flick of the hand or finger up and down, or back and forth” (Reference McNeillMcNeill, 1992, p. 15); however, beats may also be performed using other body parts, such as the head or eyebrows (Reference Krahmer and SwertsKrahmer & Swerts, 2007). Researchers often claim that beats have no semantic content (Reference Alibali, Heath and MyersAlibali, Heath, & Myers, 2001; Reference Biau and Soto-FaracoBiau & Soto-Faraco, 2013; Reference Özçalışkan and Goldin-MeadowÖzçalışkan & Goldin-Meadow, 2009). Researchers acknowledge that beats serve emphatic functions and are closely tied to information structure, comparing beats to an “all-purpose highlighter” superimposed on more objective content (Reference McNeill, Levy, Duncan, Deborah, Heidi and DeborahMcNeill, Levy, & Duncan, 2015).

In the cognitive grammar analysis proposed by Ruth-Hirrel and Wilcox, beats are symbolic structures; thus, they have both phonological and semantic import. Reference LangackerLangacker (2001) expands on the notion of symbolic structures, noting that such structures incorporate multiple channels. The semantic pole consists of several conceptualization channels, including speech management, information structure, and objective content. Speech management includes such functions as holding the floor and turn taking. Information structure includes emphasis, discourse topic, and given versus new information. Objective content is the conceptualization of the situation being described by a linguistic expression. The phonological pole consists of several vocalization channels. The core vocalization channel for speech is segmental content. Other channels include intonation and gesture.

Ruth-Hirrel and Wilcox claim that beat gestures are symbolic structures, but significantly they are phonologically and conceptually dependent structures, requiring autonomous structures for their expression. Semantically, beats are dependent structures in the information structure channel, making reference to some more objective autonomous content, the information that is emphasized or highlighted. Phonologically, beats are expressed as manner of movement, requiring an autonomous gesture carrier for their articulatory expression. This is canonically specified by the movement of a hand, since manner of movement is dependent on movement (there is a further level of dependency, because movement requires some entity such as a hand). It is also possible for the movement of any more substantive and autonomous structure, such as a head, to serve as the phonological carrier for a beat.

Ruth-Hirrel and Wilcox show that simple beat gestures, as well as beat gestures coexpressed with pointing gestures, are used to direct attention to meanings in speech that are associated with salient components of stancetaking acts. Their account reveals both that beats have meaning and the symbolic motivation for the apparent “superimposing” of beats onto pointing gestures and their integration with speech.

5 Sign and Gesture Revisited

One result of the contemporary linguistic analyses of signed languages and gesture has been a blurring, or indeed the loss, of any categorical distinction between sign and gesture. Noting these difficult issues, Reference KendonKendon (2017, p. 30) concluded that “‘gesture’ is so muddied with ambiguity, and theoretical and ideological baggage, that its use in scientific discourse impedes our ability to think clearly about how kinesic resources are used in utterance production and interferes with clarity when comparing signers and speakers.” Kendon has gone so far as to propose abandoning the categories gesture and sign altogether, instead focusing on a comparative semiotics of what he terms visible bodily action as it is used in utterances by speakers and by signers.

The dynamic usage-based approach offers a new way to reframe our understanding of sign and gesture (Reference Occhino and WilcoxOcchino & Wilcox, 2017; Reference Wilcox and OcchinoWilcox & Occhino, 2016a). The world comes to us unlabeled. We perceive not “sign” or “gesture” but perceptible usage events – Kendon’s visible bodily actions or Neisser’s articulatory events. These actions, as perceptual events, are categorized by language learners. In the present case of sign and gesture, we can restrict our focus to deaf language learners. Reference StokoeStokoe (1960, pp. 6–7) adopted this user-centered deaf viewpoint from the very start:

Having both a perceptible form and a meaning (in cognitive grammar parlance, we would say having a phonological and a semantic pole), the shoulder shrug poses a categorization problem to be solved by the deaf observer: How does this symbolic structure fit into his emerging and dynamic understanding of communicative performance? The units that compose an individual’s linguistic knowledge (i.e. grammar) are related to actual expressions that are perceived in usage events by the process of categorization. In the theory of cognitive grammar, the process works in this way (Reference Langacker, Barlow and KemmerLangacker, 2000). A particular target of categorization activates a variety of established units, the activation set. For any given usage event, taking into consideration all the linguistic and contextual factors, the language user must search the activation set for the member that will categorize the target. Members of the activation set must in a sense compete; the winner becomes the most active member of the set and the active structure which categorizes the target. A number of factors determine which member will become the active structure. One factor is degree of entrenchment of the member, which influences its inherent likelihood of activation and thus of being selected. A second is contextual priming, both phonological and semantic. A third is amount of (phonological or semantic) overlap between the target and a potential categorizing structure. These three factors are primarily linguistic, but we must also include many other factors in order to fully understand the categorization process (Reference Wilcox and OcchinoWilcox & Occhino, 2016a).

In the current case we must include such factors as individual variability (age of exposure to signed language, level of hearing loss); social variability (whether the observer comes from a deaf or hearing family, type of education, accessibility of signed language in the general environment); and cultural variability (society’s attitude toward sign and gesture in general, for example, Navajo culture and Neapolitan culture exhibit very different attitudes toward gesturing; the categories provided by a culture for naming such perceptual events). As Reference Van Hoekvan Hoek (1997) notes, these categorizing judgements determine if the construction is an instantiation of a particular schema or an extension from that schema. With a small amount of conflict, the construction may be judged to be an acceptable innovation, but a significant conflict will cause signers to judge the construction to be anomalous, that is, not a part of their grammar. Thus, the process of categorizing a target structure, be it a shoulder shrug, a facial display, or any other visible bodily action, is none other than the linguistic process of judging whether a perceived structure is well formed with respect to others, that is, it is a part of their dynamically changing grammar.

In all cases, the key to answering the question “Is it a sign or a gesture?” in the context of visibly perceptible usage events lies with the observer, not the observed. It requires that we stop assuming that these events form natural categories – that is, categories that exist in nature, independent of language users. Instead we must reframe the question and adopt an approach which acknowledges that deaf observers categorize perceptual events.

Clifford Reference GeertzGeertz (1973, p. 6) offered an example of how a visible perceptual usage event is categorized:

In the context of understanding sign and gesture, we might rephrase Geertz and say: the same speck of behavior with a fleck of cultural, contextual, and background knowledge and the act of categorization by the deaf observer, and – voilà! – sign (in this case, a grammatical facial display). Like Stokoe’s shrug and Geertz’s wink, the visible bodily actions of usage events are the very stuff from which language is made. The labeling of visible bodily actions as sign or gesture is, as Geertz would say, a matter of determining what counts as what. Language, gesture, and sign are historical-cultural constructs, folk classifications that may or may not be relevant to deaf language users. The relevant question to be examined is the dynamically emerging knowledge – or as a linguist would call it, the grammar – of deaf language users. The key is to not forget the observer. Geertz is helpful even here: We must see things, he tells us (Reference Geertz1974), “from the native’s point of view” – in this case, from the point of view of the individual deaf language user observing and categorizing visible bodily actions. The categorization of these usage events is an individual user’s cognitive activity. The linguist’s task is to discover the user’s categories.

Adopting a user-based cognitive linguistic perspective may produce a paradigm shift in the study of signed language and gesture. As Geertz pointed out, “small facts speak to large issues, winks to epistemology” (Reference GeertzGeertz, 1973, p. 23). For linguists, the small facts are intricately complex usage events, the larger epistemological issue is the construction of a grammar. If linguists are to understand how language is constructed by users from usage events, we must begin by compiling thick descriptions of actual discourse usage events in all of their expressive and conceptual complexity. This demands that linguists incorporate cultural, social, and historical data into our linguistic theories.

Ultimately, the question of what is sign and what is gesture may not be a scientific but an ethnoscientific one. The answer lies not in finding observable, I-am-a-camera photographable differences between the “sign” and “gesture” independent of who is doing the observing and classifying; rather, it is a matter of what counts as sign or gesture from the deaf person’s point of view. This, in turn, depends on received folk classifications that are handed down and change over time, that vary across cultures and across the individuals doing the classification. Whether deaf and hearing people share the same folk classifications of what counts as sign and what counts as gesture – if indeed they even have such categories – is an open question, although research suggests that the answer is likely to be quite complex (Reference Kusters and SahasrabudheKusters & Sahasrabudhe, 2018). From winks and shrugs and facial displays to depictive shapes and movements of the hands; from whether a hand is moved gently or with a sudden, quick movement; from highly innovative signed constructions expressing an actor’s action in chopping down a tree and the tree’s personified emotional reaction to being chopped to highly conventional lexical signs – these are the photographable raw data. As linguists and gesture scholars, our task is to meticulously describe this input to the categorization process and explore how the process plays out, how grammars are dynamically constructed, grow, and change. Echoing Kendon, it appears to this sign linguist that the label gesture, with its historically laden ideological baggage, contributes little to the task at hand.

1 Introduction

When, how, and why do people gesture? The Cambridge Handbook of Gesture Studies offers many answers to this question from diverse theoretical and methodological perspectives on the relation between language and gesture. The topic introduced in this chapter pulls together several strands of research that have highlighted gesture’s relation to notions and processes that are traditionally seen as “grammatical.” In particular, rich observations on gesture’s link with negation have featured in the work of several key thinkers and texts, and can therefore be said to have played a role in shaping contemporary gesture studies. Rather than emphasizing the spontaneity and idiosyncrasy of co-speech gestures, for instance, studies of gesture’s association with negation have shed light on regularities in gesture form, function, and linguistic organization, and in turn, offered evidence for the multimodality of grammar, the embodiment of cognition, and our bodies’ “potential for language” (Reference Müller, Müller, Cienki, Fricke, Ladewig, McNeill and TeßendorfMüller, 2013, p. 202).

As any linguist knows, negation involves lexical and grammatical patterns that determine word order, operate on the semantics of the utterance, and play a role in the conceptualization and pragmatics of speech acts, such as rejections, disagreements, denials, objections, and rejections (Reference HornHorn, 1989). What is also well known in gesture studies is that central to these patterns is an array of gestures that speakers perform in association with their linguistic and pragmatic expression of negation (Reference CalbrisCalbris, 1990, Reference Calbris2011; Reference HarrisonHarrison, 2018; Reference KendonKendon, 2004; Reference Lapaire, Bonnefille and SalbayreLapaire, 2006a). The headshake might immediately come to mind (with its notorious cultural variations; Reference Harrison, Müller, Cienki, Fricke, McNeill and BressemHarrison, 2014a), as well as various gestures of the hands striking outwards from the body or towards the addressee in a holding motion.

Over the last ten to fifteen years, increasing attention has been paid to these gestures associated with negation.Footnote ¹ Their forms and functions characteristically recur among members of a given linguistic and cultural community in clearly identifiable discourse contexts with relatively stable meanings. However, these gestures are somewhat distinct from “emblems” or “quotable gestures” (as discussed in Payrató, this volume). They are often considered to be exemplars of the recurrent gesture category (Reference Harrison and LadewigHarrison & Ladewig, 2021; Reference Ladewig, Müller, Cienki, Fricke, Ladewig, McNeill and BressemLadewig, 2014, this volume). Studies of gestures associated with negation have arguably helped in paving the way for a reconceptualization of the nature of gesture and its relation to linguistic structures, as well as of our understanding of common ground between spoken and signed languages, the multimodality of language, and the embodiment of cognition.

With the wider issue of grammar–gesture relations in the background, the first task of this chapter is to chart the territory of gestures associated with negation: what is known about their forms, organizational properties, and functions related to linguistic negation (explicit and covert), discourse-pragmatics, and interaction, as discernible from their study in a wide variety of linguistic communities (Section 2). Then, the range of discourse domains, interactive contexts, and methodological perspectives in which studies of gestures associated with negation have been conducted will be considered (Section 3). The empirical and theoretical contributions that such studies have made can then be evaluated by reporting the uptake of relevant research findings across different areas of the linguistic, social, and cognitive sciences (Section 4). Throughout these sections, the aim is to not only show what has been discovered, but also to disclose areas that seem ripe for further development and which raise open empirical questions.

2 Gestures Associated with Negation

In their entry for the Stanford Encyclopedia of Philosophy, Reference Horn, Wansing and ZaltaHorn and Wansing (2020) write that “Negation is a sine qua non of every human language.” As a linguistic universal, all languages have grammatical forms and structures that express negation, the subtleties of which have been widely documented and debated in decades of linguistic, pragmatic, and psycholinguistic research (e.g. for an accessible introduction to English negation, see Reference Huddleston and PullumHuddleston & Pullum, 2005, Ch. 8; for Mandarin Chinese negation, see Reference Li and ThompsonLi & Thompson, 1989, Ch. 12). What had not been as closely studied until relatively recently were the forms, meanings, and structures of gestures that can be observed when people express negation in face-to-face spoken communication, and more specifically, the intricate ways in which linguistic and gestural forms and structures relate during the expression of negation. The current section aims to convey what is known about recurrent gestures associated with negation by reporting where in the world they have been observed so far, the typologies in the literature that have been established, and our understanding of the forms, functions, and organizational properties of these gestures.

2.1 Geographical Coverage

Focusing on spoken languages, the following coverage of gestures associated with negation compiles and abstracts from different kinds of research-typologies, descriptive studies of individual gestures, experiments, etc. – conducted along different themes and subjects in relation to particular languages (Table 17.1). This is also visualized with a Google map with pins to represent the location of studies or linguistic communities under study (Figure 17.1). The main criteria for inclusion in this overview was that researchers observed and discussed the relation between gesture and negation in the locations where the studies were conducted; distinctions concerning different forms and form variants of gestures/gesturing were set aside for the present purposes. Distinctions concerning regional uses of language have been made where research permits, such as in different varieties of Spanish. Several papers listed in this review are discussed in more detail in later sections of the chapter.

Figure 17.1 Geographical coverage of the attested relation between gesture and negation in spoken languages

Figure 17.1 is no doubt incomplete, being restricted to the studies published in the languages that I can read (or could confidently cross-reference) and those that I could locate. Several gestures described in A world guide to gestures by Reference MorrisMorris (1994) are associated with negation and either located geographically or described as having “widespread” locality. They include the Head Shake (Widespread; p. 144), Palm Thrust (Greece; p. 191), Palms Front (Worldwide; p. 195), Palms Wipe (Widespread; p. 198), and others. Similarly, Reference DarwinDarwin’s (1872) The expression of emotions in man and animals is a rich resource for culturally specific observations of various head and hand gestures associated with negation that Darwin noted on his voyages or in correspondence with peers (see Reference CooperriderCooperrider, 2019). However, work by Morris and Darwin could not be easily pinned down to the above map.

Disclaimers aside, Table 17.1 and Figure 17.1 combined reveal both a number of research hotspots and data deserts, which may be useful in revealing gaps in research for future studies. Observations of gestures associated with negation in locations not observed here will add incrementally to established findings and, moreover, hopefully extend or challenge the picture of these gestures that is emerging. It is this picture to which we can now turn.

3.2 Forms, Organizational Properties, and Functions of Gestures Associated with Negation

This section’s first port of call is the landmark work on gestures with pragmatic functions in Italian and English by Kendon and his studies on the Open Hand Prone gesture family (Reference KendonKendon, 1995, Reference Kendon2002, Reference Kendon2004, Reference Kendon2017), while research into the semiotics of French gesture by Calbris then introduces an alternative perspective on similar gesture forms (Reference CalbrisCalbris, 1990, Reference Calbris2003, Reference Calbris2005, Reference Calbris2011, Reference Calbris, Müller, Cienki, Fricke, Ladewig, McNeill and Teßendorf2013; Calbris & Copple, this volume). These landmark observations provide the departure point for various lines of research that have subsequently added to our understanding of gestures associated with negation.

3.2.1 Context-of-Use, Kinesic and Semantic Core, Underlying Action

In a highly influential study by Reference KendonKendon (2004), associations between gesture and the expression of negation were observed as part of characterizing the use of recurrent gestural forms in the Open Hand Prone family. In this family of forms, “the forearm is always in a prone position so that the palm of the hand faces either toward the ground [‘ZP’ gestures] or away from the speaker [‘VP’ gestures], depending upon how the elbow is bent” (p. 248). Analyzing examples of these gestures in a video corpus of Italian speakers (supplemented with several examples of English speakers), Reference KendonKendon (2004) observed that gestures in the Open Hand Prone family occurred in discursive contexts “where something is being denied, negated, interrupted or stopped, whether explicitly or by implication” (p. 248), as well as “in contexts where a speaker gives an extreme positive evaluation of something” (p. 249). For each example, Kendon scrutinized the specific form of the gestures and their timing in relation to the verbal utterance, and analyzed their potential semantic and pragmatic contributions to the discursive context-of-use (on this methodology; Reference KendonKendon, 2004, p. 226; Reference Müller, Posner and MüllerMüller, 2004).

Reference KendonKendon’s (2004) findings led him to argue that the formational core of gestures in the Open Hand Prone family expresses a “core semantic theme” of “halting, interrupting, or indicating the interruption of a line of action” (p. 281). This theme, he proposed, was derived from the manipulatory action of the hands that motivates the core form of the gesture (see work by Calbris, Streeck, and Müller below). Gestures with the palm oriented downwards and “swept” laterally (ZP gestures) “perhaps derive from the action of cutting something through, knocking something away or sweeping away irregularities on a surface, as in rubbing out any marks or traces of something,” whereas for gestures with the palm raised vertically and oriented toward the addressee (VP gestures), “the actor engages in a schematic act of stopping something or holding something back” (Reference KendonKendon, 2004, p. 263). In contexts that seemed to be expressing meanings that are positive, Kendon argued that the performance of these gestures with their core semantic themes may be making explicit the underlying expression of an implied negative. Observations of headshakes in these contexts were also included (Reference KendonKendon, 2002). This research was foundational in flagging up gestures with the Open Hand Prone formation as candidates for studying relations between gestures and negation.

3.2.2 Analogical Links between Gestures and Negation

In a study that has been similarly influential to our understanding of gestures associated with negation, French semiotician Calbris arrived at the association between gesture and negation from a different perspective to that of Kendon. In the context of characterizing the symbolic import of certain physical components of gestures in French, Calbris discovered the salience of a number of such components to the expression of concepts related to negation (Reference CalbrisCalbris, 1990, Reference Calbris2003, Reference Calbris2005, Reference Calbris2011, Reference Calbris, Müller, Cienki, Fricke, Ladewig, McNeill and Teßendorf2013).

During Calbris’ career-long study of symbolic relations between gestures and notions, negation played a central role in developing and illustrating several key constructs, namely “gesture variants,” “kinesic ensembles,” “polysemous gestures,” and “polysigns” (Reference CalbrisCalbris, 2011, p. 24). While “gesture variants” refer to the finding that one notion may be expressed by several different gestures, which may be performed simultaneously resulting in a “kinesic ensemble” (or “cumulative variant”), the construct of “polysemous gestures” refers to gesture forms that may convey a range of different meanings, sometimes expressed simultaneously resulting in a “polysign.”

Defining negation “as an act of the mind that consists in refusing a relation, a proposition, an existence, and as the process of refusal” (Reference CalbrisCalbris, 2005, p. 2; my translation), Calbris has examined the different gestures that French speakers use to express negation and developed an explicit typology of them (Reference CalbrisCalbris, 1990, Reference Calbris2005, Reference Calbris2011, Reference Calbris, Müller, Cienki, Fricke, Ladewig, McNeill and Teßendorf2013). This typology is presented succinctly in Reference CalbrisCalbris (2005), which contains nine “variantes gestuelles de la négation,” namely: three head gestures (backwards toss, lateral sweep, and shake), three gestures with the vertical palm shaped flat (either raised, swept laterally, or oscillated), two with the extended index finger (raised, oscillated), and one involving the level hand moved horizontally.

Focusing on Calbris’ treatment of the variants involving the vertical palm and the level hand will illustrate further constructs while distinguishing her approach from Kendon’s (on this distinction, see Reference CalbrisCalbris, 2011, pp. 282–284). For Calbris, the Palms Forward (Kendon’s “VP”) and Level Hand (Kendon’s “ZP”) are not only gesture variants of negation, but also examples of “polysemous gestures,” negation being only one of their meanings, because the gestures can express a variety of notions on different occasions of use. Thus, what Calbris calls the Palms Forward gesture illustrates a semantic derivation from a singular analogical link to a physical action of the outwards-turned hand: self-protection, which depending on the context, “may express the notions of ‘opposition,’ ‘prudence,’ ‘refusal of responsibility,’ ‘stopping,’ ‘requesting someone to wait,’ ‘agreement,’ ‘refusal-negation,’ ‘objection,’ ‘restriction,’ or ‘perfection’” (Reference Calbris, Müller, Cienki, Fricke, Ladewig, McNeill and TeßendorfCalbris, 2013, p. 666). While this is a similar idea to the notion of a core semantic theme being derived from the underlying action motivating the gesture form (Reference KendonKendon, 2004), the gesture that Calbris calls the Level Hand illustrates a different relation, not of a core semantic theme, but one of plural motivation. Depending on which component of the Level Hand is salient or profiled (its movement trajectory or profiling of shape configurations) determines the analogical link established with a physical correlate, which in turn is the basis for the gesture’s meaning and array of subsequent semantic derivations, including: superlative, perfection, determinism, certainty, negation, refusal, cutting, and equality (Figure 17.2) (see also Calbris & Copple, this volume, Section 2.2.2).

Figure 17.2 Plural motivation of the “Level Hand” gesture

(Reference CalbrisCalbris, 2011, p. 183, reproduced with permission from John Benjamins Publishing Company)

Spelling out this network of plural motivations for the Level Hand leads Calbris to a different analysis than positing a core semantic theme derived from a single schematic action as per Reference KendonKendon (2004). This different position can be exemplified for the case of the gesture’s occurrence in contexts where positive assertions are being made. Whereas Kendon posits the expression of an implied negative in such contexts, which is made explicit through the performance of the gesture (enacting a “cutting,” “knocking,” “sweeping,” “rubbing”), Calbris posits a different analogical link based on singling out a physical component of the gesture’s form. As per Figure 17.2, the Level Hand’s expression of a superlative or perfection derives from the meanings of “quantity” and “totality,” which are motivated by the horizontal movement of the gesture. This is not an enactment or representation of a manual action but conveys the concept of “everywhere.” Similarly, the expression of negation by the Level Hand gesture is not related to the gesture’s origin in an action of knocking or sweeping away (as per Kendon), but to the meaning of “stop-refusal” represented as an obstacle, conveyed by the resistance of the palm down. “In short, the interpretation of a co-speech gesture supposes not only an appreciation of the contextual situation but also a physical understanding of the gesture and of the underlying symbolic system” (Reference CalbrisCalbris, 2011, p. 284, emphasis in original).

It seems that some gestures associated with negation may have a single, identifiable origin in a manual action, such as the Vertical Palm’s connection to “schematic stopping.” This may be the case for a number of other gestures that have been shown to express negation. The “brushing aside” gesture observed in Spain (Reference Teßendorf, Müller, Cienki, Fricke, McNeill and BressemTeßendorf, 2014, Reference Teßendorf, Fernández-Villanueva and Jungbluth2016) and the “dusting off palms” gesture observed in Nigeria (Reference WillWill, 2018) seem to be candidates. Variations in the performance of other gestures associated with negation, such as the Level Hand (ZP), may not be variations on an underlying action motif, but variations on analogical links that are based on fundamentally different signifiers (i.e. movement through the visual field, finger tips that draw lines, palms that resist/cover, and the edge that cuts).

Several aspects of the work discussed so far have been the subject of further studies of gestures associated with negation, including the relation between gestures and physical actions, the connection to negation in the linguistic utterance, and the occurrences of these gestures with the expression of explicit and implicit negation.

3.2.3 Relations to Aspects of Physical Action

In their typology of recurrent gestures identified in a corpus of German speakers, Reference Bressem, Müller, Müller, Cienki, Fricke, Ladewig, McNeill and BressemBressem and Müller (2014a, Reference Bressem, Müller, Müller, Cienki, Fricke, Ladewig, McNeill and Bressem2014b) identified a family of recurrent gestures clearly associated with the linguistics and pragmatics of negation. These gestures were crucially shaped by a shared formational characteristic – movement of the hand away from the body, which, according to Reference Bressem, Müller, Müller, Cienki, Fricke, Ladewig, McNeill and BressemBressem and Müller (2014b), revealed a common origin in an underlying “action scheme” motivating meanings and functions associated with negation. Specifying this scheme in a subsequent paper, Reference Bressem, Müller, Müller, Cienki, Fricke, Ladewig, McNeill and BressemBressem and Müller (2014b) stated that “Gestures may reproduce perceptually salient aspects of instrumental actions and extract distinctive elements of the action by comparing, selecting, and recombining physically pertinent elements” (p. 1600).

The action scheme underpinning all gestures included in the Away family, Reference Bressem, Müller, Müller, Cienki, Fricke, Ladewig, McNeill and BressemBressem and Müller (2014b) have argued, is “the effect that actions involving the clearing of the body space have in common: Something that was present has been moved away – or something wanting to intrude has been or is being kept away from intrusion” (p. 1596). Following the convention of referring to a recurrent gestural form based on the assumed underlying action, the recurrent gestures in the Away family were named the “brushing away gesture,” the “throwing away gesture,” the “holding away gesture,” and the “sweeping away gesture.”

Bressem and Müller articulated this relation between a physical action scheme and gestures based on analyses of spoken language use among adults and have subsequently argued it could be the basis for a multimodal construction in German (Reference MüllerBressem & Müller, 2017; see Section 4.2 below). Building on this line of work, Reference GawneGawne (2021) has observed a gesture among some speakers of Syuba in Nepal with a rotated “away trajectory” (thus similar to ‘brushing away’) that is “used only with grammatically negative forms” (p. 4) and “used to indicate the absence of someone or something” (p. 9). This relation between aspects of actions and gestures has been the focus of other research into gestures associated with negation, such as investigating children’s early language development and the connections between grammar and gesture.Footnote ⁴

3.2.4 Developmental Pathways into Multimodal Negation

While the research discussed so far has been based on video recordings of adult speakers, the multimodality of negation has caught the attention of French language acquisition researchers for its potential as a rich case study for investigating multimodal language development. For this, researchers have adopted a usage-based, video corpus methodology to explore children’s acquisition of negation with longitudinal data from a range of linguistic and interactive contexts (e.g. Reference Beaupoil-HourdelBeaupoil-Hourdel, 2015; Reference Beaupoil-Hourdel, Morgenstern, Boutet, Larrivée and LeeBeaupoil-Hourdel et al., 2016; Reference Morgenstern, Blondel, Beaupoil-Hourdel, Benazzo, Boutet, Kochan, Limousin, Hickmann, Veneziano and JisaMorgenstern et al., 2018).

Central to this body of research is the Paris Corpus (Reference Morgenstern and ParisseMorgenstern & Parisse, 2012). Hour-long video recordings were made at monthly intervals of five monolingual French children from the ages of only several months up to several years, filmed with high-quality video and audio equipment in naturally occurring settings for child–parent interaction, then transcribed and annotated by research teams using the software CHAT (Reference Morgenstern and ParisseMorgenstern & Parisse, 2007). Articulating the notion of a “multimodal pathway,” five stages of development in these children’s expression of negation have been identified, with careful consideration of the caregiver input that was also transcribed in the data (Reference Beaupoil-Hourdel, Morgenstern, Boutet, Larrivée and LeeBeaupoil-Hourdel, Morgenstern, & Boutet, 2016). With qualitative and quantitative analyses, the researchers have characterized this pathway as a gradual transition between “non-symbolic actions” (of rejection, avoidance), “symbolic actions” (gestures), and multimodal linguistic constructions similar in complexity to those of adult speakers (Reference Morgenstern, Blondel, Beaupoil-Hourdel, Benazzo, Boutet, Kochan, Limousin, Hickmann, Veneziano and JisaMorgenstern et al., 2018).Footnote ⁵ By making these distinctions, the findings are also shown to demonstrate a gradual specification of the functional roles of gestural and linguistic components being combined in the children’s expression (Reference Beaupoil-Hourdel, Morgenstern, Boutet, Larrivée and LeeBeaupoil-Hourdel et al., 2016). The researchers have proposed to view the acquisition of multimodal negation as evidence for the children’s development of linguistic and cognitive skills (Reference Beaupoil-HourdelBeaupoil-Hourdel, 2015).

Though focusing more on patterns of multimodal negation than on individual gestures, these studies have highlighted the salience of “PalmUp-shrug” and “indexWave” gestures in the acquisition of negation (Reference Beaupoil-Hourdel and DebrasBeaupoil-Hourdel & Debras, 2017; Reference Beaupoil-Hourdel and MorgensternBeaupoil-Hourdel & Morgenstern, 2021; Reference Blondel, Boutet, Beaupoil-Hourdel and MorgensternBlondel et al., 2017; Reference Morgenstern, Beaupoil-Hourdel, Blondel, Boutet, Ortega, Tyler, Park and UnoMorgenstern, Beaupoil-Hourdel, Blondel, & Boutet, 2016). Building on previous descriptions of similar forms and their association with the expression of notions related to negation (e.g. Reference CalbrisCalbris, 1990; Reference KendonKendon, 2004; Reference StreeckStreeck, 2009), studies in this line of research have advanced our understanding by proposing kinesiological analyses of these gestures, central to which is an intrinsic frame of reference. In describing gesture relative not to an observer’s frame of reference but to its own articulatory physiology, gestures are viewed as the combination of a muscular impulse constrained by the biomechanics of the human skeleton (cf. Reference BoutetBoutet, 2008, Reference Boutet2010, Reference Boutet2015, Reference Boutet2018).

3.2.5 Kinesiological Perspectives on Negation

In his “formal analysis of gestural negation,” Reference BoutetBoutet (2015) observes that previous descriptions of gestures associated with negation have adopted an egocentric frame of reference. They have restricted the description of such gestures to three-dimensional space (x, y, z) and overlooked an essential characteristic of the gestural negation system related to the physiology of human bodies. The bodily articulation of a given gesture from finger to shoulder can involve seventeen reference points (“degrees of freedom”). Taking the example of gestures described by Kendon and others as “Vertical Palm” and “Horizontal Palm,” by adopting the perspective of the gestural articulator, the negative meanings would not derive from the action motif said to be motivating the form of the gesture (i.e. of the hand sweeping aside for the ZP or schematically stopping for the VP), but from the physiological configuration that remains constant or “invariant” across all manifestations of this gesture: the pronation of the palm (Figure 17.3).

Figure 17.3 Invariant feature in different orientations of the palm: a. pronation/palm down, b. pronation/palm forward, c. pronation/palm sideways

(Reference BoutetBoutet, 2015, p. 118; article published under a Creative Commons Attribution 4.0 License)

Explaining this invariance (Figure 17.3), Boutet points out that from an egocentric reference point, “the forearm and the arm are in a different position for the three gestures,” as has been an important distinction in previous work adopting an egocentric frame of reference; however, Boutet highlights that from an allocentric perspective, “the hand does not change position relative to the forearm, being in a position of pronation across the three cases” (Reference BoutetBoutet, 2015, p. 118). In this approach, the invariance of the position of the hand (relative to articulatory segments of the forearm and arm – not to the observer) is where the different gestures derive their semantic theme from, which allows the analyst to arrive at the relation between embodiment and negation without the intermediary of an underlying action. Boutet’s analysis here is similar to Streeck’s, who also posits the invariant in these motions as their conceptual core, attributing form variations such as orientation of the gesture and position of the hand (supine-prone) to the speaker’s embodied adaptations to the interactional context in which the gestures are made (Reference StreeckStreeck, 2017, Ch. 5). From Boutet’s (kinesiological, intrinsic) perspective, the role of the difference between the palm oriented either away from the speaker’s body or facing down in distinguishing “different gestures” may have been exaggerated in the previous literature (see further discussion in Reference Beaupoil-Hourdel and MorgensternBeaupoil-Hourdel & Morgenstern, 2021, and in Boutet & Cienki, this volume).

Additional explanations for the form and organization of gestures associated with negation have been proposed based on studies that attend to the grammatical constructs of negation.

3.2.6 Kinesic Organization in the Grammar–Gesture Nexus

Building on sensory-kinesic approaches to negation (Reference Lapaire, Bonnefille and SalbayreLapaire, 2006a; see Section 4.2) with a video corpus of spoken English and form-based methods of gesture analysis (Reference Müller, Ladewig, Borkent, Dancygier and HinnellMüller, Bressem, & Ladewig, 2013), in Reference HarrisonHarrison (2018) I reported on the collection and analysis of a corpus of English spoken utterances that all involved linguistically what Reference HornHorn (1989) describes as “the traditional criteria for negativity – the presence of a negative particle, its appearance in a specified syntactic location, and so forth” (p. 34).Footnote ⁶ A second criterion for inclusion in the corpus was that the utterance involved the performance of a gesture from the Open Hand Prone gesture family (which I assumed to be potentially related to the expression of negation; Reference CalbrisCalbris, 1990; Reference KendonKendon, 2004). Based on grammatical and gestural microanalysis of over eighty examples, a number of distinctions in the form–function relations of such gestures, as well as a principle governing their temporal organization with speech, were proposed.

First of all, I distinguished between three form variations of Palm Down gestures associated with different kinds of negation or negative speech acts and, by analyzing several examples, argued that their performance with these linguistic utterances supported the view that each gesture reproduced variations on a manual action. As illustrated in Figure 17.4, the associations observed were between clausal negation and “sweeping away” for the Palm Down Across, exclusions and “clearing aside” for the 2-Palms Down Mid, and rejections and “cutting through” for the 2-Palms Down Across (Figure 17.4).

Figure 17.4 Three Horizontal Palm gestures based on different underlying actions: PDAcross (“sweeping away”), 2PDmid (‘clearing aside’), and 2PDAcross (‘cutting through’)

(Reference HarrisonHarrison, 2018, reproduced with permission of Cambridge University Press through PLSclear)

Close study of these gestures at the level of utterance revealed how speakers may organize their different phases of the gestural action (i.e. preparation, stroke, and holds) in relation to the grammatical structures in the accompanying speech, which can be viewed as creating “sync points” for gestures associated with negation (Reference HarrisonHarrison, 2018, Ch. 3).Footnote ⁷ Specifically, this research has shown the way in which the manual gestures were organized in relation to the node and scope of negation (Reference HarrisonHarrison, 2010) and to negative polarity items and negative focus (Reference HarrisonHarrison, 2013), as well as how manual, head, and linguistic elements were orchestrated in kinesic ensembles (Reference HarrisonHarrison, 2014b). Extending notions of lexical and conceptual affiliation between speech and gestures (Reference McNeill, Renals and BengioMcNeill, 2006), these examples of negation, expressed and organized multimodally, were offered as supporting evidence for a grammatical affiliation between speech and gesture (Reference Lapaire, Bonnefille and SalbayreLapaire, 2006a). The organization of gestures in the Open Hand Prone family in relation to the node and scope of negation has offered a basis for cross-linguistic comparison. One study found a similar pattern in French, whose syntax for negation on the sentence level overlaps to some extent with that of English (Reference Harrison, Larrivée, Larrivée and ChungminHarrison & Larrivee, 2016). In a study of negative utterances from the Subaya language in Nepal (Reference GawneGawne, 2021), however, the holding of gesture over scope of negation was not found to occur. Subaya is “a verb final language,” which may explain the difference in gesture organization because there is “less content within the scope of the negation that follows the verb” (Reference GawneGawne, 2021, p. 18).

Gesture studies of other key grammatical phenomena relating to negation will be discussed in further sections below, including implicit/covert negation, single versus double negation, and quantification, negation and scopal ambiguity. These studies develop explanations for aspects of gestures associated with negation.

4 Explanations for the Occurrence of Gestures Associated with Negation

What is driving the relations between gestures and negation? What is the conceptual and functional role of gestures associated with negation in spoken language production and perception? A growing body of published studies converge on answers to these questions. Linguistic, cognitive-semantic, functional, embodied, cultural, and psycholinguistic explanations can be identified.

4.3 Functional Load Sharing and Encoding Strategies

The prevalence of Open Hand Prone gestures with utterances showing no verbal negation has been the basis for alternative perspectives, such as “functional load sharing” proposed by Reference Wegener and BressemWegener and Bressem (2019). The data were observations of the “sweeping away” and “holding away” gestures examined in a six-hour corpus of narratives, procedural texts, and interviews among fifteen speakers (mainly male) of SavoSavo, a non-Austronesian language spoken on Savo Island in the Solomon Islands. In this study, the search domain for the sweeping away and holding away gestures was identified as utterances containing lexemes of explicit and implicit negation. Reference Wegener and BressemWegener and Bressem (2019) found “rather few instances of gestures associated with negation co-occurring with explicit verbal negation,” and instead, “the ‘sweeping away’ gesture, is used mostly with implicit (lexical or pragmatic) negation/negativity and only rarely accompanies explicit verbal negation” (para. 1). The researchers propose a functional explanation for these findings which can be called the “sharing the load” view: “When explicit verbal negation is used, it bears the main functional load. When explicit verbal negation is absent, negation/negativity is emphasized and made visible through gestures” (Figure 17.5).

Figure 17.5 Functional explanation: “sharing the load”

(Reference Wegener and BressemWegener & Bressem, 2019, with permission from the authors)

Wegener and Bressem’s diagram illustrates “the interplay between verbal and gestural negation,” and they argue that the link between gesture forms attested to occur with negation (such as Vertical and Horizontal manifestations of the Open Hand Palm family) may not be as tight or as frequent as suggested in the literature, and on that basis they advise “don’t look for negation gestures where verbal negation is used” (ibid., Results). Similar to Reference Inbar and ShorInbar and Shor (2019), the assumption here that the “sweeping away” and “holding away” gestures are expressions or manifestations of negation is central to this explanation, though Wegener and Bressem propose to view the relation between the gesture form and negation as a more explicit one of encoding. While this view does not entertain Reference CalbrisCalbris’ (2011) notion of polysemous gestures, the functional perspective finds some support in a rich chain of psycholinguistic experiments.

5 Theoretical and Applied Contributions

Studies of gestures associated with negation are helping to conceptualize the relationships between gesture and language. This can be seen in the uptake of the findings concerning gestures associated with negation in discussions of the multimodal nature of grammar (Section 5.1), the embodiment of cognition (Section 5.2), and the relation between gesture and signs (Section 5.3).

6 Conclusion

This chapter has offered an overview of gestures associated with negation from empirical and theoretical perspectives, developing and challenging a number of themes that have become widely acknowledged in research into gestures associated with negation specifically, and into the study of recurrent gestures and grammar/gesture relations more widely. Given the unquestionably linguistic nature of negation and its centrality to all languages, the area of research discussed in this chapter further attests that gesture is an essential part of human communication – fundamentally intertwined with language on every level. Theories of language and grammar must ultimately be able to account for these relations between gesture and linguistic concepts.