4.1 Referential

Kinesic actions contribute to utterance referential content in three ways: by pointing, depiction, and by conventionalized expressions (commonly, “emblems” or “quotable gestures”).

Pointing: In pointing at something, an actor can establish what is being referred to when a deictic expression is used. When, for example, the speaker GB, giving a tour of a church, says “There is Gill Crestwood,” only because he points to the statue of this person as he says this can his recipients interpret this utterance (Reference KendonKendon, 2004, p. 199).

In my work with Laura Versante on pointing (Reference Kendon, Versante and KitaKendon & Versante, 2003), we looked at the different hand shapes used to see if there are consistent differences in the contexts in which these occur. The material studied included video recordings at street stalls in Naples and nearby and a guide conducting a walking tour of a small town in the British Midlands. Six different hand shapes were observed in common use. Comparing the contexts in which they were used suggested that these different hand shapes related to how the speaker was using the object pointed at within his discourse. For example, if it was important for the speaker that recipients distinguish one specific object from another, the extended index finger was common. If the speaker referred to something as an example of a category (“Here you see a fine example of a war memorial”), if he is making a comment about the object indicated, or if there are features of it the guide’s recipients are to take note of (“You can see again the quality of this building”), a palm up or palm vertical open hand is used (Reference KendonKendon, 2004, p. 213).

Such observations suggest that, in pointing to an object (real or notional), the actor does not merely indicate it, but also, according to hand shape, shows how the object indicated is to be treated.Footnote ³ The pointing hand shapes employed are similar to those used in other expressions. Just as “palm-up-open-hand” may be used when a speaker gives an example of something, it is used in pointing when the object indicated is something the speaker wants the recipients to view as an example. In this case, “palm-up-open hand” is used as if to “present” the object to recipients as something to be contemplated. When a recipient only needs to distinguish one object from another, an extended index finger is used, as if the speaker would only touch the object rather than present it.

Depiction: Depiction (“performed depiction” [Reference ClarkClark, 2016]) is when someone acts as if showing what something looks like, how things are disposed in space, the manner in which an object or animate being moves, a pattern of action, to represent an emotional expression, to sketch the size and shape of something, to show how an object is handled, or to simulate using an instrument or tool, to “play at” doing something, or to act out a scene within imagined space. Whether fleeting or elaborate, depictions are widespread in live discourse of all sorts, in speakers and in signers alike.Footnote ⁴

Kinesic depictions are used by speakers in many ways. An enactment, used in conjunction with a verb phrase, can make the meaning of the verb more specific. For example, a speaker describes how someone used to “throw ground rice” over ripening cheeses to dry them off. As he says “throw” he shapes his hand as if holding a powder and does a double wrist extension as if distributing powder over a surface. The actions referred to by the verb “throw” are thereby given a more specific meaning – “scatter” might be an appropriate word (Reference KendonKendon, 2004, pp. 185–90). The same individual, in another recording, is talking about American soldiers, stationed near a village in England during the Second World War; toward the end of that war, when driving through the village, they sometimes “used to throw oranges and chewing gum (…) all off the lorries to us kids in the streets.” Here, saying “throw,” the speaker twice moves his hand rapidly backward, as if throwing things over his shoulder – a quite different sort of throwing than in the first case (Reference KendonKendon, 2004, p.186, Fig. 10.6).

Kinesic depiction allows a speaker to conjure objects into interactional space. For example, a speaker can indicate features of an object and then act as if it is actually present, holding it up exhibiting it or manipulating it, illustrating its properties: in discussing a new building, a speaker is explaining how a security arrangement will include “a bar across the double doors on the inside.” As he says “bar,” he lifts up his two hands and moves them away from each other, the hands held as if he was sliding them along a horizontally disposed elongate object. In moving apart his hands so posed, his hands seem to create the bar, making it present (Reference KendonKendon, 2004, p. 192, Fig. 10.12). A speaker, recalling lemon curd tarts and jam tarts from her childhood, moves her hands forward, fingers arranged as if holding out a little circular object in front of her (Reference KendonKendon, 2004, p. 192, Fig. 10.11).

The shape, size, and spatial characteristics of something can be shown kinesically. A speaker, explaining an archaeological excavation of an underwater village, describes finding a bronze spearhead underneath a large log. He says: “and underneath that they found a huge bronze spearhead.” When saying “huge bronze spearhead” he lifts up both hands, holding them apart with index fingers extended, depicting the spearhead’s length (also showing how “huge” was to be interpreted) (Reference KendonKendon, 2014a, Fig. 2). Again, a speaker describing how his father, a grocery store owner, received the cheeses he was to sell, said “an’ the cheeses used to come in big crates about as long as that,” holding out both hands as if holding a large elongate object in front of him, making present an imaginary crate (Reference KendonKendon, 2014a, Fig. 1b).

Spatial relationships of objects within a space can be shown, as can how objects move in relation to one another. A girl describing her aquarium refers to the two rocks inside it. As she does so verbally, she holds up her two fists side by side, modeling the rocks and showing how they are spatially disposed within the tank. Then, describing the pipe that provides aeration for the water, she draws her two hands apart, thumb and index finger of each hand extended and held with their tips separated, thus depicting a long thin object, horizontally disposed. Then she moves her two index fingers, directed vertically upwards, alternately up and down, depicting the motions of the air bubbles that come out of the pipe (Reference KendonKendon, 2004, pp. 191–194).

Objects, if depicted as if they are in front of the speaker, can then be acted upon as if still there. A speaker described a Christmas cake placed on a grocery shop counter to be sold in pieces. As he said, “the cake was this sort of size,” he sketched out a rectangular area over the table in front of him with two extended index fingers. Thereafter, as he explains how a customer can request a piece of a given size to be cut off, using a flat hand held vertically to model a knife blade, he acted as he might if cutting into the cake as placed on the table in front of him (Reference KendonKendon, 2004, pp. 194–197).

By means of such depictions, speakers provide cues, often conventionalized, which enable recipients to build in their imagination versions of what the speaker is showing. As Daniel Reference DorDor (2015) might say, they instruct the other’s imagination; the bodily movements themselves creating an imagined space within which movements and objects are shown and their spatial relations established. Verbal descriptions also enable imagined things to be present, but they do this by way of conceptual configurations designated by words. Performed depictions provide cues for percepts which allow for the building of images (Reference ClarkClark, 2016; Reference Huttenlocher, Kavanagh and CuttingHuttenlocher, 1975).

Symbolic or conventionalized kinesic actions Kinesic actions, typically manual, which have a socially shared, conventional (symbolic) meaning (“emblem,” “quotable gesture,” “symbolic gesture”) may be used in alternation with speech, but they are also often used together with speech, sometimes simultaneously with a word that has the same meaning, the speaker thus expressing the same meaning in verbal and kinesic form simultaneously. Such actions are often used to complement or to add to spoken meaning.

In my Neapolitan recordings, using kinesic and verbal forms of a lexical item simultaneously could often be observed. For example, a speaker (a shopkeeper) was explaining that nowadays in Naples there are too many thieves. As she uttered the word ladri (thieves) she used a manual expression which is always glossed with this word. Again, as a speaker said soldi (money), he rubbed the tip of his index finger against the tip of his thumb, an action always glossed as “money.” This practice of pronouncing equivalent verbal and kinesic expressions simultaneously is not seen only in Naples. A British English speaker, describing her job, says “I do everything from the accounts, to the typing, to the telephone, to the cleaning.” As she says “typing” and “telephone” and “cleaning” she does an action, in each case a conventional form, typically glossed with the same word that she utters.

In such cases the hand action seems just to convey the same meaning as the word. However, studying the contexts in which this occurs, taking into consideration how the action is performed, might suggest different effects speakers can achieve by using kinesic expressions with a fixed verbal gloss in this way. For example, duplicating a verbal meaning with a kinesic expression can ensure the meaning is present in conditions where it might not be heard or understood; it can draw the recipient’s attention to the speaker and thus assure the speaker that the recipient will have seen the meaning as well heard it. In many situations in the proverbially noisy circumstances of the Neapolitan street, speakers often find it useful to enhance their performances to facilitate the attention of addressees (Reference KendonKendon, 2004, pp. 349–354). A kinesic action can also be held beyond the duration of its verbal counterpart, thereby giving its meaning a continuing presence. This is another way in which its import can be foregrounded (see Reference KendonKendon, 2004, pp. 177–181 for examples given above).

Kinesic symbolic actions may also be used in parallel with verbal expressions, adding to the speaker’s spoken content. One illustration (Reference KendonKendon, 2004, p. 185, example [32], Fig. 10.5) must suffice: a city bus driver (in Salerno) describes the disgraceful things boys write on bus seat backs. They do this in front of girls, who are not upset but think it fun. As he says this, he holds both hands out, index fingers extended, positioned so that the two index fingers are held parallel to one another. In this way he adds the comment that boys and girls are equal participants in this activity, using here an expression glossed as “same” or “equal,” among other meanings given it (Reference De Joriode Jorio, 2000, p. 90).

4.2 Metadiscursive and Pragmatic Functions

Four ways can be distinguished in which kinesic actions operate metadiscursively in relation to spoken discourse. These are: operational, performative, modal, and parsing or punctuational (Reference KendonKendon, 2017a).

Operational: Head actions that function as an operator in relation to the speaker’s spoken meaning may be exemplified in the use of the head shake negating a verbal expression (Reference KendonKendon, 2002). A widely used hand action also may operate like this. Here the hand, with all fingers extended and adducted (so-called “open hand”), and palm facing downwards, is rapidly moved horizontally and laterally. Such an action is common in relation to negative statements or statements that imply a negative circumstance. For example, a Northampton UK shopkeeper uses this action when saying to a customer “That’s the finish of that particular brie” (Reference KendonKendon, 2004, p. 256); it may also be used with positive absolute statements, as if the hand action serves to forestall any attempts to deny what is being said: As a Neapolitan declares “La cucina di Napoli è la piú bella cucina” (“Neapolitan cooking is the most beautiful cooking”), the horizontal hand action used when saying this acts as if to say that any contrary claim will be denied (Reference KendonKendon, 2004, pp. 255–264).

Modal: Manual actions can provide an interpretative frame for a stretch of speech. The use of the “quotation marks” gesture (“air quotes”) showing that what is being said is “in quotes” is a common example. In a recording made in Salerno in 1991 (AK recording: Bocce I), someone discussing a robbery puts forward a speculation about the robber’s actions. As he does so, he touches the side of his forehead with the fingertips of a “finger bunch” hand and then moves his hand away and upward while spreading his fingers. This action, in Southern Italy, indicates something imagined. Here it serves to frame the speaker’s statement as an hypothesis.

Performative: Hand actions may be used to show the illocutionary force of an utterance. Quintilian described examples, for instance, the hand shape suited to the exordium (Quintilian, Book XI, iii, line 93 et seq. in Reference ButlerButler, 1922). Other examples include “Praying hands” (mani giunte) and “finger bunch” (grappolo), which is used for marking certain questions in Neapolitan speakers (Reference KendonKendon, 1995b; Reference Poggi, Attili and Ricci-BittiPoggi, 1983; and Reference Seyfeddinipur, Müller and PosnerSeyfeddinipur, 2004, for analogous expressions in Iran); Some of the uses of the palm-up-open-hand also serve in this manner, as when such a hand is proffered as a speaker gives an example, or when a speaker asks a question, holding out the palm-up-open-hand as if wanting something to be put in it (Reference KendonKendon, 2004, pp. 264–281; Reference Müller, Müller and PosnerMüller, 2004).

Parsing: A speaker’s actions, especially of hand or head, can serve to parse or punctuate ongoing speech. For example, as a speaker gives an account of a Christmas dinner, she lists what was served: “And then came the mince pies, the bowls of nuts and oranges” (Reference KendonKendon, 2004, p. 269). She repeats with each item an outward wrist rotation with palm up open hand as if “presenting” each item as she utters it. A speaker states four things that he supposes a burglar could have done, extending four fingers in succession, starting with the thumb, a new finger for each possible burglar action (AK recording: Bocce I). Among Neapolitan speakers, when announcing a topic, an associated “finger bunch” (grappolo) is presented, the hand is then moved outward, with the fingers spreading, as the speaker makes his comment. A thumb-tip-to-index-finger-tip “precision grip” is used by Neapolitans to mark a stretch speech of central importance, as when something specific is being said (for descriptions see Reference KendonKendon, 1995b; Reference KendonKendon, 2004, pp. 238–247; for German and American English examples of “precison grip” see Reference Neumann, Müller and PosnerNeumann, 2004 and Reference LempertLempert, 2011, respectively).

A further pragmatic function is interactional relational. This includes waving, embracing or hugging, inviting someone to sit down, offering, withdrawing, beckoning, requesting or halting turns at speaking, and so on. As far as I know, actions with these functions have received little in the way of systematic treatment, but see Reference Bavelas, Chovil, Coates and RoeBavelas, Chovil, Coates and Roe (1995).

4.3 Discussion

When we talk about things or ideas, with our words we conjure them up as objects in a virtual presence and with our kinesic actions, as noted when treating depiction, we may use various strategies to bring aspects of these objects into virtual presence, much as we do in drawing or modeling them, or in using our hands to demonstrate their functions.

These virtual objects are never produced in isolation but always in configurations with other virtual objects. These configurations are established through resources such as syntax, word choice, topic organization, and so on, but may also be established kinesically. Hands used when talking may act as if they are pushing objects into position, touching them or pointing to them, disposing them in relation to one another in space, and other operations of this sort. Many manual actions that serve in these ways are actions that can be understood as being versions of manual manipulations or manual actions upon or in relation to objects of whatever kind. The manual involvements of speaking are often schematic versions of object manipulations, as if, in speaking, objects have been conjured into a virtual space, the speaker’s hands being used to manage them (Reference Kendon, Gambarara and GiviglianoKendon, 2009; Reference StreeckStreeck, 2009).

It should be stressed that a given action can serve in more than one way simultaneously, and a given form may contribute in one way in one context and in a different way in another. The semantic functions of kinesic hand actions cannot be sorted into mutually exclusive categories. Further, the different ways we have outlined in which kinesic actions can contribute to utterance meaning have been arrived at by observers or analysts, after they have reflected upon how the form of a visible action, regarded as in some way intended or meant as part of the speaker’s expression, can be related to the semantic or pragmatic content apprehended from the speech. Our ability to do this is based upon our ability to grasp how these actions are intelligible. How do we do this? Involved here is the question of the intelligibility of kinesic expressions and how this interacts with the intelligibility of the associated spoken expressions. Reference Lascarides and StoneLascarides and Stone (2009) is a recent discussion, but this question deserves further exploration.

Finally, how can we be sure whether these kinesic actions make a difference to how recipients grasp or understand the propositional or pragmatic meanings of the utterances they are a part of? We do know, both from everyday experience and from numerous experimental studies (Reference KendonKendon, 1994 and Reference HostetterHostetter, 2011 are reviews), that these actions do make a difference for recipients, but whether they always do so, and whether they do so in the same way, we cannot say, nor do we have a good understanding of the circumstances in which they may or may not do so (Reference Rimé, Schiaratura, Feldman and RimèRimé & Schiaratura, 1991, pp. 272–275 suggest a start in how this might be investigated).

This brief survey should make clear the diverse ways in which speakers employ kinesic actions. A simple statement cannot be made about what kinesic actions do or what they are for. For me, a consideration of these different kinds of use supports the view that these actions are to be regarded as components of a speaker’s final product. That is, they are not symptoms of processes that lead to verbal expression nor are they actions brought in by the speaker as aids to the task of creating a verbal formulation. Rather, they are integral components of a person’s expression as this is achieved in the immediate circumstances of an utterance’s production. Speakers monitor their outputs and match output against some expression target. Often they follow with revisions of what was just said, and the role of kinesic action in these revisions may differ from its previous role. So although kinesic action in a current utterance can provide something that the speaker can use if a revised version is produced, and in that way “help the speaker,” kinesic expression within a current utterance is part of that utterance. It does not help the speaker’s speaking in that utterance. This is so because it is produced at the same time as the verbal expressions, not beforehand.

5.1 Papua New Guinea

My work on a sign language in Papua New Guinea, published in 1980 as three papers in Semiotica, has since been republished as a monograph (Reference KendonKendon, 2020).Footnote ⁵ This work begins with a description of the nature of the units of behavior that function as signs and how contrasts between them are established. Then follows an examination of how the form of a sign may be shaped by features of its referent. This includes a comparative study, comparing signs from American Sign Language (ASL) (Reference Stokoe, Casterline and CronebergStokoe, Casterline, & Croneberg, 1965), from an alternate sign language used by the Pitta-Pitta of Northwest Queensland (Reference RothRoth, 1897), and from one used by workers in a sawmill in British Columbia (Reference Meissner and PhilpottMeissner & Philpott, 1975). This was to test a hypothesis that the “iconic device” (Reference Mandel and FriedmanMandel, 1977) a sign language uses is related to the semantic domain of the sign’s signified. A study of how pointing was used in the Enga corpus follows, and after this there is an analysis of utterance construction. Here, actions in the face and eyes concurrent with the production of manual signs are examined for how they contribute to bringing off complex sign combinations. Sign order, subject–object relations in verbal expressions, temporal reference and interrogation are also examined. In this work I used what was then available on ASL by Reference StokoeStokoe (1960), Reference Stokoe, Casterline and CronebergStokoe, Casterline, & Croneberg (1965), Ursula Bellugi and associates (Reference Bellugi, Klima, Kavanagh and CuttingBellugi & Klima, 1975, Reference Bellugi, Klima, Harnard, Steklis and Lancaster1976; Reference Fischer and GoughFischer & Gough, 1978), and by authors in Reference FriedmanFriedman (1977).

I found Enga signers signed at a rate comparable to that reported for ASL signers; they made use of a similar repertoire of locations and movement patterns as Stokoe had described for ASL. The repertoire of contrasting hand shapes was much smaller, however. A comparative analysis of the different “iconic devices” used (using Reference Mandel and FriedmanMandel, 1977, for ASL) showed that the Enga signs and those in the three unrelated sign languages all included enactment, modeling, or sketching and these were related to the semantic domains of the referents in similar ways. Enactment, the most common device, is used when signing bodily actions of all sorts: many animals are signed by enacting a movement pattern selected as characteristic of them, and objects, when they can be designated, by a fragment of an action pattern used in manipulating or using them. Modeling, often combined with enactment, is commonly used for large animals which are not handled or for objects such as tools. Sketching (as in outlining the shape of something) was found to be the least common. These findings suggest that the processes by which signs in a sign language are generated are quite general. This is fully compatible with signs in different sign languages being diverse, not only because features selected for representation can be quite different but also because the processes by which signs are shaped as they become part of a shared system ensure that signs for similar referents in different sign languages are also likely to be different from one another.

Pointing was used by Enga signers in ways very similar to what has been described for this in ASL. Facial, postural, and gaze direction activity in relation to manual signing was shown to serve as a bracketing device, marking out distinct segments of discourse, and providing context for the manual signs, so adding directly to discourse meanings. In a conclusion, I remarked “the functions of concurrent activity in respect to signing are no different from their functions in respect to speech in users of oral language. This points out quite clearly that signing, despite the fact that it makes use of quite different expressive devices, as a functional system, is the exact analogue of speech” (Reference KendonKendon, 2020, p. 112). In the light of what we know today about the functions of mouth actions in sign languages (see Reference Boyes-Braem and Sutton-SpenceBoyes-Braem & Sutton-Spence, 2001), this statement needs some modification. Nevertheless, this foreshadows a point I elaborated much later (e.g. Reference KendonKendon, 2014a), which is that, if sign language and spoken language are to be compared, they should be compared as uttering activities, not as fixed texts. Such comparison reveals how signing and speaking are variations of a fundamentally similar activity, but it also emphasizes the point that is crucial to understanding speaking as a process: Kinesic as well as vocal actions must be taken fully into account. It points to the multimodal character of uttering and suggests that the division between “paralanguage” and “language” cannot be sustained categorically. As we remark below, this has implications for how we conceive of “language” and for how we are to approach the problem of its evolutionary origins.

5.2 Sign Languages in Aboriginal Australia

In a much larger investigation, I examined sign languages in use among Australian Aborigines. In an area of central Australia that stretches northwards from above Alice Springs in the Northern Territory as far as the border with Arnhem Land, a practice is followed in which, when a woman is widowed, for an extended period thereafter, she and certain of her classificatory relatives forego the use of speech. Where this practice is observed, complex sign languages have developed. These are used among women in mourning, but they may also be used in other circumstances of everyday life as a matter of convenience or suitability. There are many interesting aspects of these sign languages, from cultural and semiotic points of view, and when comparing them to other alternate sign languages (such as those reported from North America or in Christian monastic communities – see Reference DavisDavis, 2010; Reference Umiker-Sebeok and SebeokUmiker-Sebeok & Sebeok, 1987) as well as to primary sign languages. It is also useful to compare them with other language codes, such as writing or drum and whistle languages (Reference KendonKendon, 1988, Ch. 13; Reference MeyerMeyer, 2015 is a recent study). Here I comment on just one issue, which was central in my work, and that is how these central Australian sign languages are structurally related to the spoken languages of their users.

A comparison of signing among women of different ages that I undertook at the Warlpiri settlement, Yuendumu (Reference KendonKendon, 1984), suggested that, as users grew more proficient at using their sign language, they came to use, more and more, signs that represent the semantic units expressed by the semantic morphemes of the spoken language. A notable feature is that it appears common for signs to develop which represent the meanings of the morphemes of the spoken language. In consequence, concepts expressed in spoken language by compound morphemes get expressed by compound signs that are the equivalent of these compounds. One example must suffice (for this and other examples, see Reference KendonKendon, 1988, pp. 369–372). In Warlpiri, “scorpion” is kana-parnta, a compound of kana “digging stick” and -parnta, a possessive suffix. Thus “scorpion” in Warlpiri is, literally, “digging stick-having.” In a language of a neighboring community, Warlmanpa, the same creature is known as jalangartata, a compound of jala “mouth” and ngartata “crab” or perhaps “claw.” Note that, in creating a sign for this creature, we do not find a sign for “scorpion” derived from some feature of the animal (its action of raising its tail comes to mind) but signs based on representations of the meanings of the verbal components which make up the verbal expression. Thus, in sign languages of neighboring language communities in this part of Australia, there can be differences in signs for similar things, these differences deriving from the fact that signing in part develops as kinesic representations of the semantic units of their respective spoken languages. It is interesting to compare this with writing (Reference KendonKendon, 1988, Ch. 13).

Detailed information on other alternate sign languages is scant, and there are few studies that have investigated how they might relate to the spoken languages of the communities where they developed. However, surveying such evidence as is available shows that we need not expect other alternate sign languages to be as closely guided by an associated spoken language as the Australian central desert sign languages tend to be (see Reference KendonKendon, 1988, Ch. 13). The so-called Plains Indian Sign Language of North America (in use in the nineteenth century) studied by Reference WestWest (1960) shares many features found in primary sign languages but not found in the Australian central desert sign languages. For example, exploiting space for expressing grammatical relations is found in Plains Indian Sign Language but not in Warlpiri Sign Language. Since Plains Indian Sign Language was used as a lingua franca by people with several different spoken languages, we might not expect its users to structure it according to one specific spoken language. This is what Anastasia Bauer reports in her study of signing used at a multilingual settlement in Arnhem Land (Australia), where it is also shared by deaf persons (Reference BauerBauer, 2015). Further, many of the indigenous spoken languages of North America have “polysynthetic” morphologies. This would make the creation of signs as equivalents of recombinable semantic morphemes (possible in Warlpiri) hard to achieve.

Several aspects of these alternate sign languages are of interest. At least in the central desert sign languages, there are many signs that are used to signify several different semantic morphemes, rather than one only. Studying this throws light on how users cluster meanings into broader categories (see Reference MeggittMeggit, 1954). Examining sign formation processes can also show what gets selected as the bases of iconic signs within the culture. More broadly, perhaps, it is interesting how readily kinesic equivalents of word meanings can be created. This seems to support the idea that many spoken words are closely related to schematic units of action (Reference Gullberg, Pederson and BohnemeyerGullberg, 2011; Reference PulvermullerPulvermuller, 2005). This fits in with the idea that language begins as a form of action; its transformation into a system of abstract symbols is a later development.

2.1 Relation between Verbal and Gestural Modalities

Speech and gesture are tightly connected on the temporal, semantic, and pragmatic levels (Reference McNeillMcNeill, 1992). For analyzing gesture–speech relations and the design of coding and annotation systems, this close connection of both modalities results in rather practical consequences, because systems have to reproduce this link on different levels. Existing systems, also depending on their technical implementation, solve this problem in different ways. The majority of systems use the transcription of speech as the center for the analysis of gesture–speech relations. As a result, gestures are placed in a position that is secondary to speech, and first steps in coding and annotation concentrate on the analysis of speech (see e.g. Reference McNeillMcNeill, 1992). Gestures are added either by annotating them into speech transcription or by making them hierarchically dependent on the speech transcription. Gestures are thus viewed in relation to speech from the beginning of the analytical process. Other systems allow for a first review and analysis of the data without an instant inclusion of speech and even allow for coding and annotating gestures without sound and, initially, independently of speech. As a result, these systems initially concentrate on the form of the gestures before bringing together speech and gestures in the analytical process (Reference Bressem, Ladewig, Müller, Müller, Cienki, Fricke, Ladewig, McNeill and TeßendorfBressem, Ladewig, & Müller, 2013; Reference Lausberg and SloetjesLausberg & Sloetjes, 2009) and thus separate speech and gestures into different parts of the analysis to allow for a preferably nonbiased coding and annotation of gestures.

2.2 Facets and Specifics of Coding and Annotation

Tied to the different possible relations between, and orders of, both modalities in the coding and annotation process are differences in the type and the specificity of the aspects to be coded and annotated. As already mentioned, some systems put particular emphasis on detailed descriptions of gestural forms because this allows one, for instance, to discover regularities and structures on the level of form and meaning, along with gestures’ potential for combinatorics and hierarchical structures,Footnote ² or the best possible reproduction of gestures in technical systems, such as avatars (Reference Kipp, Neff and AlbrechtKipp et al., 2007; Reference Pouw, Trujillo and DixonPouw et al., 2019) or robots (Jokinen, this volume; Reference Kopp, Church, Alibali and KellyKopp, 2017). Connected to this approach is often the development of particular categories and a separate description of gestural form parameters to make the kinesics and semiotic processes in gestures visible (Reference BoutetBoutet, 2015; Reference Bressem, Müller, Cienki, Fricke, Ladewig, McNeill and TeßendorfBressem, 2013a). In contrast to these approaches, other systems carry out, for instance, a selective description of gestural forms that is also dependent on the meaning expressed in speech. The focus of these systems is on a functional analysis of gestures in relation to speech, such that “only the bodily movements maintaining a relation to speech – co-speech gesture – as well as the function of the gesture are of interest to us” (Reference Colletta, Kunene, Venouil, Kaufmann and SimonColletta, Kunene, Venouil, Kaufmann, & Simon, 2008, p. 59). Categories for the description of gestural forms are taken, for instance, from sign language studies and gestures are matched to the sign language alphabet (see e.g. Reference McNeillMcNeill 1992). The differences in describing and categorizing gestural forms discussed above are only meant to exemplify how the specificity of individual systems may vary; they are meant to underline the type of consequences that may be connected to different possible conceptions of, and views on hierarchies in, the relations between speech and gestures. (Further differences in the specificity and details of the particular systems along with the underlying research foci that are decisive for the structure of the systems will be discussed in Section 3.) In general, regardless of the particular approach and focus taken, the basics of current coding and annotation systems are gesture phases, gestural forms, gesture type, a transcription of speech, and the relation between gestures and speech.

2.3 Qualitative Versus Quantitative Perspectives

A further essential difference in existing systems is found in the research perspective. A qualitative perspective is characterized by more openness and flexibility, as well as often more naturalistic data from more ecologically valid settings, by which the dynamics of gesture–speech relations can be captured and hypotheses can be developed from, and based on, the material. By contrast, quantitative research aims at the validation of given hypotheses, replicable data and larger amounts of data, that are statistically evaluable. These two different positions influence the conception and design of coding and annotation systems. Systems with a stronger qualitative emphasis might show greater variability and flexibility (see e.g. Reference Müller, Müller and PosnerMüller, 2004) that is usually not included in systems aiming for a quantitative approach. Connected to this is also the use of larger corpora as data, as well as reliability or intercoder agreement measures (see e.g. Reference Lausberg and SloetjesLausberg & Sloetjes, 2009).

2.4 Procedures for Coding and Annotation

Annotation programs have become the standard for analyzing gesture–speech relations in recent years, both for qualitative and quantitative studies. Through the use of such programs, coding and annotation have become faster and more reliable, in terms of both technical practicability and plausibility. While the majority of studies use ELAN, a program developed by the Max Plank Institute in Nijmegen (Reference Wittenburg, Brugman, Russal, Klassmann and SloetjesWittenburg, Brugman, Russal, Klassmann, & Sloetjes, 2006), researchers also have the option of choosing other programs, such as EXMARaLDA (Reference Schmidt, Mehler and LobinSchmidt, 2004) or ANVIL (Reference KippKipp, 2001). Although the basic setup of these programs is similar, differences in their functionality may lead to differences in the coding and annotation systems. Whereas all kinds of annotation software offer the possibility of including three-dimensional motion data, ANVIL, in addition, allows “spatial annotation” where two points can be marked directly on the video screen so that the distance between two hands can be more accurately annotated (Reference Kipp, Neff and AlbrechtKipp et al., 2007, p. 331). The three annotation programs thus offer fine-grained possibilities for combining visual information only with automatic analyses of movement patterns (see e.g. Reference Ripperda, Drijvers and HollerRipperda, Drijvers, & Holler, 2020) and consequently allow for an analysis of gestural forms based on more material, on larger corpora, and at a more abstract level (Reference MittelbergMittelberg, 2018; Reference Trujillo, Vaitonyte, Simanova and ÖzyürekTrujillo, Vaitonyte, Simanova, & Özyürek, 2019). (For an overview and discussion of the different methods in multimodal motion tracking, see Reference Pouw, Trujillo and DixonPouw et al., 2019, and Trujillo, this volume.)

3.1 Words and Gestures: Coding and Annotation for Exploring Language (Use)

Most studies exploring the relation of words and gestures in language (use) assume that usage events are dynamic and multimodal in nature, yet the degree to which gestures are part of language is variable (Reference CienkiCienki, 2015). The relation between speech and gesture is understood to be “reciprocal” such that the “gestural component and the spoken component interact with one another to create a precise and vivid understanding” (Reference KendonKendon, 2004, p. 174, emphasis in original). Spoken words and/or phrases have a close relation with gestures on different linguistic levels (see e.g. phonology, semantics, syntax, pragmatics), but, depending, for instance, on the type of gesture, the communicative context or genre, or the language community, this connection may vary. Starting from this assumption, studies have aimed to discover the tight relation between speech and gestures on these different levels and relations. Depending on the object of investigation, different aspects of coding and annotation become relevant, both regarding the temporal coordination and as the functional relation between the two sensory modalities.

Early on, research showed that the temporal relation between gestures and body movement in general shows a “precise correlation between changes of body motion and the articulated patterns of the speech stream” (Reference Condon and OgstonCondon & Ogston, 1967, p. 227). Accordingly, it is assumed that “the pattern of movement that co-occurs with the speech has a hierarchic organization which appears to match that of the speech units” (Reference Kendon, Seigman and PopeKendon, 1972, p. 190). Research addressing this relation thus requires a coding and annotation system that captures the link of both modalities regarding intonation phrases, stress, and syllables, for instance. In addition, particular movement characteristics, such as the velocity profile, may be of interest (Reference Karpiński, Jarmołowicz-Nowikow and MaliszKarpiński, Jarmołowicz-Nowikow, & Malisz, 2008). Likewise, data from programs such as Praat, a software package for phonetic speech analysis (Reference Boersma and van HeuvenBoersma & van Heuven, 2001), should be integrable. If, for instance, in addition to other factors, the connection of gestures with the narrative structure of speech is to be investigated, gestures’ relation with discursive elements has to be included. For instance, in a study on the Palm Up Open Hand (PUOH) and the type of prosodic and discourse units which are marked by this type of gesture, Reference FerréFerré (2011) transcribes, annotates, and segments speech by using a range of different programs, such as Praat and EasyAlign, and annotates speech for speech acts, intonation phrases, words, syllables, and stress. For gesture annotations, the study uses the annotation programs ELAN and ANVIL, and using the typology of gestures proposed by Reference McNeillMcNeill (1992), codes beat gestures, their phases, and their semantic and discursive functions in discourse. With this, Reference FerréFerré (2011), for instance, shows that beat gestures accompany emphatic stress in the verbal mode and that the PUOH, in particular the hand flick, fulfills different pragmatic functions in discourse and can acquire “a judgmental or epistemic value” (Reference FerréFerré, 2011, p. 16).

If it is assumed that because gestural movement parameters vary in their characteristics depending on their correlation with prosodic characteristics of speech (see e.g. Reference Ruth-Hirrel and WilcoxRuth-Hirrel & Wilcox, 2018), a fine-grained coding of the gestural kinesics is necessary (Reference BoutetBoutet, 2015). This perspective is followed by Reference Shattuck-Hufnagel and RenShattuck-Hufnagel and Ren (2018). Addressing the temporal relationship between gesture and speech with non-referential gestures, the authors applied a coding and annotation system that starts out with an analysis of the gestures without sound, “to avoid any possibility of the labeler’s judgment about events in one channel being influenced by events in the other” (p. 4), and coded gestures for their phases, their forms (hand shape, movement trajectory, location, handedness), referentiality, and their use in sequences. Subsequently, speech was transcribed orthographically and labeled for its intonational structure using Praat and ToBI, segmented into syllables, and annotated for higher-level prosodic constituents. Using this approach, the authors show that trajectory shapes of the gestures that were investigated “are consistent across a higher-level prosodic grouping” and that the category of beats includes “gestures with multiple phases and various types of rhythmicity” (p. 1).

As illustrated by the two examples given above, studies can overlap in the categories and classification applied, in particular with respect to the coding of gestural forms and functions, yet depending on the research focus on different facets. However, annotation systems that specifically address such a prosodic perspective with the aim of providing a general basic structure that may be adjusted to the needs of particular research questions are rare. One example is that proposed by Reference Jarmołowicz, Karpiński, Malisz, Szczyszek, Esposito, Faundez-Zanuy, Keller and MarinaroJarmołowicz, Karpiński, Malisz, & Szczyszek (2007).

Similarly, only a few systems provide a basic outline for analyzing gesture–speech relations from the level of semantics and/or syntax. One system that puts forward a general framework is the Linguistic Annotation System for Gestures, or LASG (Reference Bressem, Ladewig, Müller, Müller, Cienki, Fricke, Ladewig, McNeill and TeßendorfBressem et al., 2013). It provides ways to describe and analyze the motivation of gestural forms (modes of representation, image schemas, motor patterns, and actions), addresses gestures in relation to speech on a range of levels of linguistic description (prosody, syntax, semantics, and pragmatics), and in doing so offers obligatory as well as optional categories for each of the different aspects of linguistic description. Yet, due to its claim to cover all of these levels of description, the system is of course not detailed enough for the investigation of particular semantic or syntactic research questions; it thus needs to be adjusted to address different depths and aspects of the relation of speech and gestures in language (use) (see e.g. Reference Seeling, Fricke, Lynn, Schöller and BullingerSeeling, Fricke, Lynn, Schöller, & Bullinger, 2016, for an adaptation of the system).

The majority of research rather follows the LASG procedure outlined above and, based on generally accepted categories, provides for coding and annotation systems tailored to a specific research question. This will be exemplified in Section 3.2 by studies focusing on verb semantics and their relation to syntactic constructions. Cross-linguistic studies on motion events not only emphasize a tight link between gestures with the semantics of speech, but also show a close connection to syntactic patterns in the spoken utterance (Reference Kita and ÖzyürekKita & Özyürek, 2003). For example, differences in grammatical aspects used in speech are reflected in the gestural forms, the timing of gestures relative to the verbal utterance, and in the information distributed across the modalities (Reference Cienki and IriskhanovaCienki & Iriskhanova, 2018; Reference DuncanDuncan, 2005; Reference Gullberg and PavlenkoGullberg, 2011). As a result, coding and annotations schemes need to be able to code and annotate speech in terms of how events, for instance, are semantically and syntactically construed along with gestural form and meaning on different levels. Reference Cienki and IriskhanovaCienki and Iriskhanova (2018), for example, in investigating aspectuality in German, Russian, and French, thus use a coding and annotation scheme that includes a set of verb tense coding categories, “time meaning” (past, present, future, and ø for infinitives and imperatives), gesture phases, and a kinesiological account to determine the category type of a given gestural movement (Reference BoutetBoutet, 2015; Boutet & Cienki, this volume) – that is bounded versus unbounded movements – “without consideration of the type of co-occurring verbal expression” (Reference Cienki and IriskhanovaCienki & Iriskhanova, 2018, p. 68). For each of these features, hierarchical relations between the tiers and controlled vocabularies were defined in ELAN. “Each of the verbs or its constituents (auxiliary, participle) was annotated according to the duration of its vocal production” and only gestures “that overlapped [with] the production of a verb” (Reference Cienki and IriskhanovaCienki & Iriskhanova, 2018, p. 75) were annotated. In another study (investigating aspectuality and, in particular, Aktionsart, in Mandarin Chinese and English), Reference Duncan and EmmoreyDuncan (2003) included a transcription of speech, along with speech dysfluencies, in which “utterances expressive of the target aspect and Aktionsart distinctions were noted” (Reference Duncan and EmmoreyDuncan, 2003, p. 192) along with the relevant features of grammatical structure. The gestures were coded for phases, form, type, semantic content, and function in relation to the speech; “the timing of the gesture production relative to the speech was exactly coded” (Reference DuncanDuncan, 1996, p. 21), and movements of referential gestures were coded according to Talmy’s classificatory scheme (1985) see e.g. MOTION, PATH, and MANNER. For a study on the English motion construction [V(motion) in circles], Reference ZimaZima (2014) coded gestures for motion components (Manner vs. Path), shape and orientation of palm and fingers, handedness, depiction of number of circles, and type of motion. Speech was coded for the semantic meaning of the construct (see e.g. literal motion, metaphorical use, ambiguous) and discourse genre.

All of these examples given above underline that even with similar research topics, coding and annotation procedures differ immensely between studies, not only for speech but, more importantly, also for gestures. Although similarities between the practices are visible, the biggest differences lie in the amount and degree of detail of the coding of gestural characteristics, on the levels of form, meaning, and function. As mentioned in Section 2, one reason for these differences can be found in the (hierarchical) relation assumed between both modalities and thus the “direction” from which the data is approached (speech first vs. gesture first). Depending on this, the coding and annotation practices differ. Reasons for these differences lie in the underlying research question. Compare, for instance, Reference Cienki and IriskhanovaCienki and Iriskhanova (2018) and Reference ZimaZima (2014). Whereas Reference Cienki and IriskhanovaCienki and Iriskhanova (2018) are interested in whether speakers of French, German, or Russian gesture “similarly or differently (with regard to movement quality) when talking about events in the perfect(ive) versus in the imperfect(ive)” (p. 5), Reference ZimaZima (2014) addresses the question of whether gestures can be a recurrent feature across instantiations of English motion constructions. Due to the fact that Zima is, first and foremost, interested in whether gestures occur often with these constructions, a less detailed form description is given in favor of a number of occurrences of gestures. Reference Cienki and IriskhanovaCienki and Iriskhanova (2018), however, based on their assumption that aspectuality may be visible in gestural movement characteristics, need to take a more fine-grained perspective on these parameters. Both research questions thus take a different look at gestural form characteristics, and, as a result, the coding and annotation procedures put different emphases on the gestural form characteristics.

Similar variations can also be found in studies addressing the relation between speech and gestures on the level of pragmatics.

Studies on pragmatic gestures assume that these different functions go hand in hand with differences in their context of use as well as in the gestural forms (Reference KendonKendon, 1995; Reference LadewigLadewig, 2010; Reference Müller, Müller and PosnerMüller, 2004). In order to account for these different aspects, coding and annotation procedures thus call for a detailed description of gestural forms, “micro-analyses of single sequences […] to corroborate the semiotic and context-of-use analysis” (Reference Müller, Müller and PosnerMüller, 2004, p. 12), and the exact timing of the gestures in relation to speech (see also Reference LadewigLadewig, 2010, for an adaption of this method in ELAN). As a result, in their study on discursive function of holding-away gestures, Reference Bressem, Stein and WegenerBressem, Stein, and Wegener (2017) take the form-based linguistic approach with the four-step procedure moving from a context-independent toward a context-sensitive analysis. They take the distribution of the gestures (as proposed in the LASG system) as their starting point, combining it with a functional analysis of discourse markers (Reference FraserFraser, 1999), and add additional layers of analysis to account for the particular focus on discursive functions of gestures. Based on this, the authors illustrate specific forms and a functional diversity of the gestures that are connected to different types of speech units and show that changes in the discourse type and interactional setting lead to specific forms and uses of the holding away gesture. In a study investigating pragmatic functions of pointing gestures, Reference Enfield, Kita and de RuiterEnfield, Kita, and de Ruiter (2007) “suggest that form/function differences between […] two types of pointing gesture reflect distinct types of constraints which interactants have to satisfy in confronting the online problem-solving task of designing utterances in face-to-face interaction” (p. 1724). For this, the authors coded and annotated all utterances with spatially anchored pointing gestures for the “manner of articulation of the gesture, distinguishing formally between B-points (gestures in which the whole arm is used as articulator, outstretched, with elbow fully raised), and S-points (gestures in which the hand is the main articulator, the arm is not fully straightened, typically with faster and more casual articulation)” (p. 1725). Additionally, orientation of the head was coded. Gestures with forms other than those with a single protruding digit and movements that seemed to represent motion were excluded. As the authors were interested in the relation between form and function in these gestures, their focus was on particular and quite restricted form characteristics of the gestures.

All of these examples given above illustrate how a particular focus on exploring gestures in language (use) on different levels of linguistic description may lead to very specific coding and annotation schemes and practices.

3.2 Learning to Speak and Gesture: Coding and Annotation for Examining Language Development

Research on language acquisition that is related to gesture focuses on the role that gestures play in mediating the acquisition of spoken language, in communication, and in cognition (see Morgenstern, this volume; see also Wilcox, this volume, on the place of gesture in sign language acquisition). Studies investigate how gestures develop and change in parallel to spoken language development and what role they play in the prelinguistic period (see Reference Gullberg, De Bot and VolterraGullberg, De Bot, & Volterra, 2008 for an overview). Depending on age groups, different topics emerge. In earliest development, the complex relationship between lexical and syntactic development in comprehension and production is of interest (see e.g. gesture types, difference of actions from gestures, relation with spoken syntagmas, speech acts) (see e.g. Reference AndrénAndrén, 2010; Reference Beaupoil-Hourdel, Morgenstern, Boutet, Larrivée and LeeBeaupoil-Hourdel, Morgenstern, & Boutet, 2016). In older children, their use of iconic gestures (Reference Allen, Özyürek, Kita, Brown, Furman, Ishizuka and FujiiAllen et al., 2007), narrative skills (Reference CollettaColletta, 2009), or marking of focus (Reference Koutalidis, Kern, Németh, Mertens, Abramov, Kopp and RohlfingKoutalidis et al., 2019) are investigated. For each of these topics, slightly different coding and annotation schemes are necessary to address the relation between speech and gestures at different levels of detail. In the rest of Section 3.2, four different coding and annotation practices of studies on first language development will be summarized to illustrate the diversity and variability of the approaches, depending on the research question involved.Footnote ³

In a study on the use of gestures in children between 18 and 30 months, Reference AndrénAndrén (2010) focused on the children’s gestural repertoire, gestural development over time, and the organization of gesture in coordination with other semiotic resources, such as speech. As a result, the coding and annotation scheme includes a transcription of utterances consisting of words and/or gestures in CHATFootnote ⁴ and the annotation of every instance of a gesture in terms of various deictic, iconic, and conventional features. “These categories were not predetermined before the annotation procedure [was] begun, but emerged in an iterative and ‘dialectic’ fashion during the annotation process itself. This means that all categories are essentially motivated by what was found in the data” (p. 97). In a further step, these basic annotation categories were then specified for a qualitative analysis that focused on “how deictic, iconic, and conventionalized aspects occur together in various sorts of gestures; how the more specific meaning of children’s gestures emerges from an interplay between gestural form, coordination with speech, previous utterances – and other factors” (pp. 332–333). For a study on the use of gestures in joint book reading situations with children at the age of 14 months, Reference Grimminger and RohlfingGrimminger and Rohlfing (2019) annotated all utterances and deictic gestures, specifying as to whether they were pointing, showing, or giving an object, and annotated their semantic relation with the speech (as reinforcing or supplementary). With the coding, the authors wanted to investigate whether the use of pointing gestures in combination with verbal utterances of children and mothers positively affected the lexicon of the children. Based on these annotations, the authors discovered that 98% of the deictic gestures were pointing gestures and that these pointing gestures indeed correlated significantly with the lexicon of the children at 18 months (p. 96). In an exploratory study on the status and evolution of actions, gestures, and words expressing negation between ages 1 and 4, Reference Beaupoil-Hourdel, Morgenstern, Boutet, Larrivée and LeeBeaupoil-Hourdel et al. (2016) developed a coding system combining Excel and Clan, “a program … designed specifically to analyze data transcribed in the CHAT format” (Reference MacWhinneyMacWhinney, 2000, p. 8). With the system, the authors coded all negative communicative acts, the channels of their expression and perception, and their semiotic status, and also distinguished between actions, that is, “movement produced by the child as a reaction to the environment rather than being intentional and conventionalized” (Reference Beaupoil-Hourdel, Morgenstern, Boutet, Larrivée and LeeBeaupoil-Hourdel et al., 2016, p. 101), and gestures. In doing so, they were able to conduct macro- and micro-analyses of the children’s negative occurrences. Based on the system, Reference Beaupoil-Hourdel, Morgenstern, Boutet, Larrivée and LeeBeaupoil-Hourdel et al. (2016) showed that “gestures used prelinguistically are qualitatively different from gestures used once speech is already elaborate” and that there are “different uses of multimodality and therefore, the status of multimodality changes throughout the corpus” (p. 27). Investigating children’s oral and multimodal discourse, and in particular, “linguistic and gesture production of narratives performed by children and adults of different languages, with emphasis [on] the relationship between speech and gesture and how it develops” (p. 59), Reference Colletta, Kunene, Venouil, Kaufmann and SimonColletta, Kunene, Venouil, Kaufmann and Simon (2008) used an annotation system that analyzes speech and gestures in terms of four different blocks: (1) speech transcription and syntactic analysis, (2) narrative analysis, (3) annotation of gestures, and (4) evaluation. For the gestures, they concentrated on gesture phases, the type of gesture (deictic, representational, performative, framing, discursive, interactive, word searching), the gestures’ relation with speech (reinforcing, complementing, supplementing, integrating, contradicting, substituting), the temporal placement of the gestures in relation to speech, and the gestures’ form (hand shape, movement trajectory, and manner). With the annotation system applied in ELAN, the authors point out that it is possible to “identify, count and describe concrete versus abstract representational gestures, marking of connectives, syntactic subordination, the anaphoric recoveries, hesitation phenomena, etc. as well as to study narrative behaviour from a multimodal perspective” (Reference Colletta, Kunene, Venouil, Kaufmann and SimonColletta et al., 2008, p. 66).

Comparing these four examples, it becomes clear that in order to investigate the relation of speech and gestures in language development at different ages and stages, no single approach to the coding and annotation can be taken, as the particular research question always calls for a specific focus that concentrates on certain aspects while neglecting others. As such, the perspective may vary, for instance, in the depth of the analysis both for gestures and speech (micro vs. macro) as may the research perspective (exploratory vs. guided by hypotheses). Both of these aspects highly influence the analysis and thus the results. An open coding procedure, such as the one applied by Reference AndrénAndrén (2010), for instance, leads to the adaptation of categories during the coding and annotation process, allowing for the incorporation of initially unexpected phenomena, thus making for an exploratory investigation. A rather narrow focus on a particular type of gestural phenomenon (deictic gestures), such as the one adapted by Reference Grimminger and RohlfingGrimminger and Rohlfing (2019), does not and is not meant to allow for the exploration of unincorporated phenomena. Accordingly, insights from both analytical and methodological procedures vary by providing different kinds of access to verbo-gestural phenomena of language acquisition.

3.3 Gestures as Windows onto Thinking: Coding and Annotation for Insights into Cognition

Numerous studies show that when people speak, they have, what Kita and Essegeby (2001) call, a “cognitive urge” to gesture, as gestures help us talk and think. They boost language production and comprehension processes (Reference De Ruiter and McNeillDe Ruiter, 2000; Reference Krauss, Chen, Gottesman and McNeillKrauss, Chen, & Gottesman, 2000), reduce cognitive load, are a tool for exploring reasoning and thinking, and especially reveal aspects of children’s cognitive development (see e.g. Reference Goldin-Meadow, Wagner Cook, Holyoak and MorrisonGoldin-Meadow & Wagner Cook, 2012) (see also the chapters by Alibali & Hostetter and Novak & Goldin-Meadow in this volume). In order to explore these aspects, studies addressing the relation between speech, gestures, and cognition are confronted with similar challenges to those already discussed in Section 3.2 on language use, however with slightly different foci. For studying gestures as windows onto the cognitive development of children, for instance, it is crucial to investigate the temporal and semantic relation between the relation of speech and gestures and to examine whether children are producing ‘mismatches’ between them (Reference Alibali and Goldin-MeadowAlibali & Goldin-Meadow, 1993). As a result, such studies and their coding and annotation practices focus on marking of the temporal movement phases of gestures, their semantic function (complementary or contradictory relation), the type of gestures, and aspects of their form (see e.g. Reference Alibali, Spencer, Knox and KitaAlibali, Spencer, Knox, & Kita, 2011). In order to be able to describe the form in more detail, Reference Hilliard and CookHilliard and Cook (2017), for instance, developed a coding system that, in combination with other existing systems, allows for a description of gestural forms that is suited to addressing gesture’s role in cognitive development (Reference Congdon, Novack and Goldin-MeadowCongdon, Novack, & Goldin-Meadow, 2018). Similar foci are also visible in studies addressing questions of cognitive load (Reference Wagner, Nusbaum and Goldin-MeadowWagner, Nusbaum, & Goldin-Meadow, 2004), yet, as mentioned before, approaches to coding and annotation may vary. Reference Chu and KitaChu and Kita (2008), for instance, in their study on mental rotation problems, pursue a different path that focuses not so much on the semantic role and forms of the gestures as on their functional role. Assuming that “when solving novel problems concerning the physical world, adults may start with bodily exploration of the physical world” (Reference Chu and KitaChu & Kita, 2008, p. 708), thus using gestures significantly, the authors coded gestures according to their movement segments and their location in gesture space, and they used a functional gesture classification suited to the tasks at hand (see e.g. hand–object interaction gestures, object-movement gestures, tracing gestures, rotation direction gestures). This functional classification of the gestures was brought together with an analysis of the speech focusing on agentivity, for instance. Based on this, the authors “concluded that the motor strategy becomes less dependent on agentive action on the object, and also becomes internalized over the course of the experiment, and that gesture facilitates the former process” (Reference Chu and KitaChu & Kita, 2008, p. 706).

Again, other foci are visible in studies investigating the relation between speech and gestures in light of production. In a study on speech disfluencies, Reference SeyfeddinipurSeyfeddinipur (2006), for example, pursues a strongly form-based perspective in addressing the question of “whether speech-accompanying gestures are sensitive to speech disfluency and whether gesture can provide evidence for the self-monitoring process in speech production” (p. 81). In order to account for this, Seyfeddinipur develops a particular method for describing gestural movement phases that allows for a detailed coding and annotation of different movement segments and gesture phases, and thus for close examination of speech and gesture in disfluencies.

This short discussion reveals that practices in coding and annotation, also with respect to gesture and cognition, vary greatly due to the vast range of possible research questions and the assumptions connected with them. A system that hopes to provide a perspective that bridges different interests on gesture and cognition is NEUROGES (Reference Lausberg and SloetjesLausberg & Sloetjes, 2009, Reference Lausberg and Sloetjes2015). It is a “research tool for the analysis of hand movement behavior, including gesture, self-touch, shifts, and actions” and assumes that “main kinetic and functional gesture categories are differentially associated with specific cognitive (spatial cognition, language, praxis), emotional, and interactive functions” (Reference Lausberg and SloetjesLausberg & Sloetjes, 2009, p. 1). The scheme implies that different gesture categories may be generated in different brain areas. NEUROGES is composed of three modules: (1) kinetic gesture coding, (2) bimanual relation coding, and (3) functional gesture coding. Module 1 refers to the kinetic features of a hand movement, that is, execution of movement versus no movement, trajectory and dynamics of movements, location of acting as well as contact with the body or not. Module 2 allows for the coding of bimanual relation (e.g. in touch vs. separate, symmetrical vs. complementary, independent vs. dominance). Module 3 brings in the functional aspects and determines the meaning of gestures based on a specific combination of kinetic features (hand shape, orientation, path of movement, effort, and others), which define the various gesture types. NEUROGES offers the option of analyzing gestures independently of speech, and “since it enables the investigation of processes that are not verbalized, its specific potential lies in the exploration of implicit cognitive, emotional, and interactive processes that may be conducted beyond awareness” (Reference Lausberg and SloetjesLausberg & Sloetjes, 2015, p. 2). NEUROGES also offers an ELAN template including controlled vocabularies and template files for statistical analysis, such as in SPSS.

The inclusion in NEUROGES of options for statistical analyses and, in particular, intercoder reliability, can be said to be a strong component in all systems addressing cognition, regardless of their particular research question. Reasons for including these features lie in the fact that studies on cognition usually follow a quantitative research perspective in an experimental setting. Categories for coding and annotation are thus normally designed as less variable and flexible and aim at the testing of given hypotheses using replicable methods and larger amounts of data. With these characteristics, the systems differ greatly from the majority discussed in Section 3.1 and, even more so, from the majority of those designed for exploring interaction, like those discussed in Section 3.4.

3.4 Gestures as Part of a Bodily Ensemble: Coding and Annotation for Exploring Interaction

Gestures, among other bodily movements, are one of the multimodal resources that speakers use in constructing and negotiating interaction. For an analysis of spoken language, approaches from an interactional and/or conversational point of view argue that their holistic and situated role in building human action has to be explained (Reference MondadaMondada, 2016) and that these resources need to be an integral part of the description of interactional practices (Reference Deppermann, Deppermann and ReinekeDeppermann, 2018). Thus, “the embodied way in which people communicate and gather together, as well as the ecology of the activities they engage in, and their material and spatial environment” (Reference MondadaMondada, 2016, p. 337) have to be made part of analyses. For this, actions have to be documented in their temporal and sequential order and the coparticipants’ orientation to these multimodal actions need to be captured. This “embodied turn” (Reference NevileNevile, 2015) in the study of human and social interaction results in theoretical and methodological challenges as to how to deal with the role of video material within such analysis. In general, two methodological lines may be distinguished on how this problem is solved.

On the one hand, some approaches follow a rather “classic,” qualitative conversational approach to the transcription and analysis and include video stills in the verbal transcript (Reference Stukenbrock, Birkner and StukenbrockStukenbrock, 2009). Transcripts contain the trajectories, temporal relations between speaking and other forms of visual communication, and qualities of the multiple resources from the perspective of the doers, meaning “all possibly relevant embodied actions, such as gesture, gaze, body posture, movements, etc. that happen simultaneously to talk or during moments of absence of talk” (Reference MondadaMondada, 2014, p. 1). Embodied actions are described briefly without focusing too much on their physical appearance, varying in the level of detailed description depending on the particular research question. An example illustrating this approach is found in a study on the arrival of guests at a dinner party. Here, Reference Oloff and SchmittOloff (2010) demonstrates how interactants orient themselves toward the sequentiality of the activity of waiting, and by doing so, jointly construct the beginning of the dinner. For the analysis, Reference Oloff and SchmittOloff (2010) concentrates on a precise transcription and coding of the sequentiality and temporality and short description of the bodily actions overlapping with speech (see e.g. “holds up hands,” “opens up bag,” “looks at watch”) and the spatial layout and arrangement of the room along with movements of interactants in space. Using this approach to annotation and coding, Reference Oloff and SchmittOloff (2010, p. 220) uncovers a particular temporal sequence of the individual phases of the activity of waiting and the bodily actions contained therein. In a similar yet slightly different vein, Reference SchmittSchmitt (2005) approaches the turn-taking mechanism from a multimodal perspective. The transcription follows the general principle of including bodily behavior in verbal transcripts by inserting still images. However, short descriptions of the gestures, for instance, are not included in the transcript. Rather, the still image is complemented by a short functional description in the analysis (see e.g. “She starts eye-contact with him, simultaneously lifts her left arm, stretches it completely and realizes a clear signal gesture with her arm and hand” [Reference SchmittSchmitt, 2005, p. 34], translation JB). Following this perspective, Schmitt illustrates that not only the turn and its construction are of interest to conversational analyses but so too is the status of the current speaker as an accomplishment of the other interactants.

On the other hand, studies may also follow a more “technical” way to exploring the body’s role in interaction. Often pursuing a combination of qualitative and quantitative designs, numerous studies include annotation programs for the coding and analysis of gestures. Video material thus assumes a specific role and is exploited differently in the transcription and annotation process than it was in the studies described above. Rather than concentrating on the whole temporal continuity of actions, these studies focus on particular facets of interaction and their relation with gestures and other bodily actions. Using data from head-mounted eye-trackers in a corpus of face-to-face conversations including various conditions, Reference Oben and BrôneOben and Brône (2015), for instance, explore whether gaze “fixations by speakers and fixations by interlocutors have [an effect] on subsequent gesture production by those interlocutors” (p. 546). Focusing on depictive gestures, the authors did not concentrate on a detailed description of the gestures, but rather measured how similar two gestures were to each other. In combination with the coding of regions of interest and fixation duration of the eye-gaze, the authors demonstrate that there is a “significant effect of interlocutor gaze, but not of speaker gaze on the amount of gestural alignment” (Reference Oben and BrôneOben & Brône, 2015, p. 546). Another example combining a corpus-based, bottom-up approach rooted in gesture studies and interactional linguistics exemplifying this second kind of approach is a study by Reference Debras and CienkiDebras and Cienki (2012). In order to account for “the possible functions of two types of gesture during stancetaking in the course of human-human interaction” (p. 932) (lateral head tilts and shoulder shrugs), the authors used a combination of coding and annotation in ELAN and Excel to retrieve the functions of the gestures during conversations. For this, the gestures were first described and annotated for their forms without listening to speech, and in a second step, coded in Excel in relation to verbal and other types of bodily behavior. Using this combination, that allows for a synergy of qualitative and quantitative perspectives, the authors demonstrate that “participants tend to use the two gestures in a similar way when positioning themselves with respect to a prior stance: either to affiliate with their conversation partner’s stance, or to disaffiliate with a third party’s positioning” (Reference Debras and CienkiDebras & Cienki, 2012, p. 937).

Whereas the “classical” approach to interaction analysis gives a more comprehensive understanding of the action of the interactants and of the interaction itself, the last two examples zoom in on particular gestural phenomena in relation to regulating and constructing interactional processes. Both examples therefore illustrate again the point repeatedly made throughout this chapter: Even though research interests may be similar, the methodological approaches and the perspectives taken may differ immensely and thus yield very diverse insights into, for example, interaction.

3.5 Making Gestures Reproducible: Coding and Annotation for Human–Machine Interaction

Other than in the examples given above, in which differences in the approach to coding and annotation are deeply rooted in theoretical assumptions, schemes and practices for coding and annotating human–machine interaction are guided first and foremost by technical requirements to make gestures reproducible. That is, these systems aim at reproducing the kinematics of human gestures along with the gestures’ relation to speech by “imitating” a human speaker’s gesture behavior (Reference Kim, Ha, Bien and ParkKim, Ha, Bien, & Park, 2012; Reference Kipp, Neff and AlbrechtKipp et al., 2007; Reference Theofilis, Nehaniv and DautenhahnTheofilis, Nehaniv, & Dautenhahn, 2014; see also the chapter by Jokinen, this volume). For achieving this, systems have to answer three main questions: (1) When do robots need to generate human-like behavior? (2) What human-like behavior needs to be generated? (3) How is it possible to generate human-like behavior (Reference Kim, Ha, Bien and ParkKim et al., 2012)? As a result, these systems face similar obstacles to the ones discussed in the sections above: Movement phases as well as form characteristics of the gestures need to be identified and coded, the types of gesture have to be accounted for, and of course speech and gestures need to be set in relation to each other. In a last step, these systems face a challenge that is not addressed in the ones discussed so far, namely the evaluation of the coding and annotation by recreating the human gestures; that is, the question of how well the annotation reflects the original gesture. How systems tackle those questions and challenges varies. Human gestures are either manually coded or captured by using motion capture data (for an overview see Reference Kim, Ha, Bien and ParkKim et al., 2012; Trujillo, this volume). Systems thereby concentrate on the hand shape of the gestures, their movement patterns, as well as their spatial arrangement along with their relation to speech (see e.g. Reference Kim, Ha, Bien and ParkKim et al., 2012; Reference Kipp, Neff and AlbrechtKipp et al., 2007; Reference Kopp and WachsmuthKopp & Wachsmuth, 2004; Reference MartellMartell, 2005). However, this approach is very time-consuming and a lot of the complexity, especially of the original movement, is lost. As a result, newer approaches to coding and annotation try to overcome this shortfall by making use of automatic gesture detection (see e.g. Reference Madeo, Lima and PeresMadeo, Lima, & Peres, 2017; Reference Spiro, Taylor, Williams and BreglerSpiro, Taylor, Williams, & Bregler, 2010; Reference Turchyn, Moreno, Cánovas, Steen, Turner, Valenzuela and RayTurchyn et al., 2018).Footnote ⁵ With these efforts, the systems not only wish to make generation of gestures more accurate for human–machine interaction but also aid gesture research in general by providing support for the coding and annotation of gestures.

1.1 Methods for Gesture Analysis: Overview

MGA distinguishes microlevel and macrolevel analysis. The baseline for MGA is a microanalysis that entails some account of the gesture as temporal form, and some analysis of how a gesture, a series of gestures, or a multimodal sequence is placed in an unfolding context-of-use (Figure 8.1).

Figure 8.1 Baseline of MGA: Microlevel analysis

MGA’s toolbox offers a flexible set of tools for descriptive analyses of hand gestures as temporal forms. Temporality is relevant to analysis on multiple scales: on the microlevel of a single gesture or gesture sequence and on the macrolevel of the unfolding of gesture(s) along the temporal dynamics of a discourse or conversational interaction (Figure 8.2).

Figure 8.2 Different kinds of macrolevel analyses

Which collection of tools is selected depends upon the researcher’s interest and the theoretical framework adopted. The focus of this chapter lies at the microanalytic level. Microanalytic tools constitute the baseline for macrolevel analysis of gesture dynamics. While an introduction of macroanalytic procedures would extend the scope of this chapter beyond its limit, references below point to examples of macroanalytic studies.

Macrolevel analysis may address temporal dynamics of gesture use as it unfolds across a discourse event or conversational interaction or across historical time spans. Such studies may, for example, concern returns and extensions of gestures along an interactive setting, such as a dance class (Reference Müller and KappelhoffMüller & Kappelhoff, 2018; Reference Müller, Ladewig, Borkent, Dancygier and HinnellMüller & Ladewig, 2013); or the study of historical changes of gestures as processes of stabilizations on different timescales. Examples of macrolevel studies are investigations of recurrent gestures and emblems (Reference KendonKendon, 2004, Ch. 13; Reference Ladewig, Müller, Cienki, Fricke, Ladewig, McNeill and BressemLadewig, 2014, this volume; Reference Müller, Müller, Cienki, Fricke, Ladewig, McNeill and BressemMüller, 2014b, Reference Müller2017a). Macroanalysis of gestures may furthermore contribute to comparative studies of gesture and sign (Reference KendonKendon, 2004, Ch. 15; Reference Kendon and Allen2015; Reference MüllerMüller, 2019a; Reference WilcoxWilcox, 2009; Reference Wilcox, Shaffer, Jarque, Segimon, Pizzuto and RossiniWilcox et al., 2000), of gestures across different cultures (Reference Bressem, Stein and WegenerBressem, Stein, & Wegener, 2017; Reference Bressem and WegenerBressem & Wegener, 2021; Reference KendonKendon, 1981; Reference Morris, Collett, Marsh and O’ShaughnessyMorris, Collett, Marsh, & O’Shaughnessy, 1979; Reference Müller, Müller, Cienki, Fricke, Ladewig, McNeill and BressemMüller, 2014b) or across species (Reference Müller, Liebal, Müller and PikaMüller, 2007).

1.2 Methods for Gesture Analysis: Basic Assumptions

Essential starting points for MGA are an understanding of hand gestures as temporal forms embedded in a dynamically unfolding context; and an understanding of context that itself varies with the adopted framework. Depending on a given research question and its respective theoretical framework, procedures deemed relevant for a given study can be selected from the toolbox. MGA thus offers a flexible set of tools, designed to adjust to variable analytic perspectives and theoretical frameworks. Prerequisite for this flexibility are some basic assumptions concerning the nature and character of gestures outlined in this section.

1. (1) The term “gesture” refers to movements of the body that people use in conjunction with spoken and signed languages to talk about and act within their life-worlds. Gestures are used to articulate thoughts as much as feelings and they are vital in coordinating communicative actions in social interaction and in the flow of discourse.

2. (2) Articulatory gestures. MGA focuses on gestures as hand movements, but can serve as a starting point for the inclusion of other body parts. The focus on hand movements responds to the articulatory, enactive, and mimetic complexity of the hands, which are humans’ foremost tool to act upon the world (Reference StreeckStreeck, 2009, Ch. 3). It is not by accident that movements of the hands play a central role in signed languages. The articulatory freedom of the hand rests upon a physiological flexibility of the human hand that is an evolutionary achievement of highest importance to the development of human culture including language – in whatever modality expressed (Reference Corballis, Müller, Cienki., Fricke, Ladewig, McNeill and TeßendorfCorballis, 2013; Reference Leroi-GourhanLeroi-Gourhan, 1964/1993). The hands display a unique articulatory freedom and richness comparable only to the mouth as a tool and locus of fine articulatory movement; hence, the reason why some scholars speak of spoken language as articulatory gestures (Reference Armstrong, Stokoe and WilcoxArmstrong, Stokoe, & Wilcox, 1995).

3. (3) Gestures are temporal forms. As Kendon points out: “When a person speaks there is always some movement in the body besides the movements of the jaws and lips that are directly involved in speech production” (Reference Kendon and Ritchie KeyKendon, 1980, p. 207). He put forward a description of gestures as movements that unfold in time and whose movement phases typically go from preparation – to stroke – to retraction (Reference Kendon and Ritchie KeyKendon, 1980; Reference Kendon2004, Ch. 7). Conceiving of gestures as a temporal form has since become “common sense” in the field of gesture studies; advancements of the systematics include Reference Bressem and LadewigBressem and Ladewig (2011); Reference Kita, van Gijn, van der Hulst, Wachsmuth and FröhlichKita, van Gijn, and van der Hulst (1998). Starting the analysis of gestures from their temporal nature makes a lot of sense because the recognition of gesture boundaries (i.e. specifying where a gesture begins and where it ends) determines units of analysis in the first place: In quantitative studies, it enables reliable counting, and, in qualitative studies, it creates the object of analysis that then becomes subject to further descriptive (i.e. interactional, conceptual, semiotic, semantic, pragmatic, syntactic, etc.) analysis.

4. (4) Gestures are embedded in dynamic contexts-of-use. As temporal forms, hand gestures are integrated in the contexts in which they are used. No matter how the notion of context is conceived (cf. Section 4), they are temporally unfolding phenomena; in short, they are dynamic. MGA takes account of these temporal qualities: Note, however, that the separation of gestures as temporal forms from the temporal nature of contexts-of-use is purely an analytic one (Figure 8.1). Gesture, speech, and whole-body movements emerge as one multidimensional gestalt when a person is talking and engaging in an interaction. Gesture analysis decomposes an experiential unity. Coparticipants in a conversation perceive and experience gestures and speech as dynamic and multidimensional gestalts, just as film viewers see the acting of an actor, a landscape, or a race evolving in time as an orchestrated movement image (Reference MüllerMüller, 2019b; Reference Müller and KappelhoffMüller & Kappelhoff, 2018). The type of decomposition of this gestalt is a consequence of the analytical focus and theoretical approach adopted. It is important to note this because any analysis will at some point have to reflect on how the decomposed aspects of gesture forms and contexts relate to the multidimensional unity of experience that characterizes speaking, gesturing, and understanding.

5. (5) Why microanalysis of the form of gestures is necessary. Why not, if gestures are part and parcel of holistic gestalts, simply skip the laborious analyses of gestures as temporal forms? The answer is that not only does a close form analysis provide a solid ground for seeing gestures as immersed in multidimensional gestalts, but it also prevents researchers from reading meanings ‘into’ the gestures with no substantiation apart from intuition. For example, some analyses of multimodal constructions start from a linguistic form and look for co-occurring gestures without carrying out systematic analyses of the gestural form (Reference SchoonjansSchoonjans, 2018). Often, a gesture form analysis is deemed not necessary because the meaning of the gesture is treated as obvious. To highlight the potential relevance of a close gesture analysis, Reference Bressem and MüllerBressem and Müller (2017) have suggested a gesture-first approach to the analysis of multimodal constructions, that is, starting from recurring gesture forms to identify potential multimodal constructions (see also Reference Mittelberg, Gonzalez-Marquez, Mittelberg, Coulson and SpiveyMittelberg, 2007). However, no matter whether language-first or gesture-first, a close analysis of gesture forms is an essential baseline and provides firm grounds for the analysis.

6. (6) What is considered “context” for gesture analysis. In gesture studies, “context” or “context-of-use” is mostly used in a rather nonspecific way (exceptions are Reference Kendon, Duranti and GoodwinKendon, 1992; Reference StreeckStreeck, 2009). Moreover, often quite different understandings of context-of-use are implied when using terms such as “multimodality,” “multimodal communication,” “multimodal interaction,” “multimodal utterance,” and so on. The term “multimodality” covers a wide range of different theoretical frameworks with sometimes mutually exclusive concepts of gestures and their contexts. Against this backdrop, specifying the particular theoretical framework and its associated concept of “context” appears as a useful analytic step that directs the researcher’s attention to a critical reflection upon their research focus and the approach they have adopted.

In a nutshell. MGA as a toolkit enables the researcher to work out which kind of form analysis is appropriate and which understanding of context is relevant. It also encourages the researcher to step back and consider the theoretical approach adopted as one of several possible perspectives on gesture analysis.

3.1 Establishing the Temporal Unit of Analysis

The first tool concerns gesture boundaries, for example, it addresses the establishment of units of analysis. Deciding where a gesture begins and ends may, at first sight, seem obvious. Consider, for example, the “round shape gesture” (G1), from Luis’ story. Here the boundaries of the gesture are clear-cut: the gestural movement unfolds from a rest position, where both hands are resting on the speaker’s lap, moves upward in a preparation phase, performs a stroke (e.g. the molding round shape movement) and then, with a phase of retraction, returns back to the rest position on his lap (Figure 8.7).

Figure 8.7 G1 as temporal unit

However, even with this apparently simple temporal gestural movement, questions arise as to what is included and excluded. In Kendon’s influential systematics of “gesture units, gesture phrases and the phases of gestural action” (Reference KendonKendon, 2004, p. 111), the temporal unit that is considered as the gesture would only comprise part of the movement described above (G1). In McNeill’s notation system, applied in extract (2) (Figure 8.5), the temporal unit would include the return to rest position. Kendon makes a distinction between gesture phase (preparation, stroke, recovery), gesture phrase, and gesture unit. A Kendonian gesture phrase is defined as preparation plus stroke (including poststroke holds) and does not include the phase of recovery and return to rest position (Reference KendonKendon, 2004, p. 112). Gesture units may thus contain sequences of gestures (e.g. gesture phrases). Figure 8.8 shows a gesture unit as a sequence of two gesture phrases: a succession of outlining (G2) and placing (G3) movements from the story-telling.

Figure 8.8 A Kendonian gesture unit with two gesture phrases

Between outlining a round frame (G2) and placing a crown on top (G3), one hand returns to rest position. The right hand stays up and moves directly to the position in the gesture space where the stroke (e.g. the placing movement) is made. After that, the right hand returns to rest position too. In Reference KendonKendon’s (2004, p. 112) terms, the gesture unit thus entails two gesture phrases (G2) and (G3), each characterized by a succession of preparation (hand moves into the gesture space) and stroke phase (the phase where the movement reaches its apex and is most clearly articulated, in terms of effort and shape). The concepts of effort and shape are taken from Laban Movement Analysis (Reference Bartenieff and LewisBartenieff & Lewis, 1980; Reference Kennedy, Müller, Cienki., Fricke, Ladewig, McNeill and TeßendorfKennedy, 2013). Applying the tool “establishing the temporal unit of analysis” reveals the internal linear complexity of gestures as temporal forms, which is central to the coordination of gesture and speech.

3.2 Distinguishing Pointing from Depictive and Pragmatic Gestures

For an analysis of different kinds of gestures, different analytic tools might become relevant. Therefore, a basic two-way distinction is made between pointing versus depictive and pragmatic gestures. This distinction refers to level 2 in Figure 8.6.

What motivates the distinction between pointing and other gestures in the first place? Without going into the details of gesture classification systems, it can be seen that a distinction between pointing (or deictics as per Reference EfronEfron, 1941/1972; Reference Fricke, Müller, Cienki, Fricke, Ladewig, McNeill and BressemFricke, 2014, this volume; Reference McNeillMcNeill, 1992, p. 18) and other kinds of gestures appears to be widely accepted in gesture studies. Overviews and discussions of gesture classifications have been offered by several researchers (Reference AndrénAndrén, 2010, pp. 96–105; Reference FrickeFricke, 2007, pp. 156–181; Reference GullbergGullberg, 1998, pp. 47–51; Reference KendonKendon, 2004, Ch. 2; Reference McNeillMcNeill, 1992, Ch. 3; Reference MüllerMüller, 1998b, pp. 91–113). The distinction between depictive and pragmatic gestures is informed by Reference KendonKendon (2004), although “depictive gestures” is used in MGA instead of his “representational gestures” (Reference KendonKendon, 2004, p. 160; Reference StreeckStreeck, 2009, Ch. 6). The term “pragmatic gestures” is used in Reference KendonKendon’s (2004, pp. 158–159) sense. The distinction between depictive and pragmatic gestures is basic because it reflects two essentials of language: reference to the world talked about and communicative action. In using one or the other type of gesture, interactive attention is drawn either to communicative action or to depiction (this holds, notwithstanding that every depiction implies acting communicatively) (Reference MüllerMüller, 2015).

Note that in the McNeillian tradition, depictive gestures are called “iconics.” “Metaphorics” in the McNeillian tradition are depictive gestures used metaphorically as well as pragmatic gestures. Other classifications, however, suggest a further distinction between gestures used to depict some abstract concept and gestures used with a pragmatic function. With respect to depictive gestures used metaphorically, Reference EfronEfron (1941/1972) speaks of ideographic, Reference MüllerMüller (1998b) of abstract referential, Cienki and Reference MüllerMüller (2008a, Reference Cienki, Müller and Gibbs2008b) of metaphoric, and Reference StreeckStreeck (2009, Ch. 7) of ceiving gestures. When describing pragmatic gestures in Kendon’s sense, by contrast, Reference StreeckStreeck (2009, Ch. 8) speaks of speech-handling, and Reference MüllerMüller (1998b) of performative and discursive gestures.

Distinguishing pointing from other kinds of gestures reflects a fundamental semiotic distinction. While pointing gestures are primarily based on indexicality, for other gestures, iconicity plays an important role; although they typically also involve indexical elements (see Fricke, this volume). Mittelberg’s work on the entanglement of metonymic (as indexical) and metaphoric (as iconic) elements of gesture forms disentangles this complexity both theoretically and as a way to approach gesture form analysis (Reference MittelbergMittelberg, 2019b). In semiotic theory, indexicality and iconicity are complex and widely debated issues and are highly relevant to gesture analysis (cf. Reference FrickeFricke, 2007; Reference Mittelberg, Müller, Cienki, Fricke, Ladewig, McNeill and BressemMittelberg, 2014).

Distinguishing pointing gestures from depictive and pragmatic ones responds to differences in kinesic form and in embodied motivation. Only depictive and pragmatic gestures can be accounted for as as-if actions, that is, in terms of their embodied motivation.

The form-based distinction of pointing as opposed to depictive and pragmatic gestures rests upon characteristics of hand shape and movement. Typical pointing hand shapes are the extended index finger, the palm lateral hand, pointing with the little finger, or pointing with the lips (Reference FrickeFricke, 2007; Reference KendonKendon, 2004, pp. 199–200; Reference KitaKita, 2003; Reference SherzerSherzer, 1973). Reference KendonKendon (2004, p. 200) describes how pointing gestures tend to show a characteristic movement pattern “in which the body part carrying out the pointing is moved in a well-defined path, and the dynamics of the movement are such that at least the final path of the movement is linear. Commonly, but not always, once the body part doing the pointing reaches its furthest extent, it is then held in position briefly.” Sometimes, however, pointing may only involve extending the index finger fully. In short, distinguishing pointing from depictive and pragmatic gestures guides further analytic procedures.

While all gestures can be analyzed with regard to aspects of their kinesic form (cf. Section 3.4.1), depictive as well as pragmatic gestures are special in that they are enactments of different kinds of as-if actions (cf. Section 3.3). Attempting to reconstruct which as-if action is performed with a given gesture offers a path not only to the semiotic grounding of the gesture but to an experiential base of the intersubjectivity of embodied understanding (Reference Müller, Zlatev, Sonesson and KonderakMüller, 2016, Reference Müller2019b).

Considering the example above, Luis uses no pointing gestures in his story-telling. In the following, the six gestures he uses illustrate how the different tools for gesture analysis reveal diverse aspects of gestural movements for depictive gestures.

3.3 Analyzing Depictive and Pragmatic Gestures as As-If Actions

This set of tools for gesture form analysis addresses level 3 in Figure 8.6. It concerns the gesturing hands as as-if actions. Experience has shown that the set of tools figuring under the rubric of “Gestures as As-If Actions” offers an excellent starting point for analyzing depictive as well as pragmatic gestures. Often this works best when turning off the sound and trying to answer the question: What kind of as-if action is the gesture carrying out?

The MGA toolbox distinguishes four basic kinds of as-if actions that are applicable to reconstruct a heuristic of the experiential base of depictive and pragmatic gestures: The hands act as if performing a practical action with or without an imagined object, the hands act as if molding an ephemeral object, the hands act as if drawing the shape of an object or the line of a path, the hands act as if they were an object (Figure 8.9).

Figure 8.9 As-if actions as modes of gestural mimesis

In Luis’ narration all four are applied: In G1 the speaker acts as if molding an ephemeral round object, in G2 he acts as if drawing a round shape, in G3 he acts as if placing a small object on top of the ephemeral round shape just outlined, in G4 he acts as if his flat hands were some flat object located vertically at the edge of the gesture space, in G5 he acts as-if the hand were some object falling down quickly.

These four different as-if actions involve different kinds of bodily experiences: acting, molding, and drawing are based on haptic manual experiences, and often involve imagined objects; in contrast, when the hands are used as if they were an object, this transformation more likely involves visual perception. We can only mention in passing that these different embodied motivations involve different conceptual viewpoints (Reference Dancygier and SweetserDancygier & Sweetser, 2012; Reference McNeillMcNeill, 1992; Reference Parrill, Lavanty, Bennett and KlcoParrill, Lavanty, Bennett, Klco, & Demir-Lira, 2018; Reference Stec, Sweetser, Dancygier, Wei-Iun and VerhagenStec & Sweetser, 2016). Acting as-if involves a character viewpoint, whereas molding and drawing may imply a character or an observer viewpoint, depending on the focus of the action or the depicted object. Hands representing an object typically involve the perspective of an observer (Reference Bressem, Ladewig, Müller, Hübl and SteinbachBressem, Ladewig, & Müller, 2018). Note that, in most cases, the characterization of an experiential base will be a heuristic assumption, but, if longer segments of talk are considered, the analyst may be able to observe how gestures emerge from practical actions through abstraction and schematization over the course of communicative interactions.

To analyze gestures in terms of as-if actions addresses the iconicity of gestural form in depictive and pragmatic gestures in terms of their embodied motivation. This perspective on gesture form analysis connects with a theoretical perspective that grounds intersubjectivity of understanding (Cuffari, this volume) in the embodied perception of the moving body (Reference MüllerMüller, 2019b). I have discussed this aspect of gesture form analysis in publications over the past two decades under different labels: iconicity (Reference Müller and SantiMüller, 1998a), modes of representation (Reference MüllerMüller, 1998b, Reference Müller, Müller, Cienki, Fricke, Ladewig, McNeill and Bressem2014a, Reference Müller2017a), and modes of mimesis (Reference Müller, Koch, Voss and VöhlerMüller, 2010, Reference Müller, Zlatev, Sonesson and Konderak2016). The different terms reflect different perspectives on the nature of as-if actions and their grounding of gestural meaning in different kinds of common manual actions.

Approaching the iconicity of gestures resonates with reflections on iconicity of signs in Sign Language Linguistics (Reference Mandel and FriedmanMandel, 1977; Reference TaubTaub, 2001) and with proposals of signed language classifiers (Reference Müller and MalmkjaerMüller, 2009). In gesture studies it connects with Kendon’s discussion of gesture and sign (Reference KendonKendon, 2004, Ch. 15), with Mittelberg’s cognitive-semiotic approach to gesture form (Reference MittelbergMittelberg, 2019a, Reference Mittelberg2019b) and with Reference ZlatevZlatev’s (2014a, Reference Zlatev2014b) work on mimesis, among other things.

Conceiving of gesture forms as as-if actions draws analytic attention to the fact that both depictive and pragmatic gestures operate upon the mode of “as-if.” What is so special about the mode of “as-if?” Instead of performing the action of opening a window, speakers act as if their hands opened a window. The window handle is an imagined handle; the hand shape and movement mime the actual action. Instead of performing the action of physically showing some object on the open hand, speakers act as if some argument were displayed on their open hand. This transformation from object manipulation to acting as if and upon virtual objects characterizes depictive as well as pragmatic gestures. It makes them suitable forms to communicate in the absence of things, actions, and events being talked about.

Some scholars describe pragmatic gestures as “metaphoric” (Reference McNeillMcNeill, 1992) or “speech-handling” (Reference StreeckStreeck, 2009). This terminology risks masking the fact that both depictive and pragmatic gestures are as-if actions (Reference Cienki, Müller, Müller, Cienki, Fricke, Ladewig, McNeill and BressemCienki & Müller, 2014). Both act upon virtual objects and both are recognizable as as-if action through their abstracted and abbreviated performance of a manual action. Moreover, as we know from metaphor research, gestures are frequently used to depict abstract concepts. Accordingly, the term “metaphoric gesture” is reserved here/in MGA for depictive gestures used to express the bodily base of abstract concepts (Cienki & Reference MüllerMüller, 2008a; Reference MüllerMüller, 2008). Consider the following example: In an interview, former US president Obama characterizes the story of America as a kind of “battle” between different ideas of democracy and while saying this he moves his hands and arms as if boxing. His gesture thus depicts the abstract concept of “battle” in terms of a boxing match. Such subjective bodily imaginations are considered here as a bodily base of an abstract concept: Here the gesture metaphorically depicts a political battle as boxing.

When considering pragmatic gestures more closely, we see that it makes sense to conceive of them as as-if actions. Often, experiential roots in practical actions of the hands are quite obvious. Streeck, for example, describes speech-handling gestures as as-if actions (Reference StreeckStreeck, 2009, Ch. 8), where the hands act as if manipulating some imagined object and function as handing over the speech to the next speaker. Pragmatic gestures thus present communicative actions as if they were actions operating upon imagined objects. Examples are the palm-up-open hand (e.g. presenting something as obvious, by acting as if some abstract object sat on the open palm, visible to everybody; cf. Reference Müller, Müller and PosnerMüller, 2004) and the family of “throwing away” gestures (e.g. a dismissive movement of the hand, acting as if throwing away some middle-sized object; cf. Reference Müller, Müller, Cienki, Fricke, Ladewig, McNeill and BressemBressem & Müller, 2014a, Reference Bressem, Müller, Müller, Cienki, Fricke, Ladewig, McNeill and Bressem2014b; Reference MüllerMüller, 2017a; Reference Teßendorf, Müller, Cienki, Fricke, Ladewig, McNeill and BressemTeßendorf, 2013).

To recap: regarding gestures as as-if actions applies to depictive and pragmatic gestures. While they share the mode of as-if, they differ regarding their communicative function: Depictive gestures embody what is talked about, pragmatic gestures incorporate and perform communicative action.

3.4 Analyzing Hand Gestures as Movement

Analyzing gesture forms as movement concerns depictive, pragmatic, and pointing gestures as well. It addresses the fact that, no matter what, hand gestures are movements of the body. To analyze hand gestures as movement presupposes, however, a specific understanding of body movement. Therefore, the toolbox of MGA offers three different conceptions of accounting for the hands as body movements: kinesic form, motion events, and expressive movement. The three perspectives capture different facets of the gestural movement and are not mutually exclusive in the analytic process. Importantly, the selection is not deemed exhaustive. It complements approaches such as Boutet’s kinesiology of gesture and sign (Boutet, this volume; Reference Boutet, Morgenstern and CienkiBoutet, Morgenstern, & Cienki, 2016, Reference Boutet, Morgenstern, Cienki, Cienki and Iriskhanova2018) and may be supplemented further. This set of tools for gesture form analysis targets level 4 in Figure 8.6.

3.4.1 Kinesic Aspects of Gesture Form

We distinguish four aspects of kinesic form that are simultaneously articulated: hand shape, orientation, movement, and location (level 4 in Figure 8.6, Figure 8.10). Accounting for these aspects of kinesic form targets a level of gesture analysis that is comparable to proposals for the description of sign language phonology. It is inspired by observations concerning sublexical structures of signs (Reference StokoeStokoe, 1960; Reference Wilcox and OcchinoWilcox & Occhino, 2016, p. 4). We know from signed languages how important even minor changes of one of these articulatory characteristics for the meaning and function of a sign may be. But how does this apply to gesture analysis?

Figure 8.10 Aspects of kinesic form that are simultaneously articulated

Let us reconsider the gestures used in Luis’ story-telling. Quite clearly, the speaker does not draw upon a lexicon of gesture “lexemes.” For example, five of the gestures show an interesting difference with regard to the location, that is, where in the gesture space (Reference FrickeFricke, 2007; Reference McNeillMcNeill, 1992) they are performed. While most gestures (G1, G2, G3, G5) are located in front of the speaker’s body, G4 is performed at the far upper left periphery of the gesture space. This is interesting and may trigger further research questions concerning how the body space is used in gesturing. In this example, it is clear that there is a difference between a semantically relevant location of gestures and a neutral gesture space. While the periphery of the gesture space evokes the actual location of the picture in another room and is thus part of a narrative space, the placement of the other three gestures is not part of the narrative space: For example, the picture in the story was not located in front of the speaker’s chest nor did it fall down in front of his body. Instead, the space is used as a kind of neutral space to depict the shape and falling-down of the picture.

The set of tools marked as “Aspects of kinesic form” may thus become relevant to the analysis of gestures created on the spot (singular gestures) but it applies also to more or less stabilized gestural forms (recurrent and emblematic gestures; cf. Reference Bressem, Müller, Cienki, Fricke, Ladewig, McNeill and TeßendorfBressem, 2013a; Ladewig, this volume; Reference Müller, Müller, Cienki, Fricke, Ladewig, McNeill and BressemMüller, 2014b, Reference Müller2017a). Applying these tools to gesture analysis opens up pathways for a comparative analysis of co-speech gestures, cosign gestures, and historical changes in conventionalized gestures and signs (Reference MüllerMüller, 2019a).

3.5 Summary

The set of tools offered as methods for gesture form analysis addresses four basic aspects of gestural forms (Figure 8.6): (1) the boundaries of gestures as temporal forms, (2) a distinction of pointing from depictive and pragmatic gestures based on hand shape and movement, (3) the embodied motivation of depictive and pragmatic gestures as “as-if” actions, and (4) three possibilities to approach hands as body movement, that is, aspects of kinesic form, hand gestures as motion events, and hand gestures as expressive movements.

Note that the decision as to which aspects of gestural forms are deemed relevant for a given analysis of gestural forms depends upon the research question and the theoretical framework against which it has been formulated. Depending upon the research question, the analytic process might start from a gesture form analysis without sound (Reference Bressem, Müller, Cienki, Fricke, Ladewig, McNeill and TeßendorfBressem’s 2013a and Reference Bressem2021 work on repetition are examples of this procedure) and only include speech analysis in a second analytic step. Another option is to start with gesture analysis and then move back and forth between a close analysis of the gesture forms and their embedding in contexts-of-use: Examples of this type of procedure are what Reference Bressem and MüllerBressem and Müller (2017) termed “gesture-first analysis of multimodal constructions,” or the analysis of multimodal metaphors (their dynamics and foregrounding, cf. Reference MüllerMüller, 2008; Reference Müller and TagMüller & Tag, 2010). Section 4 illustrates how the theoretical frameworks chosen influence the interpretation of gestural forms as contributions to multimodal utterances.

4.1 Cognitive Linguistics: Context as Usage-Based Grammar and Semantics

A cognitive-linguistic analysis leads to context being considered from the point of usage-based grammar and cognitive semantics (Reference BybeeBybee, 2010; Reference Cienki, Müller, Cienki, Fricke, Ladewig, McNeill and TeßendorfCienki, 2013, Reference Cienki2017; Reference Croft and CruseCroft & Cruse, 2004; Reference FillmoreFillmore, 1985; Reference LangackerLangacker, 2008). Here, the notion of context may refer to specific semantic contexts such as lexicalization patterns of motion events (Reference Talmy and ShopenTalmy, 1985, Reference Talmy1991) but may also address questions of viewpoint (Reference Dancygier and SweetserDancygier & Sweetser, 2012), conceptual blending (Reference Parrill and SweetserParrill & Sweetser, 2004), or metaphor and thought (Reference MüllerCienki & Müller, 2008b). Furthermore, protoforms of grammaticalization in gesture have been subject to a cognitive linguistic interpretation of context-of-use (Reference BressemBressem, 2021; Reference Ladewig, Müller, Cienki, Fricke, Ladewig, McNeill and BressemLadewig, 2014; Reference Müller, Müller and PosnerMüller, 2004, Reference Müller2017a; Reference Müller, Ladewig, Borkent, Dancygier and HinnellMüller & Ladewig, 2013; Reference Müller, Ladewig, Bressem, Müller, Cienki, Fricke, Ladewig, McNeill and TeßendorfMüller et al., 2013), as has the gestural expression of aspectuality in terms of event structure (Reference Cienki and IriskhanovaCienki & Iriskhanova, 2018). Context-of-use in cognitive linguistic terms could also include the grammatical integration of gestures in multimodal utterances (Reference LadewigLadewig, 2020). For a cognitive-semiotic framework for gesture analysis within a larger cognitive-linguistic usage-based approach to context, see Mittelberg’s work (Reference MittelbergMittelberg, 2017, Reference Mittelberg2019a, Reference Mittelberg2019b), or, taking a slightly different angle, Reference AndrénAndrén’s (2010) developmental study.

In the analysis of “hand gestures as movements” (Section 3.4.2), the motio-event perspective is formulated within a cognitive semantic framework (Reference Talmy and ShopenTalmy, 1985). The relevant “context” for gesture analysis is here a typological analysis of lexicalization patterns apparent in the motion verbs that are coarticulated with gestural expressions of motion events. This would lead to seeing Luis’ falling-down gesture G5, which is temporally synchronized with the motion verb se cayó, as expressing a path (downward). Analyzed with a cognitive semantic understanding of “context,” the gesture would be analyzed as following the specific lexicalization pattern of Spanish motion verbs which merge motion and path.

Another potentially relevant context within a cognitive-linguistic framework would consider the grammatical notion of “aspect” as a conceptualization of an event-structure that distinguishes bounded from unbounded, or perfective from imperfective, events. Luis’ gesture G5 would then be considered as a case of a bounded gesture. The analytic focus for the gestural movement would be its marked endpoint. When “aspectuality” is taken as the relevant usage-context for gesture analysis, a given gestural movement can have various functions at the same time. It can express a downward path and at the same time be performed in a bounded or unbounded manner.

These examples illustrate that form and context-analysis merge and are informed by the specific theoretical frameworks chosen; here, Talmy’s cognitive-semantic typology (Reference Talmy and ShopenTalmy, 1985) and a cognitive-linguistic understanding of aspectuality (Reference Cienki and IriskhanovaCienki & Iriskhanova, 2018).

4.2 Conversation Analysis: Context as Social Action

Considering gesture form analysis from the point of view of conversation analysis implies an understanding of context as social (inter)action. It highlights the temporal character of gestures as parts and parcels of sequentially structured interactive processes of conversations’ social organization (Reference Mondada, Müller, Cienki., Fricke, Ladewig, McNeill and TeßendorfMondada, 2013a, Reference Mondada, Müller, Cienki., Fricke, Ladewig, McNeill and Teßendorf2013b). It reveals how gestures participate in the taking of turns (Reference BohleBohle, 2007; Reference SchmittSchmitt, 2005) in cooperative actions (Reference GoodwinGoodwin, 2018) and shows how they contribute in a wide range of embodied communicative activities (Reference StreeckStreeck, 2009, Reference Streeck2017). Connecting gesture form analysis with conversation analytic perspectives may help to answer questions such as: Why does the speaker use the gesture “tracing a round frame” to depict this object at that very moment in the conversation? Why doesn’t he simply present the object on his open hand or not use a gesture at all, given that he provides similar “information” verbally: “a round frame”?

Considering the structure of turn-taking as a social activity, we see that both gestures (G2, G3) are coordinated with the core piece of the conversational turn: the turn-constructional component (Reference Sacks, Schegloff and JeffersonSacks, Schegloff, & Jefferson, 1974). Positioned at another site in the conversational sequence, they could function as turn-entry or turn-exit devices. At the beginning of the speaker’s turn, they would indicate the wish to become the next speaker; at the end of the turn, they would complete the turn, for instance, by filling up a speech-pause. If they were placed in the transition space between two conversational turns, they could indicate the wish to maintain the right for the succeeding turn. Furthermore, the fact that the two gestures are placed in synchrony with the verbal turn-constructional-component indicates that they are part of the most relevant part of his turn. By doing this, the speaker draws interactive attention to particular semantic aspects: the roundness of the frame and the fact that it carries a small crown, marking this information as particularly relevant for his attending coparticipant (Reference Müller and TagMüller & Tag, 2010).

Extending the scope of analysis and taking into consideration the larger conversational unit in which this little sequence of gestures is embedded, it becomes clear that the two gestures are part of a conversational activity of explaining something. Notably, this explanatory sequence is a consequence of a longer process of word searches (on the side of the speaker) and requests for clarification on the side of his interlocutor. In this sense, the extremely precise verbal and gestural description of the kind of picture frame, that figures as “main topic” in a narrative of the speaker, is a consequence of the conversational activities preceding them. The foregrounding activity therefore appears as an interactional consequence.

A conversation analysis of gesture forms reveals why the speaker places his gestures at this moment and place in his utterance. It focuses on gestures as communicative social activities and uncovers the concatenation of cooperative activities of coparticipants. Context for gesture form analysis is here the social organization of language use.

4.3 Discourse Dynamics: Context as Dialogic Process of Creating Mutual Understanding

Gestures often take part in dynamic processes of meaning-making. This concerns very small sequences of gestures, such as the succession of G2 and G3 (round-frame drawing followed by crown-placing), but it may also concern the use of gestures over larger time spans, as our example of Luis’ story shows. Used in a loose succession, each of the five gestures displays a different perspective on the same object and participates in different ways in creating the storyline: G1, G2 and G3 introduce the key object in the story (the royal portrait) by forming its shape and character as a royal portrait; G4 prepares the plot, for example, it locates the portrait in another room: and G5 is part of the plot, for example, it depicts how the portrait fell down all by itself. With these gestures, a salience structure on the level of the narrative is established, highlighting new rather than given information; information is foregrounded that is relevant and central to ensuring mutual understanding of the story told in a conversation.

Conducting gesture analysis from a discourse dynamics point of view starts from an understanding of context as a dialogic process of attempted mutual understanding. Cameron’s metaphor-led discourse analysis (Reference Cameron, Maslen, Todd, Maule, Stratton and StanleyCameron et al., 2009) offers a theoretical and methodological background for a form-based gesture analysis (Reference Kappelhoff and MüllerKappelhoff & Müller, 2011; Reference Müller and KappelhoffMüller & Kappelhoff, 2018). Another example for a discourse-based concept of context in gesture studies is McNeill’s adaptation of “communicative dynamism,” which addresses narrative structures as dynamically unfolding within discourse (Reference McNeillMcNeill, 2005, p. 55).

In short, analyzing gesture forms from the perspective of discourse dynamics addresses the contribution of gestures to various dialogic processes of building mutual understanding.

4.4 Expressive Movement: Context as Multidimensional Experiential Gestalt

An analysis of gestures as expressive movements starts from the assumption that the multimodal orchestration of speech and body movement forms multidimensional experiential gestalts which arise in the process of felt perception of interlocutors (Reference Müller and KappelhoffMüller & Kappelhoff, 2018). Contexts-of-use from this perspective are shared realms of experience, or experiential frames, which ground embodied meaning (Reference Müller, Zlatev, Sonesson and KonderakMüller, 2016, Reference Müller2019b) and, when they recur within a community, may become the stabilized “meaning” of a gestural movement (Ladewig, this volume; Reference MüllerMüller, 2017a).

Analyzing gestures in their contexts-of-use from a perspective of expressive movement assumes an understanding of gestures as temporal forms immersed in dynamically evolving contexts-of-use. As outlined and illustrated above (Section 3.4.3), conceiving of contexts as multidimensional experiential gestalts opens up a way of analyzing gesture forms as facets of embodied interaction where intersubjectivity arises from intercorporeal and interaffective understanding (Reference MüllerMüller, 2019b; Reference Müller and KappelhoffMüller & Kappelhoff, 2018). Such a perspective is in line with ecological approaches to gesture analysis (Reference Cuffari, Jensen, Müller, Cienki, Fricke, Ladewig, McNeill and BressemCuffari & Jensen, 2014; Reference Jensen and GreveJensen & Greve, 2019) or approaches to intercorporeality (Reference Meyer, Streeck and JordanMeyer, Streeck, & Jordan, 2017) and interaffectivity (Reference Fuchs, Meyer, Streeck and JordanFuchs, 2017; Reference Horst, Boll, Schmitt, Müller, Müller, Cienki, Fricke, Ladewig, McNeill and BressemHorst et al., 2014; Reference Koch, Fuchs, Summa and MüllerKoch, Fuchs, Summa, & Müller, 2012).

Conceiving of gestures as expressive movements connects with Kendon’s analysis of gestures as movement phases, phrases, and units, that is, as temporal structures (Reference KendonKendon, 2004, p. 112). To define the “stroke” of the gestural movement, Kendon draws on Laban’s dance theoretical analysis of body movements as expressive movements (cf. Reference Kennedy, Müller, Cienki., Fricke, Ladewig, McNeill and TeßendorfKennedy, 2013). The concept of expressive movement has played important roles in the history of philosophical reflections of gesture: Reference WundtWundt (1973) developed a monadic, solipsistic understanding of the concept; Reference BühlerBühler (1933) saw expressive movements as communicative actions; and Reference Plessner, Dux, Marquard and StrökerPlessner (1925/1982) and later Reference Merleau-PontyMerleau-Ponty (1945/2005) underline the intercorporeal and interaffective nature of expressive movements. Granted by bodily perception, understanding is grounded in feeling movement and in moving together. Notably, this notion of expression was central to modern dance as well as to film theory, both facing an inherently temporally structured medium of expression (Reference Kappelhoff and MüllerKappelhoff & Müller, 2011; Reference MüllerMüller, 2019b; Reference Müller and KappelhoffMüller & Kappelhoff, 2018).

Such theories offer paths toward reconstruction of the “meaning” of gestures from expressive movements as multidimensional gestalts in dynamic contexts-of-use. It remains fascinating how, through Kendon’s original analysis of manual co-speech gestures, the temporality of gestural form has created a common ground for analysis in the field of gesture studies.

4.5 Metaphor Research: Context as Multimodal Expression of Metaphoricity

Including gestures in metaphor analysis involves a semantic perspective on the coarticulated verbal utterance. In verbo-gestural metaphors, gesture and speech work together in expressing metaphoricity (Cienki & Reference MüllerMüller, 2008a, Reference Cienki, Müller and Gibbs2008b; Reference MüllerMüller, 2008, Reference Müller and Hampe2017b). Gestures often enact the source domain of a verbal metaphoric expression. For example, when Barack Obama speaks about those American people who look back and cling to the past, and points backward over his right shoulder, the gesture locates “looking back in time” in the space behind the speaker, that is, “Back in Time” is seen as “Back in Space.” In this context-of-use, the gesture is an expression of time as space, where the past is associated with what is behind a speaker and the future is what lies ahead, as is typical for western European languages. Reference Nùnez and SweetserNùnez and Sweetser (2006) have documented how such conceptualizations of time may vary across cultures: For Aymara speakers of the high Andes, they demonstrated that the spatial location of future and past is reversed, and this is reflected in their gesturing.

A careful analysis of gesture forms and their temporal unfolding along an interaction may reveal that sometimes metaphoric verbal expressions evolve from gestural body movements performed in the absence of speech. For example, in a dance training study, the notion of the dancer as standing in the center of a coordinate system emerges as an embodied conceptualization of balance and anchoring (Reference Müller and KappelhoffMüller & Kappelhoff, 2018, Ch. 11).

Analyzing gestures in the context of multimodal expressions of metaphoricity uncovers how metaphors are experienced while speaking, whether – at a given point in time in a discourse – they are foregrounded, “alive and awake” for a given speaker, whether they receive interactive attention, and whether they are taken up and unfold within the dynamics of discourse (Reference Cameron and GibbsCameron, 2008; Reference Cameron, Maslen, Todd, Maule, Stratton and StanleyCameron et al., 2009; Reference MüllerMüller, 2008; Reference Müller and KappelhoffMüller & Kappelhoff, 2018; Reference Müller, Ladewig, Borkent, Dancygier and HinnellMüller & Ladewig, 2013).

4.6 Pragmatics: Context as Communicative Action

Applying a pragmatic perspective to gesture form analysis assumes an understanding of context as communicative action where language use is conceived of as performance of speech acts (Reference AustinAustin, 1962; Reference SearleSearle, 1969). Although speech-act theory has received a lot of criticism, the fact that we “do things with words,” as Austin put it, has inspired much gesture research. Kendon characterizes gestures as “visible actions” that are used as utterances and addresses the pragmatic dimensions of gestures in manifold ways (Reference Kendon, Duranti and GoodwinKendon, 1992; Reference Kendon2004, Chs 9, 10, 12). For example, his pragmatic analysis of the ring gesture (index and thumb forming a ring shape) focuses on the illocutionary force or more generally on the communicative actions performed with a given gestural movement (Reference KendonKendon, 2004, Ch. 12).

For the ring gesture, this means that only because the gesture is performed in the context of a communicative action, the action of picking up some tiny object can be transformed into a communicative action of making precise arguments (Reference MüllerMüller, 2017a). When such a gestural form is repeatedly used within similar contexts-of-use, these repetitions may stabilize and become a conventionalized gestural form, an emblem, such as is the case with the ring gesture used to express excellence and perfection in many Western cultures (Reference Müller, Müller, Cienki, Fricke, Ladewig, McNeill and BressemMüller, 2014b). Another example is the palm-up-open-hand gesture that may be used to depict the handing over of small objects lying on the palm of the hand, or may be used to hand over speaking turns (Reference Streeck, Hartge, Auer and di LuzioStreeck & Hartge, 1992), and thus have a pragmatic function (Reference Müller, Müller and PosnerMüller, 2004). This means that the analytic decision about whether such an as-if movement is used referentially or pragmatically depends upon its placement in a given context-of-use.

Connecting gesture form analysis with a pragmatic perspective addresses gestures as communicative action. It opens up a path to reconstructing the embodied history of some of them, a kind of “etymology” of conventionalized gestures as in the ring or the palm-up-open-hand gesture. It may document the spontaneous emergence of gestural communicative actions from manual actions (Reference StreeckStreeck, 2009, Reference Streeck2017), reveal processes of decomposition of gestural forms (Reference MüllerMüller, 2017a), and show how gestures contribute to multimodal constructions with a pragmatic meaning (Reference Bressem and MüllerBressem & Müller, 2017; Reference LadewigLadewig, 2020).

4.7 Summary

Section 4 has illustrated how context analysis implies a decision to work within a specific theoretical framework. To illustrate potential (and currently common) understandings of context for gesture analysis, a selection of frameworks has been briefly presented: (1) cognitive linguistics, (2) conversation analysis, (3) discourse dynamics, (4) expressive movement, (5) metaphor research, and (6) pragmatics. Some of the tools for gesture form analysis are intrinsically connected with a specific framework: For example, describing gesture form as a motion event implies a cognitive-linguistic understanding of context, whereas analyzing gesture form as expressive movement implies a notion of context where gestures, other body movements, and speech merge in a multidimensional gestalt. Other “tools” go along with several theoretical frameworks: For example, noting down aspects of kinesic forms can be relevant for all theoretical approaches; together with the description of hands as performing as-if actions, they allow for a form-based heuristics of gestural meaning and a gesture-first approach to the analysis of multimodal utterances. The framework chosen can be an entryway to analyzing different aspects and kinds of gestural meaning. Whether a certain gesture form is to be considered as depicting the concrete or the abstract, or whether it is used pragmatically, depends upon the context-of-use with which it is entangled. Applying different frameworks in form-based analysis thus uncovers different ways in which gestures contribute to multimodal utterances in unfolding interactions.

1.1 Voice and Face

The first line of research focused on the semantic contribution of intonationFootnote ¹ that can differentiate many situations expressed by the same sentence. For example, 18 different situations can be evoked by modifying the intonation of “Il a une voiture” (He has a car) (Reference CalbrisCalbris, 1973, pp. 104–117). While recording intonations for an experimental auditory recognition test with 90 French subjects, the performer was not able to produce the appropriate intonation without reproducing the physical attitude associated with it. This finding provided an opportunity to observe that the differentiating intonational attitudes constitute “audio-visual units,” both in transmission and reception, that are structured by the same vocal-kinesic dynamics (Reference Calbris and MontredonCalbris & Montredon, 1980). One of the consequences of this test was to demonstrate that phonogestural parallelism actually manifests itself on many levels: (1) in time, (2) in form, (3) in meaning, (4) in function, and (5) in coding (Reference CalbrisCalbris, 1989).

1.2 Face and Gesture

The second type of research investigated the semantic contribution of speechless facial-gestural expressions (Reference Calbris and MontredonCalbris & Montredon, 1986). This involved testing the meaning and the signifying structure of conventional French facial-gestural expressions (emblems), for example, the gesture of twiddling one’s thumbs meaning “idleness.” Thirty-four emblems involving different body segments at different levels of the body were captured on film simultaneously by two cameras; one filmed the whole expression, the gesture and the face, while the second simultaneously filmed just the gesture. For the purposes of the experiment, the 34 expressions were divided into two equivalent lists of items and presented to two equivalent groups of French subjects. Each subject had to match an expression (signifier), partially or completely presented, to its meaning (signified) on the given list of items. For example, the verbal cliché “il fout rien de la journée” (he does nothing all day) associated with the meaning of the thumb-twiddle emblem appeared on the list and not the verbal cliché “il se tourne les pouces” (he twiddles his thumbs) that describes the emblem. The results of the test (85 percent success rate) show that the gesture was sufficient in 83 percent of the cases to identify the signified. The contribution of the face was most often irrelevant, positive if it disambiguated the gesture, and negative if it was superfluous or if its affective connotation seemed to contradict the connotation signified by the gesture (Reference CalbrisCalbris, 1980; Reference Calbris1990, Ch. 1).

To verify the cultural character of these emblems, the semantic test was repeated with Hungarian subjects, with a different native language but the same Western culture (civilization), and with Japanese subjects, with a different native language and culture. The rate of comprehension decreased from 85 percent for the French group to 46.5 percent and 29 percent for the Hungarian group and Japanese group, respectively. It appears that cultural similarities and differences may be more important than linguistic differences in accounting for these results.

The results of this experimental study demonstrated not only the cultural (or conventional) character of gesture but also its analogical (or motivated) character: by systematically searching for analogical links in the foreign subjects’ data, analysis of the errors revealed the motivated character of gesture and, taking into account the numerous links imagined by these subjects while interpreting the same gesture, the possible plural motivation of one single gesture (Reference CalbrisCalbris, 1981; Reference Calbris1990, Ch. 2).

1.3 Gesture on Its Own as a Nonverbal Sign

The third study concerned the semantic contribution of the gesture as a nonverbal sign. Whereas the verbal sign is generally arbitrary (extrinsic), analytical (discontinuous), and explicit (invariant), the nonverbal sign by contrast, whether vocal or gestural, is generally motivated (intrinsic and part of the referent which it signifies), synthetic (continuous, so that any modification implies a parallel modification of the referent), and probabilistic (e.g. Was that a wink or was there dust in your eye?); as such, the nonverbal sign is ideal for expressing what is implicit (Reference Scherer, St. Clair and GillesScherer, 1980). If the verbal sign is conventional and arbitrary, the nonverbal sign is conventional and motivated (Reference FónagyFónagy, 1961), that is, its physical aspect and its meaning are analogous, even though its meaning differs from one culture to another. For example, the finger Ring, as an emblem or a co-speech gesture, is always an analogical sign (Figure 9.1) because it is always based on a link of contiguity or physical resemblance established through analogy between a relevant physical feature of the gesture and physical experience of the world. This analogical link is the source of the gestural semanticity that is tapped to create the specific contextual meaning of a gesture from a range of possibilities offered by its iconic and cultural nature. Hence the finger Ring is an intercultural analogical sign (compare Malta, France, and Japan). Moreover, it is an intracultural analogical sign: in France, as an emblem, it signifies “perfect” and sometimes “zero,” and the facial expression determines whether it means positive or negative appreciation; as a co-speech gesture accompanying a verbal utterance such as “Cela concerne 0,25% de familles” (That concerns 0.25% of families),Footnote ³ it expresses extreme precision, not by depicting a circular form but fingernail pincers. Both cultural and contextual, the gestural sign is always an analogical sign.

Figure 9.1 The gestural sign is always an analogical signFootnote ²

A parallel attenuation on the physical and the semantic levels confirms the analogical nature of the gestural sign. Figure 9.2 shows four typical gestures that express decreasing degrees of threat from left to right: the surface of the body part (hand/index finger) and the dynamics of the movement (segment shaken/simply lifted) decrease correspondingly with the intensity of the notion expressed. We find this parallel attenuation in the physicosemantic expression of quantity as well as in that of opposition (see Reference CalbrisCalbris, 2011, pp. 167–170). Isomorphism is the term that Reference CalbrisCalbris (1990, Reference Calbris2011) uses to describe such parallelism in gestural signs that share a core meaning and present graded expressions of it.

Figure 9.2 A continuous analogical sign

1.4 Distinction between Gestural Sign and Gesture

The semiotic unit constituted by an analogical sign can be common to several gestural units cumulated for more expressivity within the same kinesthetic ensemble, for example, as shown in the simultaneously raised eyebrows, head, and hand expressing “increasing exclamation” in Figure 9.3. The gestural unit defined by the movement of a body segment is more precisely described by means of the various gestural components detailed in Figure 9.4.

Figure 9.3 Identifying gestural units

Figure 9.4 Gestural components

2.1 Physical Classification of Gestures

The most relevant physical criteria determined the hierarchical presentation of a physicosemantic dictionary of all observed gestures referring to the concrete and to the abstract (Reference CalbrisCalbris, 1984, pp. 330–856; see Reference Calbris1990, pp. 211–227 for excerpts) as shown in Table 9.1:

• – According to the active body segment, either mobile (arm(s), hand(s), finger(s)) or not very mobile (head gestures), or the passive object of a gesture (body-focused gestures).

• – According to the movement. A distinction must be made between gestures that draw straight lines or flat surfaces (straight-line gestures) and those that draw curved lines or surfaces (curved gestures) because their respective, relevant secondary components differ.

For straight-line gestures, the most relevant physical features are the directional axis of the movement (up–down, front–back, or right–left) in conjunction with the active body part and the plane in which this is oriented (frontal, sagittal, or horizontal). Furthermore, in the case of a flat hand, the palm, fingertips, or the edge of the hand can be relevant (see “Relevant feature” in Figure 9.7).

For curved gestures, the direction of the circular movement (clockwise/anticlockwise) is of utmost relevance. The active body segment is not really relevant. Curved gestures are therefore classified according to the curved line or surface drawn (e.g. an arc vs. a circle, each possibly repeated with or without displacement) and not according to the active body segment.

2.2 Relations between Gestures and Notions

Polysemy on one side, variation on the other; by comparing gesture variants, the common physical element bearing the analogical link with a common signified is revealed.

2.2.1 Variation

Several gestures, either alternatively or simultaneously (Figure 9.5, right column), can represent one notion. Referring back to Table 9.2 and Figure 9.3, an upward movement of the hand and/or head are gesture variants expressing “increasing exclamation.” Here is another example: One can refer to the recent past by moving the thumb or, alternatively, the head over the shoulder; moving both simultaneously creates a cumulative variant that reinforces the expression (Figure 9.6.1). And to signify “distant past,” the hand and/or head raised high and moved backward refer to a space-time far behind you and no longer just behind you (Figure 9.6.2).

Figure 9.5 Relations between gestures (g/G) and notions

(n/N)

Figure 9.6.1 Recent past

Figure 9.6.2 Distant past

Whereas stylistic variants either amplify or attenuate the degree of expressivity when conveying the same meaning, semantic variants offer the possibility of introducing different shades of meaning when conveying the same basic notion by changing a secondary component.

Variants and motivation.

The four semantic variants of “cutting” are related to the plane in which the cut is made: in the sagittal plane, a “cut in two”; in the frontal plane, a “cut that stops,” and in the horizontal plane, a “total cut” are signified, respectively. In the latter plane, turning the hand so that the palm faces upward to make the transverse cut specifies a “total cut at the base” or “undermining-felling.” Note that the transverse movement is specific to both “total” cuts. But it is the rapid movement of the abruptly stopped cutting edge of the hand expressing the proprioceptive schema of cutting that is common to all the four variants. Each of these variants does not express just one analogical link, but two (Reference CalbrisCalbris, 2003a, pp. 19–46).

2.2.2 Polysemy

One gesture can alternatively represent several notions (Figure 9.5, left column). The best example of polysemous gesture is the transverse movement of the Level Hand(s) in the horizontal plane (Figure 9.7) observed in 131 examples from Source 1; it can represent about 20 different notions depending on the context. This gesture exploits all the relevant elements of the flat hand (fingertips, palm, and edge).

This gesture has not only numerous analogical links, which is why it is said to have plural motivation, but each link is also applicable in numerous domains, giving rise to semantic derivation. For example, directness represented by drawing a straight line from A to B expresses immediacy on the temporal level and an immediate consequence on the logical level, that is, determinism, certainty, or obligation, whereas on the level of moral judgment, directness expresses frankness.

Figure 9.7 The plural motivation of a polysemous gesture

Originating from our motor or perceptual experience, gestures that employ the Fist configuration contain several possible analogical links: A clenched fist (1) is strong, (2) can hold something firmly, and (3) can enclose something. Each of these analogical links may be subject to semantic derivation: strength can be physical (example [1]), psychological (example [2]), or moral (example [3]), and an enclosed object that a fist represents may be real, that is, graspable by the hand, or abstract (example [4]).

1. 1 A pianist is talking about the role of the left hand in Beethoven’s music:

   [left fist] La main gauche, c’est une, c’est une, comme ça, une, une présence d’éner [opens] énergétique.

   [left fist] The left hand, it’s a, it’s a, like that, a, a presence of ener [opens] energetic.

1. 2 [right fist] Je me suis obligée tous les jours

   [right fist] I forced myself every day

1. 3 [both fists] Parce que je pense que on ne peut pas, passée la cinquantaine, ne pas s’occuper des jeunes citoyens.

   [both fists] Because I think that one can’t, once past fifty, not do something for young citizens.

1. 4 A psychologist interviewed at home is talking about educational assessment:

   [left fist lifts up] tellement c’était un [left hand falls on to the sofa] un secret bien gardé

   [left fist lifts up] it was such a [left hand falls on to the sofa] a closely guarded secret

To summarize, the physical polyvalence of the fist determines its polysemy as a sign. The context determines the choice of both the analogical link and the semantic derivation.

Polysign gesture.

In contrast, the polysign gesture simultaneously conveys multiple signs (Figure 9.5, left column) because it contains different analogical links that depend on different relevant physical features associated with the context. Let us take the example of the raised fist that is generally used as a sign of threat. In example (4), it is used as a polysign that provides two simultaneously conveyed pieces of information that are confirmed by the spoken utterance. The analysis of example (4) in Table 9.3 shows that the raised fist contains two physical relevant features: its closed configuration and its upward movement (signifiers) are possible supports of analogical links in accordance with the context. Note that the synthetic gestural information (“so enclosed”) that occurs synchronously with the verbal segment “tellement c’était un …” precedes the confirmatory verbal information “un secret bien gardé.”

Table 9.3 The polysign gesture

Speech	[left fist lifts up] it was such a, a closely guarded secret
Gestural signifiers	Configuration: Fist	Movement: Upward	>	Raised fist
Analogical links	Enclosure	More is up
Gestural signifieds	Enclosed object	Increasing exclamation	>	Very well enclosed
Gestural signs	Gestural sign 1	Gestural sign 2	>	A polysign gesture

Furthermore, more than one sign may be conveyed by just one component, for example, by the type of movement (straight or curved) and by its direction. In example (5), loops, signifying “the course of time,” in combination with their direction toward the speaker, signifying “self-centering,” make the idea of “introspection during action” explicit.

1. 5 Donc on a cette cette double mission de de le faire [the index fingers, facing each other, draw two parallel series of loops going toward the speaker whilst the torso moves backward] et de nous regarder le faisant [smile].

   Therefore we have this this twofold mission to to do it [the index fingers, facing each other, draw two parallel series of loops going toward the speaker whilst the torso moves backward] and to watch ourselves doing it [smile].

If we reconsider the gesture variants that signify “cut,” we can see that each variant, nuanced by the plane of cutting, functions as a polysign: cut + division, cut + obstacle, and cut + totality.

Note that the signifying combination of a polysemous configuration (e.g. the Fist) and a polysemous movement (e.g. forward) is apt to produce a large number of varied polysigns. Forward is polysemous: one moves forward to go toward someone/something/somewhere or to counter an opposing force (two analogical links). Moreover, the former link is subject to semantic derivation: spatial or temporal progression. So, according to the context, a fist moving forward can signify: psychological strength + toward something, hence, the effort required to reach for a goal; psychological strength + against something, hence, the will to attack; or strength + temporal progression, hence, strength and even modernism.

Complex gesture.

Finally, the complex gesture is a particular case of the polysign gesture: it performs the respective figurations of two known gestures by creating a third one that synthesizes them in an original way. For example, in order to simultaneously represent mixing, usually signified by the two hands turning around each other (Figure 9.8.1) + approximation, usually signified by a rotational oscillation of the concave palm facing downward (Figure 9.8.2), a screenwriter (example [6]) and a philosopher (example [7]) perform the same synthesis (i.e. an alternating oscillation of the two concave palms, one behind the other, as if they were interlocked*) to express a “kind of confusion” and an “approximate mixture,” respectively.

1. 6 The screenwriter says:

   *l’espèce de désarroi dans lequel se trouvent les hommes et les femmes maintenant.

   *the sort of disarray (“kind of confusion”) in which men and women find themselves now.

1. 7 Here it is a question of belief:

   Je n’aime pas beaucoup *le judéo-chrétien.

   I don’t like very much *the [term] Judeo-Christian (“this approximate mixture”).

Figure 9.8 A complex polysign gesture

Here are some examples of a complex head gesture: an abrupt lateral lowering of the head*, combining a movement downward and sideways, is often a way of emphasizing something negative, for example, a way of reacting to the exaggeration of others: “C’est maintenant que tu arrives? (You’re coming now?) *Pas trop tôt! (*Not too early!)” or to strongly disagree: “En 74? (in 74?) *Ca m’étonnerait! (*That would surprise me!).” This synthetic movement (downward and sideways) cumulates the motivations corresponding to each of the directions: pressing downward (insistence) and turning away from (negative element).Footnote ⁵

2.2.3 Polysemy and Variation

In order to elucidate the polysemy of a gesture, it is sufficient to research the gesture variants of each signified in order to identify the analogical link inherent to it. For example, in France, the lateral shaking of the head is polysemous: as an emblem, it signifies “negation,” and, as a co-speech gesture, it signifies “totality” or “approximation” (Figure 9.1). However, each of these notions is expressed by a gesture variant with a single movement: the lateral shake repeats the lateral turning away from one side to the other (negation), repeats the transverse movement (totality), or repeatedly sweeps the gap between the two designated boundaries on the right and left of the head (approximation). The polysemy of the lateral shaking of the head is elucidated by the single-movement gesture variant of each notion.

Figure 9.9.1 Polysemy

Figure 9.9.2 Variation

Research and confirmation of the analogical link.

In Source 1, the co-speech gestures that refer to the notion of totality are characterized by:

• – a transverse movement of one or two Level Hand(s);

• – a transverse movement of the head;

• – the lateral shaking of the head.

What is the analogical relation between the transverse movement of the head, hand, or fingers and the notion of totality? Logically, it is a natural reference to the horizon signifying “everything, everywhere” (Figure 9.2), concretely represented by a glance that sweeps across the horizon (in a single or repeated movement of the head) or by the palm that glides across it with a transverse movement (symmetrical or not). Transverse movement, the element common to the gesture variants, is the relevant element that can be linked to the common notion. By scanning the entire perceptible field from left to right, it represents the perceptual schema of totality and becomes the gestural signifier of the notion of totality. This is what we see in the following sequence:

1. 8 Je serai un opposant, mais résolu [thumb and index finger form the Ring configuration (representing “precision,” “rigor”)], j’allais presque dire systématique contre la proportionnelle, et ce [transverse movement, (representing “totality”)] à toutes les élections

   I will be an opponent, but [a] resolute [thumb and index finger form the Ring configuration (representing “precision,” “rigor”)], I was almost going to say [a] systematic [one] against the proportional system, and this [transverse movement, (representing “totality”)] at all the elections

The transverse movement is performed by the Ring configuration because the rigorous attitude of the speaker concerns all elections. The gestural representation of the idea is thus maintained for as long as the idea lasts (cf. the concept of catchment in Reference McNeill, Quek, McCullough, Duncan, Furuyama, Bryll and AnsariMcNeill et al., 2001, p. 11).

Generalizing from this example, one identifies the analogical links in gestural signs by comparing gestures on the paradigmatic (vertical) axis of substitution and on the syntagmatic (horizontal) axis of combination (Figure 9.10).

Figure 9.10 The analogical link

To conclude this section on Calbris’ approach to analyzing the gestural sign, one can say that the elements of the “symbolic Meccano” game (or “symbolic construction kit”) are the analogical links (Reference CalbrisCalbris, 1987, Reference Calbris2008a). Physicosemantic classification reveals that the polysemy of a gesture is not explained by a simple semantic derivation, but that the gesture that represents several notions is apt to contain several analogical links, alternatively or simultaneously “activated” by the context. There is no correlation between the number of analogical links and the number of kinesic entities. Indeed, one observes that one single link is expressed by the same movement of the hand and the head in a facial-gestural ensemble; conversely, one manual configuration may express several links. The relevance is not in the physical element but in the analogical link it contains.Footnote ⁶

3.1 Utterance Functions

As shown in Table 9.4, the three communication channels are associated differently to serve different functions: expressive, appellative,Footnote ⁷ referential (Reference BühlerBühler, 1934), and phatic (Reference Jakobson and SebeokJakobson, 1960). In terms of communication, emblems and co-speech gestures can serve all of these functions. When accompanying speech, gesture also serves utterance functions connected with speech rhythm (a demarcative function) and speech formulation (a predictive function). Gesture thus serves several functions and can simultaneously cumulate them (Reference CalbrisCalbris, 1997).

3.2 Alternation and Synchronicity

Gesture and speech may alternate within an utterance in a form of mixed syntax (cf. Reference Slama-CazacuSlama-Cazacu 1977, pp. 118–122). Regardless of whether it occurs before or after speech, a gesture may function as if it were a word or other unit of linguistic expression, hence, as a component of syntax in the construction of the utterance. In this case, gesture substitutes for speech. The speaker can thus exploit it as an implicit, expressive, synthetic sign, for example, to show diplomacy, increase expressiveness, or make the interlocutor wait for what is to be formulated verbally.

When gesture and speech occur simultaneously, the gestural expression and the co-occurring verbal expression may be semantically related in two different ways: (1) as a coverbal sign, gestural expression augments what is being expressed in words, that is, the respective information units are synchronized, or (2) as a preverbal sign, gestural expression anticipates the verbal expression of an idea, that is, there is a synchronization of information units but an asynchronous distribution of the information they convey.

Co-verbal sign.

A complementary piece of information supplied by a co-speech gesture can:

• – indicate the speaker’s attitude or comment on what s/he is saying in relation to the object of the utterance (Reference CalbrisCalbris, 2011, pp. 250–252);

• – explain something to the interlocutor with a pedagogical aim in mind, for example, giving the visual representation of a spiral quickly helps to make it clear what a spiral staircase is (Reference CalbrisCalbris, 2011, pp. 253–254);

• – disambiguate a polysemous word in the verbal message, for example, the French word “régulier” can mean regular, cyclical, uniform, or in accordance with regulations. Accompanying the phrase “c’est pas régulier” (it’s not …), the circular and repetitive movement of the hand drawing vertical loops specifies that “cyclical” is the intended meaning (Reference CalbrisCalbris, 2011, pp. 254–255).

It also happens that the simultaneous confrontation of gestural and verbal information that reciprocally specify each other results in conventional gestures and locutions acquiring a new, original, contextual meaning. Such individual creativity with socially established metaphorical expressions is demonstrated by example (9), produced by a politician who was in the Opposition at the time of the interview. He is complaining that promises have been made that something will be done but, according to the speaker’s media searches, so far nothing has happened:

1. 9 [the left index finger inclines the ear pinna forward as if to hear better] Et depuis un an, [then the left hand forms the Palm Forward configuration] j’attends les actes: [both hands transformed into the Ring configuration, side by side, symmetrically draw a transverse line] je n’ai rien trouvé.

   [the left index finger inclines the ear pinna forward as if to hear better] And for a year, [then the left hand forms the Palm Forward configuration] I’ve been waiting for actions: [both hands transformed into the Ring configuration, side by side, symmetrically draw a transverse line] I’ve found nothing.

Figure 9.11 details the process of constructing the meaning of the message, in which each of the three gestures that cut the sentence into rhythmic-semantic groups conveys a complementary piece of information. Derived from action, the gesture of placing the index finger behind the ear and pushing the pinna forward, when seen out of context, means something like “to hear better.” In co-occurrence with the phrase “And for a year” its meaning changes to suggest the idea of a watchdog with his ears pricked up and thus to refer to the figurative French locution “tendre l’oreille” (to prick up one’s ears).

Figure 9.11 Interactive construction of meaning by the two types of sign

The Palm Forward gesture that follows, which may be derived from a self-protective reflex, is essentially a defensive stop. In this context, it signifies that the person has been waiting on standby, immobile, for a year.

The last gesture is a polysign: the Ring gesture used to grasp a very small object refers to something minuscule and precise. As discussed above, transverse movement sweeping the horizon can signify “everywhere.” Out of context, the gestural meanings “minuscule” and “everywhere” combined with the verbal context “I’ve found nothing” become, respectively, “not the least little thing” and “nowhere.”

The interaction between the gestural meaning in context and the meaning of the verbal message produce the total verbal-gestural utterance: “And for a year” I’ve been pricking up my ears, immobile “I’ve been waiting for actions: I’ve found absolutely and strictly nothing.” The gestural contribution confirms the implied metaphor of the alert and motionless watchdog. The referential function is performed by the two channels in interaction. The semantic interaction of gesture and speech reported by Kendon (cf. Reference KendonKendon, 2004, pp. 158–175) in situations referring to the concrete is developed here in the abstract domain.

Preverbal sign.

Diverse examples across the corpus provide Calbris with instances of gesture anticipating speech, whereby there is an asynchronous distribution of information across synchronous kinesic units of gesture and prosodic units of speech. Reference CalbrisCalbris (1998) hypothesizes a relation between the gestural anticipation of speech and the type of mental images involved (Table 9.5). The reasoning is based on a possible correlation between the different types of image and the difficulty of encoding them verbally:

• – An original mental image, being initially quite difficult to encode verbally (V), would be formulated gesturally (G) so that its presence in the mind can be maintained concretely for the duration of the verbal formulation. For a fairly uncommon image (i.e. a metaphoric extension or a particular mental image whose verbalization is not automatically available), anticipatory gesturalization would serve as an aid to verbalization (left column).

• – Inversely, in the case of a known, commonly shared mental image transformed into a figurative locution, the initially unsynchronized association between gesturalization and verbalization would be readily transformed into a synchronous ensemble due to its frequency of use (central column).

• – If a speaker-gesturer is not conscious of the analogical link contained in a figurative locution or its gestural representation that s/he is producing, then habit and automaticity would engender synchronous transmission of the analogous bimodal information (right column).

Table 9.5 Relation between the gestural anticipation of speech and the type of mental imagery

G –> V	V|G	V G
Anticipation of gesture for original images	Simultaneity of gesture for common images	Non-consciousness of simultaneous gesture

Gestural information conveyed during speech production often precedes analogous verbal information (example [4] showed a preverbal polysign); it gives an idea of the content to be put into words. As a preverbal sign, it can:

• – summarize the whole content of the utterance that follows and thus function as a gesture-title (Reference CalbrisCalbris, 2011, pp. 257–258);

• – help the speaker to find an appropriate formulation for what s/he wants to say and thus function as aid to verbalization (Reference CalbrisCalbris, 2011, pp. 258–263);

• – create an interplay between tension and relaxation in communicating information for the interlocutor, who is led to guess what will be said in an intellectual game of switching between the two information channels.

Example (10) produced by the former French prime minister Lionel Jospin shows gestural information preceding each piece of verbal information in an astonishing way. Is this to emphasize his viewpoint right from the very start?

1. 10

   [Fingers pointing to the chest] L’euro | [palm opened outward] pour moi | [and moved forward]: c’est un instrument de puissance.
   [Fingers pointing to the chest] The euro | [palm opened outward] for me | [and moved forward]: it’s an instrument of power.
   Gestural information:	For me		it’s		it’s that:
   Verbal information:		The euro,		for me,		it’s an instrument of power.

Let us now consider example (11) showing how gesture can create an interplay between tension and relaxation in the bimodal information that is progressively communicated. It was produced by a technician being interviewed about construction work at the Musée d’Orsay, which he metaphorically refers to as a boat on the banks of the River Seine:

1. 11 Ces micropieux sont ancrés dans le calcaire que l’on retrouve sous le musée pour empêcher le bateau de se soulever avec la montée des crues.

   These micropiles are anchored in the limestone that one finds under the museum in order to prevent the boat from rising with the rise of flood waters.

Stripped of its nouns (micropiles; limestone; museum; boat; flood waters), the message is reduced to a skeleton that gesture articulates: “That anchored in that under that raised by that.” Table 9.6 shows gestures representing: (1) the micropiles, by the thumb and the index finger directed outward like hooks, (2) where they provide anchorage, by lowering the palms facing downward, and (3) rising movement, by raising the palms facing upward. A change of gesture (1, 2, 3) initiates a new rhythmic-semantic group, and the modulated repetition of each gesture segments each group into subgroups of meaning (A, B, C):

Table 9.6 Interplay of tension-relaxation between gestural and verbal information

A	B		C
Ces micropieux These micropiles (1) hooks	sont ancrés are anchored hooked in		dans le calcaire, in the limestone hooked in
que l’on retrouve that one finds (2) under	sous under under		le musée, the museum –
pour empêcher in order to prevent (3) rising	le bateau the boat	de se soulever avec from rising with rising	la montée des crues the rise of flood waters –

A, B, C: rhythmic-semantic subgroups

1, 2, 3: each new type of gesture, accompanied by its contextual meaning

Underlined: gestural information anticipating verbal information

The gestural information conveyed in A anticipates and complements the verbal information conveyed in B and C. For example, Figure 9.12 shows that the visuokinesic information conveyed in (3) is later confirmed by the verbal information. This bimodal structuring of the information flow across the whole sequence creates tension and relaxation as it progresses.

Figure 9.12 Tension-relaxation in the bimodal information progressively communicated

Finally, let us consider example (12) in which gesture is an anticipatory and complementary synthesis of the utterance (Reference CalbrisCalbris, 2011, pp. 303–312). Sitting at his desk, a management specialist is explaining how to design a project in order to achieve an objective:

1. 12 “On part d’un monde | d’hier | dans lequel on entassait les connaissances.”

   “One departs from a world | of yesterday | in which one accumulated knowledge.”

He sums up his previous worldview in a single, rapid, transverse movement to the right along the edge of the desk while saying “of yesterday” (Figure 9.13.2). For ergonomic and semantic reasons, he chooses the transverse axis. This allows him to represent: (1) anteriority on the left and two developments that are intimately connected here: (2) the temporal progression from left to right and (3) the cumulative progression of knowledge. Furthermore, (4) the rapidity of the movement of the back of the hand shifting to the right implies that this world is over, and (5) the gesture is performed along the edge of the table in front of his torso, indicating “here and now”: we are in the present. Suddenly motionless and gazing straight into the camera, he says “in which one accumulated knowledge,” and thus verbalizes the information that has already been gesturalized (Figure 9.13.3). The gesture testifies to the cognitive activity that programs the two-channel communication of the idea. The multireferentiality enabled through the gestural channel is confirmed through the verbal channel (Reference Calbris, Rector, Poggi and TrigoCalbris, 2003b). Figure 9.14 gives a visual synthesis of the cases in which gesture anticipates speech.

Figure 9.13.1 On part d’un monde “One departs from a world”

Figure 9.13.2 d’hier “of yesterday”

Figure 9.13.3 dans lequel on entassait les connaissances “in which one accumulated knowledge”

Figure 9.14 Temporal perception of the gestural-verbal flow

4.1 Gestural Sign and Perceptual Schema

If we consider the ACTION schema “subject-action-object-result,” the result of the action seems to be the intermediary element between the concrete and the abstract expressed by co-speech gesture:

1. 1. A gesture that accompanies the verbal evocation of a concrete action often shows its result, that is, the idea resulting from the action (the written trace and not the typing on the keyboard). The gesture represents an abstraction made out of the concrete – the result of the concrete action. For example, Can you switch the light off? to “reduce” the light explains the concomitant gesture of lowering the palm.

2. 2. On the other hand, abstract notions derived from the perceived results of a cut are accompanied by a gesture that recalls the original action. The gesture reconcretizes the abstract – deduced from the results of the action. For example, a dichotomy occurs …, “a cut in two” explains the abrupt downward movement of the edge of the hand in the sagittal plane. In the same way, if we consider the OPENING schema “resulting openness-clarity and discovery,” the abstract action (clarification or revelation of content) derived from the concrete result is spontaneously reconcretized by an opening gesture, the opening of configurations with one or both hands (Reference CalbrisCalbris, 2003c), as exemplified in Figure 9.16.

The analysis of gesture reveals that the abstract object is conceived in the image of the physical object, perceptually defined by its contour and shape that determine its mode of prehension. One could call it representational deixis. Indeed, while the speaker is using the index finger to indicate* the abstract object uttered by the interlocutor by saying “*that’s it!”, s/he is positioning and giving shape to the abstract object s/he is thinking about. This is conceived as: a distinct entity (encircled, enclosed by the grasping hand); a definite object (framed by the sagittally held palms); a condensed object (constricted in the hand closed in a Pyramid configuration); a precise object (pointed and able to be grasped by the first three fingers joined together); an extremely precise object (to be grasped between the fingernail pincers of the thumb and the index finger). The speaker reifies the abstract object s/he designates (Reference Calbris, Müller and PosnerCalbris, 2004).

Figure 9.16 Opening a hand configuration to explain something

Whatever the type of abstract object the speaker has in mind, s/he opens it at the moment of giving its content verbally (cf. Reference CalbrisCalbris, 2011, pp. 331–342). In Figure 9.16.1, the object in question is spontaneously delimited by the palms because it is mentally defined, as indicated by the concurrent use of the French demonstrative “ces” (these). In Figure 9.16.2, the object in question is considered essential: an essential value which is … hence the choice of the hand closed in the Pyramid configuration to signify a condensation of something. This OPENING schema is also found in Italy: “the grappolo-to-open is associated with the comment … ” (Reference KendonKendon, 2004, p. 234).

4.2 Multiplicity of Perceptual Schemas Gesturally Represented on the Transverse Axis

The variety of gestural signs expressed on the transverse axis alone (Table 9.7) proves how much our mental world is shaped by our perceptual-motor experiences (Reference CalbrisCalbris, 2002a).

The three axes – vertical from bottom to top, sagittal from back to front, and transverse from left to right – point in directions according to our perceptual experiences of the process of physical growth upward, the direction of walking forward, and the direction of writing to the right in the West, respectively. For ergonomic reasons, there is a permutation observed between the axes of growth and progression toward the transverse axis, which cumulates their respective symbolic values that are easier to localize on this axis. In addition, the transverse axis reflects another perceptual schema, that of the symmetry in all living bodies (plant, animal, and human) divided into two halves that are equivalent, opposable, complementary, and essential for balance. It makes it possible to create oppositions by localizing symbolic values on the left and on the right, respectively: the bottom and the top, below and above, the lower and the higher, less and more, anteriority and posteriority, the old and the new, the cause and the consequence. In accordance with the symmetrical development of any living body, the comparison of two evolutionary processes is made by the left hand to the left, and by the right hand to the right. Another schema that impresses itself on the mind is our bipedal locomotion toward a goal, which makes us move forward symmetrically (dichotomous enumeration) along a path from which one deviates only momentarily to take a glimpse to one side (digression), indifferently located on the right or on the left. However, an aside, if conceived as a secondary element, is placed on the left. On this axis divided into two halves, the equivalence of two similar entities disappears when it is a question of otherness. The self takes precedence over others. The notion of self (extendable to the group or country to which one belongs) is naturally (logically) expressed on the right for a right-handed person (Reference CalbrisCalbris, 2003c). Last but not least, this axis is the only one that can represent our maximal field of vision of 180° to signify “totality.”

4.3 Multiplicity of Notions Derived from the Schema of Cutting

The schema of cutting appears to be traced in numerous and various notions: SEPARATION, CUTTING INTO PARTS, DIVISION INTO TWO HALVES, BLOCKING, REFUSAL, NEGATION, END, STOP, DECISION, DETERMINATION, MEASUREMENT, CATEGORIZATION, CATEGORICAL CHARACTER, and INTERRUPTION. The action of making a clean cut involves the characteristics of the tool (1) and of the act (2), as well as of its result (3): first of all the hardness and rigidity of the tool (1), then the force and brevity of the strike (2), the impact of the cutting edge against a material object, and finally, the notching and the separation, the production of pieces, and the irreversible nature of the act (3). The schema of cutting is a visual and proprioceptive percept. The meaning is already in the gesture, in the physical feeling of the gesture performed. The understanding of the gesture is physical. Drawing on perception, the semantic extension is already based on the physical and temporal metonymy supported by the act itself, linking the tool, the action, and its result. The gesture thus serves the metaphorical transfer pertaining to each component: instrument, act, and result (Reference CalbrisCalbris, 2003a).

It is important to note that the permanent gestural reactualization of the various foundational perceptual schemas of abstraction at the core of spoken discourse is the most flagrant proof of the physical anchoring of cognition. The premises of the concrete process of abstraction, observed in young children and called “Mimisme” by Reference JousseJousse (1936), seem to be corroborated by the discovery of mirror neurons in humans (Reference Rizzolatti, Fogassi and GalleseRizzolatti, Fogassi, & Gallese, 2001).

4.4 Summary

The analysis of gestural signs referring to the abstract reveals traces of the perceptual schemas that underlie them: our perceptual experiences are bearers of meaning at the origin of our mental constructions, and gesture is the witness of this mental imagery at work during utterance. If a gesture represents the visible form of an idea (i.e. the sketch of a concept), it is to be expected that the gestural formulation anticipates and probably facilitates the verbal formulation of the concept. The gestural sign’s anticipation of the verbal sign is the logical correlate of the gesture’s semiotic specificity.

The gestural expression of a notion anticipating its verbal expression during the utterance process is for Reference FónagyFónagy (2001, pp. 580–581) just the ultimate extension of the essentially preverbal character of gesture that one recognizes in the ontogenesis of language and which may possibly stem from its phylogenetic source.

5.1 Research Results Related to Each Source

1. Source 1. By comparing the physical and semantic data in the database of co-speech gestures referring to the concrete or to the abstract, it was possible to bring to light: (1) at the physical level, the numerous gesture variants of a notion, be they stylistic or semantic variants, (2) at the notional level, the possibility of several gestural signs within the same gesture, be they signs alternatively revealed by the verbal context (polysemous gesture) or simultaneously cumulated (polysign gesture and complex gesture), (3) the discovery of the gestural sign thanks to the gesture variants of a notion, their common physical element being the signifier linked to the gestural signified, and (4) that as a stylized representation of the concrete and a concrete representation of the abstract, gesture is an intermediary between the concrete and the abstract.

2. Source 2. Facilitated by the software of a prototype videodisc, the meticulous segmentation of the various filmed utterances (first acoustic, then visual by identifying co-occurrent kinesic changes at different levels of the body) and the subsequent identification of verbal and gestural units and the respective information they conveyed enabled their interaction to be studied. An in-depth analysis revealed: (1) complementarity as well as reciprocal disambiguation in the case of concomitant information, (2) the simultaneous multireferentiality of co-speech gesture (polysign gesture), (3) gesture anticipates speech in the spontaneous expression of an original idea to be formulated: In concurrence with speech, the preverbal gesture that visually concretizes an idea maintains its presence in the mind and thus facilitates the verbalization of an idea whose confirmation the interlocutor is waiting for, and (4) gesture is never superfluous: It expresses, animates, explains, synthesizes information, and anticipates speech; it participates and creates participation in the utterance.

3. Source 3. The statistical analysis of certain recurrent gestures performed during the six television interviews of the former prime minister Lionel Jospin made it possible to show how a frequently occurring gesture potentially reveals the unconscious: (1) the split personality of a person who systematically expresses himself with the left hand as the leader of the Left and with the right hand as a private person, (2) his attitude toward his new function is even reflected in his choice of gesture variant for referring to an abstract object; and (3) gesture reveals the mental imagery underlying utterance (Reference CalbrisCalbris, 2003c; for a review, see Reference De ChanayDe Chanay, 2005; for a video presentation in French, see Reference CalbrisCalbris, 2000 [www.canal-u.tv/intervenants/calbris-genevieve-028605403]).

5.2 Application of the Research to Language Teaching

Based on this research, and since 1975, Calbris has produced pedagogical works in collaboration with Jacques Montredon, pedagogue and linguist, to integrate multichannel communication into the teaching of French as a Foreign Language so that students can physically and intellectually assimilate French prosody (Reference Calbris and MontredonCalbris & Montredon, 1975), voco-kinesic expression (Reference Calbris and MontredonCalbris & Montredon, 1980), gestural expression of notions (Reference Calbris and MontredonCalbris & Montredon, 1986), figurative expression of the language (Reference Calbris and MontredonCalbris & Montredon, 1992), and the semantics of verbal connectors in conversational argumentation (Reference CalbrisCalbris, 2002b; Reference Calbris and MontredonCalbris & Montredon, 2011).

5.3 Position in Relation to Other Researchers

Calbris’ (1983) doctoral dissertation “was one of the first studies ever to attempt a systematic analysis of the ways in which [co-speech gesture] can serve as a vehicle for the representation of conceptual meaning” (Kendon in Reference CalbrisCalbris, 2011, p. xvii). Her work is closely related to the pioneering work of Reference Lakoff and JohnsonGeorge Lakoff and Mark Johnson (1980) on metaphors that draw upon our visual and physical experience of our bodies and of the physical world in which we live. She holds that gesture extracts symbolic material from our physical interactions with the world we inhabit. For her, gesture is situated at the interface between mind and matter and reproduces perceptual schemas (cf. embodied schemata and image schema, Reference JohnsonJohnson, 1987) that underlie verbalization. Hence, her research corroborates Johnson’s theory of “bio-functional embodied understanding” (Reference Johnson2015, p. 7) gained through organism–environment interactions.

Adam Kendon acknowledges that Calbris’ analytic approach partly inspired his concept of gesture families (Reference KendonKendon, 2004; cf. Kendon in Reference CalbrisCalbris, 2011, p. xviii), that is, gestures that share the same configuration and orientation but may differ in respect of movement. Nevertheless, Calbris contends that Kendon overlooks the possibility of the simultaneous plural motivation of gesture that she has found in her corpora (Reference CalbrisCalbris, 2011, p. 283).

Calbris was one of the first researchers to demonstrate and hypothesize about the now widespread observation that gesture often anticipates speech in encoding allied information. Her finding that gesture may aid verbalization concurs with David McNeill’s concept of catchment, that is “recognized from recurrences of gesture form features over a stretch of discourse” (Reference McNeill, Quek, McCullough, Duncan, Furuyama, Bryll and AnsariMcNeill et al., 2001, p. 11), by showing that gestural representation can indeed maintain the presence of an idea over a stretch of discourse. Her investigation of this phenomenon takes into account the possible physical, cognitive, and communicative reasons that may underlie it as well as the perspectives of both communication partners. Her semiological theory of gesture thus interconnects cognitive and communication models of language and furthermore accords with Raymond Gibbs’ theory of “the social nature of embodied, contextually embedded cognition” (Reference GibbsGibbs, 2011, p. 81).

1.1 Biographical Note

Elena Aleksandrovna Grishina (1958–2016) graduated from the Department of Russian Studies at Lomonosov Moscow State University, where she also later received her PhD (the Russian “candidates’ degree”). She was the editor of several dictionaries of the Russian language, working as the managing editor for dictionaries of Russian at the Russkij Jazyk (Russian Language) publishing house. After becoming a senior researcher at the Vinogradov Institute of the Russian Language of the Russian Academy of Sciences, she became one of the visionaries and creators of the Russian National Corpus (RNC),Footnote ¹ which was specially created by linguists (together with computer scientists) and for linguists. She played a crucial role in setting up the principles behind the corpus and in the annotation of the corpus in the fundamental stages of its development; this included the creation of separate subcorpora for poetry, spoken discourse, and accentology of Russian. Her enthusiasm, collegial support, and selfless dedication created a unique context for working on this project. It was she who came up with the idea of creating a special Multimedia Russian Corpus (MURCO). This corpus is what allowed her to create a fundamental description of the system of Russian gestures in a series of articles as well as in a comprehensive monograph in Russian of 742 pages: Russian gesticulation from a linguistic point of view: Corpus studies,Footnote ² published posthumously in 2017. She completed the book during the last stages of her final serious illness. The following sections of the present chapter provide an overview of the multimedia corpus and of the findings from Grishina’s detailed analysis of it.

1.2 Ideas and Methodological Starting Points

Grishina’s scholarly work was greatly influenced by two sources: one was the oeuvre of Adam Kendon, as found in his Reference Kendon2004 book and his talks at conferences which she attended; the other was Andrej Zaliznjak’s approach to linguistic material. It was Kendon who convinced her that gesture is an inseparable part of linguistic communication, but one that is underappreciated by linguists; this was something that she also realized in a practical sense. After several years of working on the corpus, she was invited to give a course to the students of theoretical linguistics at Moscow State University in which she pointed out that, watching a video clip without sound and only looking at the gestures, one could reconstitute the emotions, intentions, and possibly even the presumptions of each speaker. (See Reference Savchuk and MakhovaSavchuk & Makhova, 2021, on recent, related research.) She later carried out studies to follow up on this, testing the “deciphering” of the discourse situation of those watching videos without the sound.

The term “decipher” is no accident here. Having been a student and acolyte of Zaliznjak (see Reference Iosad, Koptjevskaja-Tamm, Piperski and SitchinavaIosad, Koptjevskaja-Tamm, Piperski, & Sitchinava, 2018, for an overview of his work), Grishina took part in various projects of his, beginning with the republication of his Grammatical Dictionary of Russian and his work on Russian accentology (Reference ZaliznjakZaliznjak, 2010, Reference Zaliznjak2011). But she was also active in his expeditions to Novgorod to take part in a large project to decipher texts from the tenth to the fourteenth centuries written on birch bark (known as the “birchbark letters”) (see e.g. Reference ZaliznjakZaliznjak, 2004). Zaliznjak’s method of working with this medieval form of writing highlights how researchers solve linguistic problems, bringing to them from the start a confidence that any given puzzle must have a particular solution. It means that the given text, in the form that it has, is meaningful, and that meaning is expressed by its components and their interrelations. This approach demands the full commitment of the researcher to search for systematic rules and correspondences for every sign, and not to take ideas about meaningless accidents such as “scribal errors” as a starting point.

It is this approach which Grishina applied in full in her studies of gesticulation, and she was always fully convinced that every movement accompanying speech was motivated and replete with meaning, semantically or pragmatically: one need only decipher the functions and their combinations incorporated (in all senses of that word) in the gestures. However, to study this, she needed a new tool. This led to the establishment of the multimedia Russian corpus, something which she created practically single-handedly. After preparatory work in the first decade of the 2000s, a pilot version of the multimedia corpus was available, starting in 2010, and even though she was already ill at this time, Grishina worked on this project with complete dedication, providing the lasting legacy we have today.

2.1 Compiling the MURCO Corpus

Building the corpus required dividing up each video with care into self-contained clips which were not too large and of approximately equal size (deleting pauses and also segments without speech), for which the speech in each was carefully transcribed. As a result, each clip was turned into three parts: the video footage, the audio, and the transcribed speech which was available for use in a search engine. The corpus could then be searched for any text (i.e. speech) fragment using the wide range of options available for conducting searches in the RNC. The options then include grammatical and lexical searches, searches by elements of word formation (derivational morphemes) and lexically based grammatical constructions, and others. Conveniently, the searches can be ordered as left-or right-searches (searching for what comes before or after the search term) and in terms of chronological order. The many fragments that can be found in this way can be analyzed for, among other things, their semantics, intonation patterns used with them, or co-occurring gestures.

Consequently, the MURCO corpus contains 4.7 million lexical tokens which have been manually transcribed and morphologically annotated, of which 3.4 million tokens come from Soviet and Russian films and recordings of staged drama and 1.1 million from academic lectures in Russian and documentaries. The films, chosen from those made by the best directors and actors, provide a balanced and representative overview of films in Russian from the era after silent movies, starting in the 1930s.

As an example, a search for Idi[te] sjuda,Footnote ³ “Come here” in the singular and plural imperative forms in Russian, yields 638 clips. Every clip is annotated on several levels (i.e. it is meta-annotated). In addition to the customary attribution of the source, information is provided about the speakers – namely, their sex and age. From the given search, we can note an historical difference in the accompanying gestures. There is a reduction from the gesture of “beckoning with the whole hand” – flapping the hand from the wrist toward oneself – which was practically obligatory in the 1930s (e.g. Rebjata! Idite sjuda! “Guys! Come here!” in the 1935 film Novyj Gulliver “New Gulliver’), to a movement of the head, tilting it to the left and back, as of indicating the intended movement of the subject from his current position toward the speaker (e.g. Slushaj/devockha/idi sjuda. U tebja chego-nibud’ pokushat’ ne najdëtsja? “Hey/miss/come here. You wouldn’t happen to be able to find anything to eat, would you?” in the 1967 film V ogne broda net “There are no shallows in a fire”). Often this head movement is reduced to the point of barely being barely noticeable (e.g. the black marketeer saying Idi sjuda! “Come here!” in the 2008 film Izcheznuvshaja imperija “The disappearing empire”).

Of course, the database of films must be taken on its own terms: it is an empirical question as to what degree the actors’ gestures produced on a film set are representative of what speakers at the time would do gesturally in a conversation that was spontaneous and took place in a location of their choice. For example, it is curious that the gesture of “beckoning with the index finger” is not used in the films with the phrase “Come here,” as might be expected, and rather only appears in an actor’s (I. V. Iljinskij’s) dramatic reading of Anton Chekhov’s short story Khameleon “Chameleon” in 1952. The gestures produced are thus a feature of the genre in which they occur, but nevertheless, they are presumably to be recognizable to viewers who are part of the target-audience-culture (if not also others beyond it), at least at the time the film was made. Following the same principle, the gestures of the characters in the films are taken as “speakers’ gestures” in this research, even though the actors may gesture differently as private individuals, not playing a role and speaking spontaneously.

2.2 Annotation and Coding of Gestures in the Corpus

Compiling the corpus itself was just the first part of Grishina’s work. The second, more significant part, involved the detailed gesture annotation and coding of a rather large portion of the corpus, consisting of six feature-length films, nearly ten hours in total. These are films from different periods that are classics of Russian cinema and reflect a sixty-year period, from 1936 to 1996. All of the gestures in these films have been annotated in the clips excerpted from the movies and have been coded for parameters including the active (moving) body part, any passive body part in the given gesture, the direction of movement, mirrored gestures (repeating an interlocutor’s gesture), and many more categories (described in English here https://ruscorpora.ru/new/en/instruction-murco.html). The categories used for the different parameters are quite detailed: for example, some 50 possible parts are distinguished for the category “active body part.” That list includes not only the fingers, shoulders, head, mouth, eyes, and eyebrows, but also the forehead, lower lip, ear, soles of the feet, and others. For the latter set, various motions are distinguished, such as rubbing one’s forehead, protruding one’s lips (e.g. in excitement or when offended), moving one’s ear up to someone or something (e.g. to listen closely), stamping one’s feet (e.g. alternately, in agitation), tiptoeing, and so forth. Each chapter of the book has a link to an internet database with corresponding illustrative contexts of use of the categories discussed.

The coding of the gestures in the six films was performed by Elena Grishina and refined through repeated checks and revisions. The value of this lies in the consistency with which decisions were made and the level of detail constantly attended to in the forms and movements. Difficult cases were discussed by Grishina and colleagues at conferences and in small seminars to arrive at decisions, but in the end, the corpus is the product of Grishina’s consummate expertise in the study of Russian gesticulation. The principle involved is akin to that described by Reference Stelma and CameronStelma and Cameron (2007); in their case, it concerned the coding of intonation units in transcribed talk. The method described and recommended there involved the annotation and reannotation by a given individual, with refinements over time based on consultation with other experienced researchers, the goal being to produce an expert transcript in which many factors have been taken into consideration.

Ultimately, this multimodal corpus, prepared with such care and expertise (and the expert’s love for her work), constitutes a set of highly valuable material for research in linguistics and gesture analysis. Most prominently, it served as the basis for many of Grishina’s own studies, which are brought together in her Reference Grishina2017 monograph. The book contains four main sections: pointing gestures, representational gestures, auxiliary gestures (a category combining several types of gestures that accompany other functions, such as blinking, licking one’s lips, certain facial expressions, and others, as explained below), and, in conclusion, certain overarching topics (see Reference FedorovaFedorova, 2018, for an overview of the book in Russian). Highlights from each of these parts of the book are provided in Sections 3–6 of this chapter.

3.1 Etimons of Pointing Gestures

Another distinction used in Grishina’s system of characterizing deictic gestures is what she calls “etimons.” These are bases of gestural forms, sometimes involving practical actions which can be seen as serving as the “etymology” of certain gesture form/function pairings. The etimons that Grishina describes for deictic gestures are divided into geometric ones and pragmatic ones. The geometric etimons consist of the following:

• a point, a unit which cannot be further subdivided, as seen in what the extended index finger points to in a classic pointing gesture;

• a line, a distance indicated (one might say traced in the air or on a surface) between two points;

• a line segment as an extent, indicated by two endpoints, one of which is the tip of the pointing index finger, and the other being the location of the speaker (compare Reference Hassemer and McClearyHassemer & McCleary’s [2018] model of pointing);

• a surface, which may be indicated by the palm of the flat hand, having a given orientation in space;

• a volume, which can be shown between flat open hands, or can be understood to be on the palm-side of an open hand, particularly with the palm up, as if holding an object on it by virtue of the force of gravity.

Compare Reference Mittelberg, Evans and ChiltonMittelberg’s (2010) analysis of gestures showing lines, surfaces, and “containers” of volume.

Pragmatic etimons are defined in terms of the pragmatic relation of the speaker and hearer in the act of communication, which, in deictic contexts, is carried beyond the prototypical zone of communication (the space between interlocutors) and related to the referent, indicated via the deixis. Deixis, here, is thus understood more broadly than with the prototypical pointing form, but as generally encompassing gestures with the function of fixating the location of a concrete or abstract referent in space (Reference GrishinaGrishina, 2017, p. 17). Here, the following etimons are described.

• The object located on the palm, which of course involves the metaphorical idea expressed with the palm-up open hand, as described in Reference Kendon, Versante and KitaKendon and Versante (2003), Reference Müller, Müller and PosnerMüller (2004), and Reference Bressem, Müller, Müller, Cienki, Fricke, Ladewig, McNeill and BressemBressem and Müller (2014b). The palm-up open hand can be used to metaphorically give or request something, with that “thing” usually being a metaphorical discursive object, such as a claim, a question, or another form of speech act.

• Suppression – again, metaphorically – of someone’s claim or someone else’s will to do something, controlling the situation in some way with one’s palm-down open hand. (Cf. Reference Bressem, Müller, Müller, Cienki, Fricke, Ladewig, McNeill and BressemBressem & Müller, 2014a, on “away” gestures, of which this is one category; and the discussion of gestural discourse markers in Reference KendonKendon 2004, Ch. 12.)

• Vertical orientation of the palm is used with a variety of motions and so covers various types of “away” gestures (Reference Bressem, Müller, Müller, Cienki, Fricke, Ladewig, McNeill and BressemBressem & Müller, 2014a). In that sense, it is characterized as the “null” etimon, in that this palm orientation can serve many functions. It is interesting that it is included with pointing gestures in this classification; whereas other researchers might focus more on the pragmatic functions of such gestures, the idea here is that the palm is still used as the means of orienting deictically to a referent, whether it is physical or abstract.

• A vector that can be imagined projected from the tensely extended index finger with other fingers in a grip, as in Fig. 10.1a. This serves such functions as fixation of a referent or more generally as a way of directing the interlocutor’s attention along the axis of the vector.

• The “signal” etimon is the engagement of the interlocutor in the communicative context or the drawing of his/her attention to one or another gesture via the increased use of tension of the pointing hand. This tension can occur in the arm, in the hand itself, and/or in the grip of the other fingers with index finger pointing (see in Figure 10.1a, less so in Figure 10.1b, and not at all in Figure 10.1c). As Grishina discusses in Section 2.5.4 of her book, this gestural etimon is often used with explicit, stressed verbal demonstrative expressions, for example, with the open hand, Zachem èto nado bylo davat’ chitat’ {vot èto vot pis’mo} postoronnim ljudjam? “Why in the world did you have to let outsiders read {this very letter}?” (where curly brackets mark the word cooccurring with the gesture, in this case a tense palm-up open hand with the arm stretched straight out) (Reference GrishinaGrishina, 2017, p. 102).

Further analysis of the etimons described above, and the various finger and hand configurations described for deixis, lead to one of the characteristic features of Grishina’s work, which is a set of analyses of co-occurrence patterns. These cover form features occurring with other form features and forms occurring with functions as used by speakers of Russian (compare Reference Bressem, Müller, Müller, Cienki, Fricke, Ladewig, McNeill and BressemBressem and Müller’s [2014b] analysis of recurrent form-function pairings for speakers of German). All are counted across the entire video corpus used, and then tested using chi-square analyses to assess which combinations occur significantly more often, statistically speaking. For example, one analysis (total N=1287 gestures) of the form combinations compares use of the orientation of the palm with the extended index finger hand shape (with the other fingers closed in a fist) versus with the open hand. It shows that the palm-up orientation occurs significantly more with the open hand, while the palm-vertical (i.e. palm facing laterally) and palm down occur significantly more with the extended (pointing) index finger. Further analyses consider referential, semantic, and pragmatic factors influencing the configuration of the pointing hand.

3.2 Pointing with the Thumb

To give a further taste of Grishina’s approach to analysis of deictic gestures, we can consider the special attention she gives to pointing with the thumb. According to Grishina, pointing with the thumb, by contrast with pointing with the index finger or the head, has some specific characteristics in the Russian system of gesticulating. Some of these are familiar from descriptions of thumb pointing by speakers of English or Italian, known from Reference KendonKendon (2004), but Grishina’s analysis of them adds some specific insights.

The first specific feature to note with pointing with the thumb in the given corpus concerns its spatial characteristic that it is usually directed to the back, behind the speaker. This means that it is indicating a zone that the speaker cannot see. Therefore, the pointing is, as a rule, not directed at any object in the speaker’s field of vision, and often attributes to this object the quality that its specific localization is not important or is not known.

Deriving from this, the second feature is functional in nature: pointing with the thumb often realizes an anaphoric function. Its meaning is revealed by the previous context, which may have involved pointing with the index finger or the open hand. In addition, whereas the speaker’s gaze when pointing with the index finger may have been directed at a specific concrete object, pointing with the thumb was found to usually be accompanied by gaze at the interlocutor. The anaphoric function appears in the fact that thumb pointing is often used with lexical and morphological units which mark contrast (e.g. use of comparative forms). Pointing with the thumb thus falls under the category of Russian gestures which convey the idea of distancing, and anaphoric contexts can be seen as examples of a form of distancing between a speaker and an object (or referent) which is not located in his or her zone of visibility. This can concern visibility either in the physical sense or metaphorically in the case of abstract referents, for which “visibility” can be understood in terms of cognitive awareness of the referent. Thus, what elsewhere has been named “abstract pointing” (Reference McNeill and LevyMcNeill & Levy, 1993) is labelled “cognitive pointing” by Grishina, and has to do with pointing to spaces to indicate ideas, sometimes using contrasting spaces to differentiate different ideas (see also Reference McNeill, Cassell and LevyMcNeill, Cassell, & Levy, 1993, on “abstract deixis”).

The idea of being distanced can also appear in other ways. For example, pointing with the thumb can orient one to assertions made in the past, rather than in the present. One other phenomenon that is fundamentally communicative is that by pointing with the thumb at someone, speakers may distance themselves from those pointed at in a way that impersonalizes them or objectifies them. It is for this reason that doing this is seen as rude (and not only in Russian culture).

Finally, the lack of concrete localization of an object can be related, in a certain sense, to the opposite situation: rather than distancing oneself from an object or interlocutor, there can be a kind of reduction of distance when an as yet unfamiliar object is introduced in the listener’s attentional field (think of referring to something “on the side”). In this situation, the speaker also might not specifically localize that object in space, but simply inform the listener of its existence through thumb pointing in order to activate that referent in the conversation.

3.3 Pointing with the Head

We can compare this analysis with Grishina’s basic distinction between how deictic head gestures can be differentiated: the inclination of one’s head to the side (tilting), turning one’s head, moving one’s head forward, and pointing with the chin, as illustrated in Figure 10.2.

Figure 10.2 Types of deixis indicated with the head

(adapted from Reference GrishinaGrishina, 2017, pp. 137–140, with permission from the publisher, Languages of Slavic Culture)

One of the findings of her analyses of head gestures was that in deictic situations, head pointing always preceded hand pointing. The head and hand pointing can be seen as a deictic configuration in which the head gesture sets up what Grishina called “a degree of abstractness in the indication of an object.” If the deictic referent is located in the general zone of visibility, a forward movement of the head or chin pointing is used. If the object of pointing remains in an invisible zone – then a head tilt or head turn. Following the head gesture, pointing with the hand determines more fine-grained differences more precisely between the potential target referents within the targeted zone and provides a further, more detailed “referential and pragmatic calibration of the object of reference.” However, functionally, the hand pointing correlates with the head pointing and, practically speaking, cannot contradict it. Any head gesture can be combined with pointing with the open hand or the index finger, since these hand gestures can serve for the speaker’s visible zone as well as the zone that is not visible. Pointing with the thumb – as discussed above, connected with the nonvisible zone behind the speaker – correspondingly “prefers” the head tilt or the head turn, but not movement of the head forward or pointing with the chin. Grishina summarizes the various functions of these main categories of head and hand pointing for Russian speakers in Table 10.1.

3.4 Pointing Gestures: Some Conclusions

The main idea running through these analyses is one which was noted earlier, particularly for the gestural etimons, namely that of the motivation of a gestural form and the non-randomness of the functional-semantic parameters connected to it. There is an interdependence between them and there is a foundation for this connection, to be found, for example in the use of a specific body part in specific communicative and informational contexts, and this use has certain potential consequences in terms of the possible inferences that can be drawn from it. We can see this in the examples discussed above of the use of the thumb versus the head for pointing, summarized in Table 10.1.

Descriptions of some of the gestural forms considered by Grishina are still relatively rare in the gesture literature, such as head gestures, and among them, pointing with the head (as opposed to with the lips, as described in Enfield, 2009, Ch. 3, and Wilkins, 2003). Furthermore, as with the analysis of thumb pointing, the author pays attention to the combination of several gestures in one situation – and reveals the potential motivation of this combination. Finally, we have observed that Grishina’s analyses do not just speak of the functions (deictic, representational, pragmatic, etc.) of gestures in relation to the forms, but also of the meanings of certain uses of gesture, even using the term “semantics” in doing so. In this way, another aspect of her linguistic point of view on gestures comes through, as the term “semantics” is applied not just to the verbal elements, but also to the gestural ones, allowing for a distinction between semantic and pragmatic functions, as will become clearer below.

4.1 The Relation of Linguistic Units to the Gestures that Accompany Them

The first theme noted above, the relation of specific linguistic units to the gestures that accompany them, appears to be rather standard in gesture studies, if we think, for example, of the substantive amount of research on the lexicalization of motion events in different languages and the gestures used with them (e.g. Reference Gullberg and PavlenkoGullberg, 2011; Reference Kemmerer, Chandrasekaran and TranelKemmerer, Chandrasekaran, & Tranel, 2007; Reference Kita and ÖzyürekKita & Özyürek, 2003; Reference Özyürek, Kita, Allen, Furman and BrownÖzyürek, Kita, Allen, Furman, & Brown, 2005). In Grishina’s book, however, this theme was picked up in some other ways. For example, a large section is devoted to patterns of gesture used with Russian prefixes and verb roots, with attention to the many lexicogrammatical meanings expressed by Russian prefixes, which are known for their polysemy. One of the results of her research in this area – which, as usual in her work, was illustrated in detail and supported with statistical analyses – was that Russian verb prefixes are distinctly divided into two groups: the “path-based” and the “vector-based.” “Path-based” prefixes include, for example, pere- “re-”Footnote ⁴ and raz- “apart”; such prefixes can be seen as functioning like analogues of adverbs. The “vector-based” prefixes include v- “into,” do- “as far as,” and na- “upon,” for example. They are usually used in constructions with prepositions that pair with the prefixes (e.g. v-khodit’ v [literally “in-go in”] “to go into,” or do-khodit’ do [lit. “as far as-go as far as”] “to get to”). The vector-based prefixes are often used with verbs in metaphorical senses (e.g. vybrat’ “to choose” [lit. “out-take”], dovesti “to lead to” a given situation [lit. “as-far-as-lead”]).

In terms of gesture use, it is only the path-based prefixes whose trajectory and direction are conveyed in gestures; see Reference GrishinaGrishina’s example (2017, p. 257, example [7.3_14]) for the prefix-suffix combination s- -sja “come together”: “Rjabchik molchit, lesnichij krichit, vse {sbegajutsja}” “The grouse is silent, the forest ranger shouts, and everyone {comes running}” – with a two-handed gesture on the word {sbegajutsja} moving from a peripheral to a central gesture space after making a circular motion (example from the 1979 film Letuchaja mysh’ “The bat” by Ja. Frid et al.). The vector-based prefixes, however, were not found to express trajectory and direction of the motion being verbalized as much as simply indicating the starting points (e.g. with ot- “from,” s- “from,” vy- “out of”) and end points (v- “into,” do- “as far as,” pod- “up to”) of the movement. For example (Reference GrishinaGrishina, 2017, p. 262, example [7.3_24]) for the prefix vy- “out of”: “Vsjo otnjali prokljatye kapitalisty, s {raboty vygnali}!” “The damn capitalists have deprived us of everything, they fired people [lit.: from {work out-chased-they}” – with the speaker’s open left hand, palm down, moving rapidly in a straight line from right to left (from the central to the periphery).

The examination of gestures used with words involving these morphemes thus provides new insights into semantics from a usage-based perspective. This is particularly clear in cognitive linguistics, for example, where meaning can be viewed in terms of the concepts speakers are expressing (Reference Cienki, Müller, Cienki, Ladewig, McNeill and TeßendorfCienki, 2013): gesture can provide additional insights into speakers’ semantics.

4.2 The Relation of the Geometry of Movement to Semantics

The second of the three themes is in itself no trivial matter, and it is one that is also of interest from the point of view of typology. At issue here is how a given spatial trajectory relates cognitively to certain aspects of semantics. Grishina’s book considers, in detail, types of oscillating movements in gestures with two hands (both symmetrically with, and in opposition to, each other) or with one hand making circular movements (as if encircling a sphere or something round, making cyclic rotations, and others) and their general and individual meanings in relation to the speech. This shows in detail how tightly, and in what contexts, oscillating movements are connected with the semantics of “uncertainty” (which one sees even in the linguistic metaphor of to be wavering about something to mean to be uncertain about it), “ambivalence” (e.g. the temperature is vacillating between two values), “variety,” and others; the circular movements, however, convey meanings of “non-specificity,” “repetition,” “transformation of an object,” and others. (Compare the “swaying” gesture with similar functions found among German speakers, reported in Reference Bressem, Müller, Müller, Cienki, Fricke, Ladewig, McNeill and BressemBressem and Müller, 2014b, also noted semantically as the “Vague” recurrent gesture in Ladewig, this volume.)

4.3 The Relation of General Grammatical and Lexical Meanings and Functions in Language to Gestures Correlated with Them

The third theme builds a bridge from several large semantic categories – those such as quantifiers, negation, characterization of sizes of objects, expression of realis versus irrealis (hypotheticals), and others – to the gestural layer of communication. On this level of generalization, the research comes full circle: individual gestures and geometric movements express particular meanings and relate to specific situations, and specific types of gestures are characteristically used with particular linguistic units – all of this because there are concepts and categories for which gesture provides a natural, and perhaps necessary, means for marking. Here with representational gestures, we see what could also be considered “etimons” as discussed above – bases of gestural forms which can be seen as constituting the “etymologies” of particular gesture form/function pairings.

While it is known in general that gestures can serve as markers of sizes or quantities that can be easily perceivable visually, Reference GrishinaGrishina’s research shows (2017, p. 563) that in Russian, small sizes and quantities (both physical and abstract/metaphoric ones) are shown with at least three kinds of markers: the “fraction,” in which the pad of the index finger touches the thumbnail; the “pliers,” when the pads of the thumb and index finger as if hold a small object between them; and the three-finger pinch (thumb, index, and middle), as if holding a small amount of a substance, such as salt. These categories reflect specifications of gestures which otherwise might be grouped under a general label of the “precision grip” (cf. some similar but different distinctions made in Reference KendonKendon, 2004, Ch. 12, for Neapolitan Italian).

Gestures corresponding to lexicogrammatical categories are shown in this line of research to express various kinds of abstract meanings. Putting aside negation for the moment, a topic well known to gesture researchers (see Harrison, this volume) which is also illustrated in detail in various sections of Grishina’s book, we find consideration of gestures for the categories of irrealis (p. 244 ff.); in the MURCO corpus, the forms for expressing hypotheses and intentions correspond to gestural marking of future time (e.g. manual gestures with left to right motion), and for unreal conditions they correspond to the marking of past time (e.g. involving movement toward behind the speaker).

Separate sections of the book are devoted to the gestural marking of tense and aspect. In particular, the analysis of aspectual forms in Russian (perfective and imperfective aspect) reveal (pp. 573–587) that it is important to consider not only formal parameters of gestures, like “duration” or their “multiplex” nature (which are typical for use of the imperfect aspect in continuing and iterative contexts), but also the intensity of the gestures (the parameter of “energetics,” characteristic of the perfective aspect), which is connected with how the stroke of the gesture is produced, allowing for a more abrupt motion. It is significant that this semantic category is studied here not as an isolated subject, but in the form of gestalts – that is, the potential mutual influences the grammatical categories and gesture may have on each other. This includes consideration of linguistic parameters on different levels, such as the grammatical factors of tense, aspect, and modality in relation to the pragmatic factor of the type of illocution.

These kinds of interrelations are illustrated well in the detailed analysis of gestures used with quantifier words in Russian (pp. 463–481). In particular, four such words were selected for in-depth study due to their frequent use as quantifiers in the language and the fact that they are known to be semantically rich. These are: ves’ “whole/total,” vse “all,” “kazhdyj “each/every,” and ljuboj “whatever/anybody.” All of them can be used in singular and plural forms with any given lexeme. The gestures used with them, however, indicate that they all differ from each other conceptually. Gestures with ves’ tend to involve representation of a round form, reflecting the referent as a whole object, as if the quantity were being viewed from the outside; however, gestures with vse tend to show the indexical selection of objects in different ways, such as in a row or on a surface, and thus are as if viewing the quantification from the inside. As for kazhdyj, which concerns extensional quantification, the hand typically does not move, but fixates (e.g. with the three-finger pinch) on only one point, presenting in this way the whole selection of referents as a quantum. The quantifier ljuboj, which is semantically similar to kazhdyj in representing positive semantic polarity in this domain, was found to be used with a completely different type of gesturing, namely manual gestures of negation (e.g. palm down, sweeping to the side/peripheral space [as in Reference Bressem, Müller, Müller, Cienki, Fricke, Ladewig, McNeill and BressemBressem & Müller, 2014a]) or a side tilt and turn of the head. This reflects implicit negation, characteristic of the intention to express “it doesn’t matter what kind/it’s all the same which one”; this is negation which, interestingly, does not appear in syntactic constructions with the word ljuboj.

These various strands of research show that gesture accompanying lexicogrammatical categories contributes new qualities to them and creates new connections in terms of their meaning – and this multimodal perspective raises questions about the boundaries of these categories. The following section continues with consideration of movements accompanying speech which play a supporting role – auxiliary gestures, which few scholars in gesture studies have paid attention to before.

2.1 On the Hand Only (the “What”)

The typologies frequently used for gesture analysis in the past few decades may have been influenced by, and may be entangled in, an old tradition of taxonomy coming from antiquity (Reference Quintilian and CousinQuintilian & Cousin, 2003, Lib. XI). Quintilian’s classification is based on the hand and fingers. Witness the exaggeration of the hands in the early medieval illustrations of the Roman dramatist Terence’s works (Reference AldreteAldrete, 1999; Reference Radden KeefeRadden Keefe, 2021), and the focus on the hands continuing into Reference BulwerBulwer’s (1644) Chirologia. Reference EfronEfron (1942/1972), too, classified gestures focusing on the hands, leaving the other segments of the upper limb “in limbo.” Reference KendonKendon’s (2004) taxonomy is a hierarchy based first on hand shapes, second on the orientation of the palm, and third on the movement of the hand. Reference CalbrisCalbris’ (1990) typology of gestures highlights the importance of the palm of the hand, the fingers, and the head as “bodily vehicles” of meaning. In all of these approaches to gesture studies, the hand as a tool of interaction with the world is literally considered as the core of gestures. For Sign Languages, the tradition is similar. The first such notation system, created by Reference BébianBébian (1826) for French Sign Language, also focused on the hands. The twentieth century was no different, from Reference StokoeStokoe (1960/2005) to Reference SandlerSandler (1989), Reference BrentariBrentari (1998), Reference CuxacCuxac (2000), or Reference LiddellLiddell (2003). They all present a modeling of the hand excluding, in fact, the forearm, the arm, and the shoulders. The four parameters of Sign Language – the hand shape, the orientation of the palm, the location of the hand in space, and the movement – are all centered on the hand.

The distal part of the upper limb, the hand, is considered as the effective part of bodily communication. In the main studies of gesture, especially during the twentieth century, the part carrying the essential meaning is the hand: the rest of the upper limb is merely considered as a mean of displacement for the hand, allowing it to reach a given location. Reading between the lines, the conception of the meaningful part of gestures has probably been influenced by the practical dimension of gestures and the usefulness of the grasping part of the upper limb – the hand and the fingers.

It is true that analysis of certain gestures considers other segments of the upper limb, for example: shrugs of the shoulders expressing some kind of incapacity (Reference DarwinDarwin, 1872/1998; Reference DebrasDebras, 2017; Reference StreeckStreeck, 2009); quick forward movement of a shoulder expressing disinterest (Reference BoutetBoutet, 2010); a posture of arms akimbo as a mark of social supremacy (Reference Spicer, Bremmer and RoodenburgSpicer, 1991); crossed forearms in front of a speaker meaning prohibition (Reference CaradecCaradec, 2005, p. 145); the phallic forearm jerk as an insult (Reference CalbrisCalbris, 1990, p. 6); the “sidearm gut punch,” as Calbris wrote, expressing a retort (Reference CalbrisCalbris, 1990, p. 4); the forearms beneath one another on the chest in medieval illustrations depicting a situation of distress (Reference GarnierGarnier, 1989, p. 152), to name but a few. However, most of the analyses made during the last century seem to be cut off from or a bit disconnected from this reality of gestures.

2.2 Visuality as Exteriority (the “How”)

Beyond the potential influence of manual praxis on the way gestures have been analyzed, taking the more distal segment – the hand – as the locus of gestural activity could also come from the conception of what is relevant for speech in linguistics. The externalization of speech as a signal cut it off from the body. Apart from an articulatory approach to phonology (e.g. Reference Browman, Goldstein, Kelso, Rubin and SaltzmanBrowman et al., 1984), which seems to constitute a minority in the field, the main trends in phonology are based on acoustics. Therefore, what is seen as significant in speech is the externalized signal and not the possible bodily structuring leading to its production, which could also have been considered as speech. The constraints on how the most widely analyzed means of communication – speech – is most commonly studied skews how most researchers consider gestures.

The attention to externalization, noted above, detached from the producing body, focuses, de facto, on the receptive modality of language: the auditory channel. For gestures, the receptive modality – vision – is obviously sensitive to the segment that moves the most. This trend, considering meaning as an externalization (“ex-pression”), pushes attention toward that side of things more than to the gesture in its own right and in its unfolding. The image produced or the trajectory created, as results, are perceived as having more value than the gesture which builds them. Therefore, awareness of such significance is diverted from the track of production to the manual gesture and the trail it traces. In other words, preference is given to studying frozen drawings (artifacts) over analysis of the quality of the movement – the kinematics of the hand – which comes far before the construction of a gesture along the upper limb, and which has its own geometry, giving rise to how it moves. The latter is what constitutes a kinesiological approach.

Beside the praxiological argument and the “external” one (which is actually more of an epistemological issue), a third reason explaining the focus on the hand may exist. It is linked to the frozen representation we have of symbolic phenomena. As linguists, we are used to working with transcriptions of speech adapted from systems of writing. The flow of the speech is segmented and labelled. Writing fixes speech into static representations (e.g. letters, for languages written with an alphabet). Except for work by linguists who research prosody, a vast majority of publications deal with static representations of language because of the transcriptions used. Our difficulty in grasping dynamic phenomena is related to this kind of representation, namely a writing system. We do not know clearly what is at stake. The writing system could be the basis of our difficulty in understanding the dynamics of what is involved, but, more ontologically, writing could be a solution, coming from our difficulty with embracing dynamics. Regardless of the direction of the causality, transcription leads us toward tracks (writing, in this case) – leaving out, in the shadow, the trails traced and the articulatory gestures of the speech. This has consequences for the visual analysis of gesture, noted above.

Despite the existence of transcription systems for Sign Languages (such as HamNoSys [Reference Prillwitz, Leven, Zienert, Hanke and HenningPrillwitz, Leven, Zienert, Hankke, & Henning, 1989], SignWriting [Reference SuttonSutton, 1995], and the Berkeley Transcription System [Reference Hoiting, Slobin, Morgan and WollHoiting & Slobin, 2002]), none of them have become prominently widespread in their use; they are usually applied locally (e.g. in Hamburg or Berkeley for the systems mentioned). No conventionalized writing system exists for cospeech gestures. This lack is compensated for by using drawings or screenshots of gestures, chosen for their prototypicality, and sometimes complemented by arrows or lines indicating the direction of the movement (trajectories). These photographic representations, appearing to respect what is going on with peculiar gestures, provide the illusion of reality. They freeze gestures into postures, removing the dynamics. These representations cut off the gestures from the naturalness of their embodiment (kinesiology) and, then, the process of their production (kinematics) is devalued, yielding only the benefit of information on the direction of the movement of the hand (sometimes depicted by arrows).

These three reasons give arguments that explain the focus on the hand and the static conception we have of gestures. We describe and preferentially analyze one segment of the gestures, and do so according to an image of one moment; this sort of compression of time primarily creates a representation at the expense of leading to a genuine study of gestures and their deployment. Following the presentations above of the object studied and the conception of time in which it is placed, we present now the frame of reference of the space with which gestures are described.

2.3 Egocentric Frame of Reference (the “Where”)

The question of the space where gesturing unfolds has actually been addressed by Reference Kant, van Cleve and FrederickKant (1768/1991) in a wider spectrum. The issue tackled in his essay concerns the potential bases for differentiating a region of space as a relationship between two entities (situated in relation to one another) as opposed to absolute space – in other words, the question of the grounds on which the geometry rests. In the absence of any Cartesian coordinates or any measurement units, we can situate any object in relation to the living body. The categories left and right, which we always have “on hand,” establish the ultimate ground of this differentiation. In this sense, we do not need any coordinate to localize any entities (Reference Richir and SiksouRichir, 2005). In this way, a hand is situated up or down, a bit higher or less so, at the left or to the right, in front, far in front of myself, or behind. The body as the tool of this conception of measurement seems to be particularly appropriate as the origin and the frame of reference for gestures.

Considering the way gesture has been studied so far, the object of study is roughly unique (the hand), and is as if cut off, at least intellectually, from the upper limb; its representation, in drawings or screenshots, is unique as well, and, hence, determined by the singular point of view of the image. The hand (at the tip of the forearm) is the most mobile segment. It can translate (move without rotation or angular displacement) or rotate in space, if not freely, at least in almost all locations. Considering its distal specificity, almost detached from the body, a similar frame of reference could suffice for the nature of this segment seen in this way: (1) distal, (2) still dependent on the bodyFootnote ¹ but not really linked to it, and (3) lateralized. Among the three frames of reference described by Reference Levinson, Bloom, Peterson, Nadel and GarrettLevinson (1996) – absolute (and usually geocentric), intrinsic (and centered according to an entity), and relative (generally oriented according to the speaker) – the last one meets the three features mentioned above: (3) A lateralized, egocentric frame of reference (henceforth: “ego FoR”) applies simultaneously to the (2) specific features, still dependent on the body, and (1) the distal feature of the hand. Reference McNeillMcNeill (1992) situates the location of the hand as being around a speaker, in an ego FoR, as Kendon does (Reference KendonKendon, 2004). All of the approaches in Sign Language phonology (e.g. the Hand Tier Model [Reference SandlerSandler, 1989], the Prosodic Model [Reference BrentariBrentari, 1998], and the Hold-Movement Model [Reference Johnson and LiddellJohnson & Liddell, 2010, Reference Johnson and Liddell2011]) are based on this type of FoR. The terminology could change (ipsilateral vs. contralateral for the lateralization, distal vs. proximal for the distance from the body), but the principles of the location still rely on an ego FoR.

The choice of the ego FoR has another consequence for gestural phenomena. It is quite impossible to situate simultaneously two segments attached by a joint without having to refer to a local landmark situated on the segment considered (maybe an intrinsic FoR) or without any reference to the orientation. The description of any free moving entity in space – as the hand seems to be considered – entails six possible degrees of freedom (logically independent parameters along which values can vary): translational movements in the three directions of space (XX’, YY’, ZZ’) plus three rotations of the entity on itself: the yaw (rotation in the horizontal plane along its vertical axis), the pitch (rotation along its lateral axis), and the roll (rotation along its longitudinal/sagittal axis).

Considering, for instance, a gesture produced with the hand in front of the chest, fingers aligned, pointing toward the central space in front of the speaker, the palm presents several possible locations. The palm might be in a frontal plane in either of two orientations: toward the speaker or away from him. Alternately, the palm might be in a transverse plane in one of two orientations: palm down or up. The palm could be positioned in any intermediate location between these two planes. In this transversal plan where the palm is facing down, the elbow might be elevated higher than the palm, for example, in a gesture of pushing down, or, on the contrary, it might be lower than the palm, as in the beginning of a gesture indicating a growing size. If the elbow is in the same transversal plane of the palm, the gesture might depict a size (“this high”), or the beginning of an expression of totality (with a horizontal movement outward), or any growing or lowering movement. A movement of the wrist might express a withdrawal (extension of the wrist upward), a rollover (rotation of the wrist), or (at least in French culture [Reference Calbris and MontredonCalbris & Montredon, 1986, p. 75]) a “go away” gesture (pulling of the hand outward to the side, away from the median axis of the body, in what is known as abduction of the wrist).

This suggests that the framework of spatial reference above is ill suited to describe the requirement of the movement, and that we need to situate all segments of the upper limb according to their potentialities. A kinesiological basis for gesture analysis thus seems to be needed.

3.1 Intrinsic Frame of Reference on Each Segment

An intrinsic FoR locates the environment according to the entity considered. For the hand or forearm, their environment matches the possibilities of motion, that is, categories grouping the axes of the movements of each segment, in and of themselves, named proper movement. We differentiate a proper movement from a displacement. A forearm might be displaced by a proper movement of the upper arm and hence move incidentally, without any proper movement of the forearm itself, carrying the hand to the same location as the one that could be reached by a proper movement of the forearm.

The geometry of the joints between the segments determines the possible proper movement of each segment itself. Anatomy is relevant here. Movements are categorized according to dof which in this context correspond basically to a rotation of a segment around an axis which is usually situated on a joint. Nevertheless, we distinguish two types of dof: the one whose axis crosses joints, and the other whose axis runs along bones. The second type is distributed in two places: along the humerus (the bone from the shoulder to the elbow), and along the two bones extending from the elbow to the wrist: the radius (the outer and slightly shorter of the two bones) and the ulna (the inner and longer of the two). The other, whose axes cross the joints, are in the complex of the shoulder, in the joint of the arm, in the elbow, in the wrist, and at each knuckle of the fingers.

These dof involve a specific geometry per segment. All movements are generated on axes and each segment turns around them to a certain amplitude. This fact has consequences. It means that straight movements are not simple, but complex, requiring the involvement of at least two segments and three dof. They need a high level of coordination. On the contrary, arc movements may involve just one degree of freedom from one segment. Our conception of gesture complexity is changed. We are not dealing with a disembodied Euclidian space where arcs are the addition of a translation (movement along a line) coupled with a rotation, where circles are the continuity of a certain arc, and where lines seem to be the simplest geometric realization. With the body, space(s)Footnote ² is/are not just a locus where the hand traces a form, but one(s) where potentially all segments are able to “inform” us. A hierarchy can be drawn up. First, gestures inform us thanks to proper movements. They delineate both forms and the information so designed. We need to find out which gestures are concerned with which forms. Second, movements of segments can be recruited, by the tip of the upper limb – the hand or the fingers – or by any of the dof of any segments, to gesture in order to draw or point to something significant. These latter gestures might be seen in a Euclidian space, and we can speculate in these cases that the traditional ego FoR is available to situate the form drawn.

Two types of gestures have thus been identified here. In the first type, the embodied gestures design meaning by forms in non-Euclidian spaces of gestures, sometimes through their deployment along several segments. The second type depicts meaning in space through the form that is deployed.

3.2 Gestures Are Embodied before They Are Seen

Gestures are embodied before being seen and interpreted as drawings. Their structuration is made first logically and then physiologically by the production part of this gestural/visual channel of expression. The visual way of structuring gestures (categories of depiction and drawings of them, à la Reference Müller, Müller, Cienki, Fricke, Ladewig, McNeill and BressemMüller, 2014) comes afterward. Embodied gestures inhabit gestural forms produced in certain spaces. Obviously, these motions are made in space, but this fact is relevant just for gestures as structured by vision. A gesture structured by the arm puts the hand in a wide but delimited zone. For example, when the arms are pulled backward along one’s sides, with the palms of the hands facing forward (as in Figure 11.13.1), expressing incapacity, the hands might be in a zone which could be covered by a cone with its widest part on the sides of the trunk and in front of the speaker. Within this cone, the space usable by the hand or the forearm when they are not involved in the expression of incapacity might involve gesture structured by the environment – oriented, for instance, toward a situation on the side of the speaker. If the gesture is structured with the hand indicating a refusal (in the traditional description with the palm facing away from the speaker), the hand could be in one of several locations, facing the position of the object refused. For embodied gestures, space is not valuable; the forms of the segments with their movements or their postures are the only elements of meaning. These kinds of gestures can be structured visually at a secondary level, in that their meanings can be anchored in the space of interaction or in the environment. Certainly, the form of the gestures then seems to belong to the environment and, therefore, to be structured by how they are seen, but this only adds an indexical part to the already structured meaning. For the refusal gesture – an embodied gesture – the location of the hand in space certainly comes from the visual structuring. Whenever possible, embodied and visual structuring add to each other in gestures.

We will present the characteristics of embodied gestures through the dof distributed over the segments of the upper limb, focusing on their geometries, and their chaining. We will show how movement is transferred from segment to segment and concentrate on one of the key principles of the distribution of movement: the flow of gestures.

4.1 Characterization of the Degrees of Freedom (Geometry and Amplitude)

4.1.1 The Arm

The shoulder joint has three dof. Because of these three axes of rotation, it actually belongs to the enarthrosis family of ball and socket joints (Figure 11.1 inspired by Reference KapandjiKapandji, 1997, p. 5). Considering axis 1 in the first image in Figure 11.1, the transverse axis that is contained in the frontal plane, this degree of freedom defines the flexion/extension movement of the arm (Figure 11.1, second image, inspired by Reference KapandjiKapandji, 1997, p. 7), involving movement of the arm in the sagittal plane, passing through the shoulder. The amplitude of the flexion goes from 0° (Figure 11.1, first image) through 90° (Figure 11.1, second image), up to 180° (maximum, Figure 11.1, third image), whereas that of extension reaches a maximum at 50° (Figure 11.1, fourth image). These two extremes are only very rarely used in symbolic gestures and in Sign Language. On the other hand, a range between 30° of extension and 90° of flexion constitutes the usual zone of production for these gestures.

Figure 11.1 The three axes of the arm and the flexion/extension. First image: the three axes. Second and third images: the flexion of the arm. Fourth image: the extension.

The rotation of the shoulder joint around the sagittal axis (axis 2) brings the arm in a frontal plane to move away from the median plane by up to 180° by abduction (see Figure 11.2, second image, inspired by Reference KapandjiKapandji, 1997, p. 9). In the other direction, the arm approaches the median plane by passing in front of the chest up to an amplitude of 30° (Figure 11.2, third image [cf. Reference KapandjiKapandji, 1997, p. 7]). It should be noted that the presence of the trunk prevents a pure adduction (pulling toward the median axis of the body) from occurring. It is always made possible by a flexion of the arm so that it passes in front of the trunk or by extending the combined arm behind the trunk (Figure 11.2, third image). The adduction of the arm behind the trunk is very low. The abduction/adduction ranges used in gestures and Sign Language are between 90° and 20°.

Figure 11.2 Axis 2 and abduction/adduction of the arm. First image: axis 2. Second image: abduction. Third image: adduction.

The third axis of the shoulder runs along the humerus (the bone from the shoulder to the elbow). It defines an internal and external rotational movement (Figure 11.3, second image, adapted from Reference KapandjiKapandji, 1997, p. 11). The maximum amplitude of the exterior rotation is 80°, and the total amplitude of the interior rotation is 95°. The latter is about 30° when the arm remains stuck on the side of the body. To reach the remaining 65°, the arm must be shifted to the side by an abduction movement.

Figure 11.3 Axis 3 and exterior/interior rotation. First image: axis 3. Second image: exterior rotation. Third and fourth images: interior rotation.

4.1.2 The Forearm

The elbow joint is of the trochlear type (pulley-shaped), that is, it involves a fossa (depression or hollow area) on the humerus in front of a corresponding bone ridge on the ulna side. This type of articulation allows only one degree of freedom (axis 1, Figure 11.4). However, there is an overlap between the ulna and the radius (axis 3, see Figure 11.5, first image). It determines a second degree of freedom, as we will see. Thus, on the forearm, including the elbow, two dof exist (Figure 11.4, first image, and 11.5, second image, inspired by Reference KapandjiKapandji, 1997, p. 99). The flexion/extension moves the forearm forward from the frontal plane (around axis 1, see Figure 11.4) or, in other words, it aligns the forearm with the upper arm. The reference position (aligned upper arm and forearm) sets the extension and flexion at 0°. The flexion amplitude does not exceed 145°, the limitation being given by the touching muscle masses. The extension can exceed the set 0°, especially in the case of hyperlaxity (“double-jointedness”). The entire amplitude of this movement is used in both gestures and Sign Languages.

Figure 11.4 Axis 1 and flexion of the forearm. First image: axis 1. Second image: flexion of the forearm.

Pronation-supination is a rotational movement visible on the hand due to the overlapping bones of the forearm (around axis 3, see Figure 11.5). When the palm is turned forward, this is a complete supination position; the ulna and radius are in the same plane and do not cross each other (see Figure 11.5, third image). When the palm faces backward, the position involves a complete pronation; the two bones of the forearm overlap (Figure 11.5, first image). The axis of rotation of this movement runs along the forearm. The maximal amplitude of these two movements (pronation and supination) is 85° to 90° each (Figure 11.5, first and third images; cf. Reference KapandjiKapandji, 1997, p. 107). The neutral position, called the intermediate position of the hand, has the palm facing a sagittal plane, when one stands with one’s arms hanging relaxed on the sides of one’s body (Figure 11.5, second image).

Figure 11.5 Axis 3 and supination/pronation of the forearm. First image: pronation. Second image: axis 3. Third image: supination.

4.1.3 The Hand

The hand has two dof. In the palm plane, an abduction/adduction movement and perpendicular to the palm plane, a flexion/extension movement. In what is called the anatomical position (with hands hanging by one’s sides, but with the palms facing forward), the movement leading to an abduction pushes the distal end of the palm toward the side of the thumb. The amplitude does not exceed 35° (Figure 11.6, third vertical pair of images). Adduction, however, is carried out in the opposite direction, to the pinkie this time, and always in the plane of the palm. Its maximum amplitude is about 45° (Figure 11.6, first vertical pair of images; cf. Reference KapandjiKapandji, 1997, p. 151). This pair of movements involves the smallest amplitudes of the upper limb. We will return in Section 4.2 to the impact that these amplitudes have on the deployment of gestures and signs.

Figure 11.6 Axis 2 and abduction/adduction of the hand (two views of each). First vertical pair of images: adduction. Second vertical pair: axis 2. Third vertical pair: abduction.

The second degree of freedom – flexion/extension – carries the hand in front of the frontal plane for flexion and behind the same average plane of the anatomical position. The amplitudes of these poles are similar: 85° for each (Figure 11.7).

Figure 11.7 Axis 1 and extension/flexion of the hand. First image: extension. Second image: axis 1. Third image: flexion.

4.2 Variations of the Geometry of the Degrees of Freedom during Movement

Beside these simple movements, joints with two dof, such as enarthroses (ball and socket joints) and joints with condyles (bones with a rounded ending), when set in motion together, have an impact on a third dof when it is a longitudinal rotation (interior/exterior rotation for the arm; supination/pronation for the forearm). The two movements in question are referred to as diadochal movements, that is, successive movements around the same joint that are not performed in the same plane (Reference MacConaillMacConaill, 1946, Reference MacConaill1948). To illustrate this type of movement, let us place the left arm along the body; the forearm, in a flexion of 90°, points forward; the hand is in a so-called intermediate position: the palm is in the vertical plane, the fingers pointing forward, the thumb is oriented upward (see Figure 11.8, first image). The forearm is then affected by a movement of exterior rotation (Figure 11.8, first intermediate position). In a second step, the forearm is carried upward by a flexion movement followed by an interior rotation movement that seems parallel to it (see Figure 11.8, second intermediate position). At this high point, the forearm is returned to its initial position by a simple movement of extension. The final position differs from the initial position in that the thumb is now inward, with the palm facing downward (see Figure 11.8, final position). In fact, an involuntary pronation movement has occurred between the initial and final positions without this degree of freedom being set in motion at any time.

Figure 11.8 Pseudo paradox of Codman.

When we break down the voluntary movements, we notice that it is precisely in the sequence of the flexion of the forearm and the interior rotation that this pronation movement appears. The inversion of the order of movements has a repercussion on the dof affected by this involuntary movement. Let us put ourselves in the same initial position and reverse the movements. This time the forearm undergoes an interior rotation initially bringing the inside of the forearm against the chest and then a flexion followed by an exterior rotation. Finally, by the same extension movement of the forearm, we return to the initial position. The palm is in a supination position, turned upward this time. The order of the movements voluntarily activated on the forearm determines the repercussion it has in the involuntary movement.

The same mechanism of appearance of joint movement is at work in what is called Codman’s paradox (Reference CodmanCodman, 1934). This time, the movements of the arm are involved and the repercussion in the form of involuntary movements is spread on the arm as well as on the forearm. Let us start from a position with the arms and forearms along the body, with the palm of the hand in the sagittal plane, facing inward (see Figure 11.9, initial position). A first flexion movement of the arm of 180° carries the hand over the shoulder (see Figure 11.9, second position). A second 180° adduction movement of the arm also brings it back to its initial position on the side. The palm is still in a sagittal plane, but this time it is turned outward (see Figure 11.9, final position). Two conjunct movements have appeared: an exterior rotation of the arm and a complete supination. The hand has turned 180° (90° of exterior rotation and 90° of supination).

Figure 11.9 Codman’s paradox. First image: the initial position. Second image: flexion of the arm. Third image: adduction. Fourth image: final position; notice the position of the thumb.

(Adapted from Reference Boutet, Morgenstern, Cienki, Cienki and IriskhanovaBoutet, Morgenstern, & Cienki, 2018a, p. 150, Figure 5.4, with permission of John Benjamins Publishing Company.)

To show that these last two conjunct movements take part in the cycle of voluntary movements (abduction and flexion of the arm), we start from the same initial position of the arm, but with the palm in the sagittal plane turned outward. The forearm is therefore in a position of total interior rotation and the hand in a position of maximum pronation. In this new initial condition, the abduction movement of the arm does not reach 180°; the arm remains locked around a 90° position. The interior rotation and pronation are already in place, blocking the movement of the arm. The involuntary movements on the forearm and hand therefore serve as an “escape.” Like the diadochal movements seen on the forearm and hand, Codman’s paradox is sensitive to the order in which the cycle of voluntary movement is set in motion.

The third place of conjunct movements is on the hand. It is also due to a biomechanical mechanism and relates to the kinesiological approach to gestures. The situation on the wrist is basically the same as for the elbow and shoulder. The wrist has two dof (flexion/extension and abduction/adduction) on which is added pronation/supination, a longitudinal rotation that corresponds to this escape of a diadochal movement seen for the elbow.

In the same way as forearm movements, hand movements, when combined or sequenced, have an involuntary repercussion on pronation. The sequence of the movements of the two dof specific to the hand gives rise to an involuntary repercussion: pronation or supination, as shown in the images in Fig. 11.10.

Figure 11.10 Gimbal lock on the hand. The alignment is sensitive to the pole maintained. In extension, abduction and pronation are linked, whereas in flexion, abduction and supination are aligned.

A few gestures attest to this type of association between manual abduction/adduction and pronation/supination. The “goodbye” gesture (waving the hand) can be done via a slight oscillation in abduction and adduction or by a repeated movement of supination and pronation (Figure 11.11, first image) when the hand is in a position of extension. In another context, a gesture of reprimand in French culture (“Beware of spanking!”) shows the hand in a marked extension position also oscillating either slightly according to an abduction/adduction movement or according to an alternating and wider pronation/supination movement (Figure 11.11, second image, with the three robots). A third gesture in this extension position means a negation made with the index finger that presents an alternating movement either in abduction/adduction or in a wider way in a combination of pronation and supination (pronosupination, Figure 11.11, third image). A gesture with a marked meaning made in a position of flexion of the hand (expressing the exclamation “Oh là là!” in French) offers both possibilities of movement (abduction/adduction or pronation/supination) for the same meaning (Figure 11.11, fourth image). This last gesture – in a flexed position – reverses the associations. This time abduction is aligned with supination and adduction with pronation. The association of the poles is opposite for gestures in the extension position. These gestures therefore integrate perfectly a biomechanical/kinesiological fact without the alternation of abduction/adduction or pronosupination changing their meaning.

Figure 11.11 Examples of gestures with manual alignments. First image: “goodbye” gesture. Second image: “beware of spanking.” Third image: negation gesture. Fourth image: “Oh là là!” gesture

(screenshot from “10 Gestes et expressions françaises” https://youtu.be/qIJHm0wb3EE?t=102 by permission of the owner, Vincent Lefrançois).

The three sequences of movement – on the arm (Figure 11.9), forearm (Figure 11.10) and hand (Figure 11.11) – show (1) an involuntary movement on at least one degree of freedom, (2) a polar sensitivity of the involuntary movement to the sequence order, and (3) a physical problem of gimbal lock.

A gimbal is a ring suspended in such a way that it can rotate around an axis. On a ship, one may see several rings mounted inside each other in a fixed frame, holding the ship’s gyroscope level thanks to the different axes of rotation of the rings. However, if two of the gimbals end up in a parallel configuration, one of the axes of possible rotation is lost, which is called a gimbal lock. Similarly, the occurrence of involuntary bodily movement, mentioned above, occurs when two of the three axes are almost aligned, and even as this alignment approaches. Thus, the spaces of the embodied gestures are not Euclidean and they also interpenetrate from one segment to another. Their geometry is modified, reducing the number of dof, in an apparent way. Is the transient reduction in the number of dof – physiological or even kinematic – integrated into the very structure of the meaning of gestures? Or, on the contrary, do these reductions not have any effect on the structuring of meaning? The polar inversion of involuntary movement observed on the hand (the combination of supination/abduction or supination/adduction depending on flexion or extension) answers these questions. The oscillating examples, seen above, show that the meaning of the gestures uses the constraints of the gimbal lock by stabilizing their shapes. In general, the involuntary movements are downward, that is, the gimbal lock exerts its action mainly on the adjacent distal segment (the forearm for the arm, and the hand for the forearm).

After detailing the components of upper limb movement and the kinesiological relationships between dof, it is necessary to define how movement propagates along the upper limb in order to better understand how a regularity of gestural forms provides the stability needed for the emergence of meaning.

After recalling the definitions of proper movement and displacement of each segment, and the movement transferred or received from another segment, we will see how to determine from which segment a gestural movement originates. Finally, after having defined the propagation and origin, we will see that the notion of movement propagation flow allows us to identify the course of movement along the upper limb.

4.3 Proper Movement, Transfer, and Displacement

A segment can move as the result of its own mobility (proper movement), or as a result of the movement of another segment (displacement). These two cases do not produce the same effect. The actual mobility of a segment participates in the process itself, while displacement seems to contribute more to maintaining the gesture during the stroke: to situate it, to show its extent. We can define the proper movement of a segment as the set of rotations around at least one degree of freedom of the target segment, without this movement coming from a degree of freedom of another segment. We will call displacement of a segment a (series of) translation(s) due to a more proximal segment, without any change in the position of the dof of the displaced segment. In cases where a movement affects a segment, a distinction must be made between proper movement and transferred movement. We will call transfer of movement all rotations around at least one axis of one of the dof of the segment involved due to movement of at least one other degree of freedom, that is: Movement of one degree of freedom results in movement of another. This transfer may come from another proximal or distal segment or from the same segment.

Let us consider how these characteristics of movement play out in the gesture of “presentation” (Reference Calbris and MontredonCalbris & Montredon, 1986, p. 29; Reference KendonKendon, 2004, pp. 210–214, 265). A complete execution of this gesture involves movements of supination, abduction, and extension of the hand. This triple movement of the hand spreads over the forearm according to an extension and an interior rotation (see Figure 11.12).

Figure 11.12 Gesture of presentation while the speaker says, and seems to notice, “they have returned” (“Ils sont revenus”). This small clip corresponds to a decomposition of each image. There is a movement of extension, supination, and adduction of the right hand from image 1 to image 6.

(Reproduced from Reference Boutet, Morgenstern, Cienki, Cienki and IriskhanovaBoutet, Morgenstern, & Cienki, 2018b, p. 119, Figure 4.6, with permission of John Benjamins Publishing Company.)

The movement(s) of supination, abduction and/or extension of the hand can therefore be totally transferred to the forearm, but then for this gesture to be identified as a presentation, the hand must be in a position of minimum supination. This is then a transfer of movement, and the hand is not “displaced,” as the movement it undergoes is not only due to the movements of a proximal segment, since the movement of the forearm comes from the hand.

Let us now examine the case of a gesture where the forearm moves according to the same factors (extension and exterior rotation) but with the hand held in an intermediate position (neither pronation nor supination) (Figure 11.13, first image, Vladimir Putin’s example).

Figure 11.13 Gestures expressing incapacity. In the first image, Vladimir Putin’s hands are in an intermediate position (neither pronation nor supination), the forearms in an external rotating position and the arms in an abduction position due to the highly visible shrugging of the shoulders. In the second image, Nicolas Sarkozy’s hands are in a position of supination, the forearms in a position of exterior rotation, while the arms stuck to the sides are in an adduction position. Kobe Bryant, the basketball player in the third image, has his hands in a marked supination position with a strong flexion position of the forearms, with his upper arms at his sides. It seems that the supination position does not condition the positions of the forearm or upper arm.

This gesture (Figure 11.13) does not have the same meaning as the previous one (Figure 11.12); it expresses epistemic negation, in the sense that Reference GosselinGosselin (2005, p. 4) defines it, that is, a “de facto judgment resulting from a subjective evaluation,” an incapacity (Reference KendonKendon, 2004, p. 275; see also Reference Calbris and MontredonCalbris & Montredon, 1986, p. 80). Being held in this position, the hand is displaced this time. It could be in a more pronounced position of supination and extension (Figure 11.13 second image, Nicolas Sarkozy). The supination position, in particular, should be marked as in the previous gesture (see Figure 11.12) to allow a transfer of the manual extension as an extension of the forearm and the manual abduction as an exterior rotation. But this is not the case. It can therefore be concluded that the movement of the forearm does not depend on the movement of the hand. But, conversely, the movement of the hand comes from that of the forearm. The propagation of the movement along the upper limb is, this time, proximal-distal.

Does the segment that does not move constitute the origin of the total transfer of the movement or is it only affected by a displacement? To find out, we need to compare the number of dof moving on each of the segments, including the number that would have moved if movements had affected it. The number of oppositional types of movement involved (abduction/adduction, flexion/extension, pronation/supination) determines the origin of the gesture. The smaller the number, the closer we get to the segment that initiated the action. If we review some of the last gestures seen above, the one expressing incapacity (Figure 11.13) shows three kinds of manual movement (supination, extension, and abduction), two kinds on the forearm (exterior rotation and extension), and only one kind on the (upper) arm (adduction). The segment at the origin of this gesture is therefore the (upper) arm. The gesture of presentation – as soon as it shows a supination position, even an average one – can then be performed by an extension movement of the hand and can be transferred to the forearm by an exterior rotation and a movement of extension. The hand is therefore the segment at the origin of this gesture.

Another clue, more related to the deployment of the gesture, consists in looking at how the gesture runs along the upper limb, that is: determining the flow of propagation of the movement. This is what we will see in Section 4.4.

4.4 Flow of the Propagation of the Movement

The upper limb is a portion of the body across which movement propagates in a temporal order. Determining this order is difficult, primarily due to the fact that two time-delayed movements are in a causal relationship. For this, we need to set out rules.

With the inertia rule, proximal segments prevail as the movement proceeds along the upper limb. From the inertial point of view alone, the flow of movement is from the arm to the hand. Indeed, in relation to the body, the average adult proportion of the center of mass of each segment is 2.3 percent for the arm, 1.5 percent for the forearm, and 0.55 percent for the hand (Reference Dumas, Chèze and VerriestDumas, Chèze, & Verriest, 2007). There is therefore a decreasing gradient of the center of mass going from the arm to the hand. The inertia of these segments if they are in motion depends directly on the mass of each segment; it follows the gradient of the center of mass.

A second rule (joint limit rule) is counterbalanced by a gradient that marks the potential for transfers as a movement approaches its maximum amplitude. The effect of this rule may counteract the effects of the previous rule by shifting the transfer of movement to a larger inertia segment. Thus, maximum flexion of the hand boosts a transfer to the forearm, a segment with greater inertia. The abduction/adduction of the hand, whose amplitude is very limited, very often has its movement transferred to the forearm.

The third rule (geometric rule) concerns the parallelism of the axes of rotation between the segments. It concerns the direction of transfers. A motion transfer occurs under the necessary (but not sufficient) condition that the axis of rotation of the movement is parallel and adjacent to the one or ones on which the motion will be transferred. The direction of movement determines the type of the polar transfer (e.g. abduction vs. adduction).

The fourth rule (diadochal rule) concerns involuntary conjunct movements. Involuntary movements occur as soon as a joint has two dof and a movement affects both of them (so-called diadochal movement [Reference MacConaillMacConaill, 1946]). We then have an involuntary conjunct movement of the third degree of freedom.

The movements that appear in a gesture without being in this transfer ratio summarized in Table 11.1 constitute independent movements. Let us consider the cases of movements transferred along the upper limb. Three situations are defined: one for which the flow of movement propagation during a gesture or sign goes from a proximal segment to a distal segment (proximal-distal flow); another for which the flow of propagation goes from a distal segment to a proximal segment (distal-proximal flow); a third possibility for which there is no flow (without flow).

These flows are generally detectable by the staggered movement of adjacent segments. The handshape of the sign [PLACE] (“location”) in Figure 11.14 (upper image) in French Sign Language (LSF) starts with a movement of flexion of the proximal knuckles and spreads along the fingers with this movement of flexion before the fingertips come into contact with the pad of the thumb; this movement on the fingers corresponds to the transfer from the hand. The flow is proximal-distal. The handshape of the sign [SAISIR] (“grasp”), Figure 11.14 (lower image), shows an identical flexion movement of the fingers, but propagating from the tip to the root of the fingers. The flow is distal-proximal. The only difference between these two types of hand shape changes is the flow of the movement. Beyond the other differences between these two signs (one hand or two), the gestural expression of the idea of grasping and that related to location can only be distinguished from each other by the flow of the movement.

Figure 11.14 Sequencing of two signs in LSF. [LOCATION] (or place in spoken French) in the upper image and [GRASP] (saisir) in the lower image. The distinction between these two hand shapes is based on the flow of movement. Use of the images licensed by the online resource Spread the Sign

(https://www.spreadthesign.com/fr.fr/search/).

The two gestures of “presentation” and “epistemic negation,” seen in Figures 11.12 and 11.13, also present opposite flows: The gesture of “presentation” takes place according to a distal-proximal flow, while the gesture of “epistemic negation” shows a proximal-distal flow.

Behind these more obvious examples, some gestures and signs show less discernible flows. Here are some hints to be taken into consideration when determining the flow of movement of a gesture or a sign. The temporality of transfers is different depending on whether they respond to inertia (by a decreasing gradient from arm to hand [inertial rule]) or to a gradient from distal to proximal segments (the closer the movement is to a maximum amplitude, the greater the potential for transfer is [joint limit rule]) or when the movement transferred is involuntary (so-called diadochal movement [eponymous rule]). The latter is obviously the fastest since it is inseparably attached to the other two dof that move. Thus, a movement of flexion and exterior rotation of the forearm instantly implies the occurrence of an involuntary supination movement. In terms of differences in speed, the transfers coming in second place are those related to a movement of a distal segment close to its maximum amplitude (the joint limit rule). Finally, the slowest transfer is the inertial type that occurs on the more distal segments.

In the case of a total transfer of movement from a proximal segment such as the arm to a distal segment such as the hand, can the flow of movement be reversed? In other words, does the flow of movement depend on the gesture when it is moved to another segment or does the flow of movement depend on the segment and its situation? The answer seems to lean in favor of the second option. This is the case with the following “epistemic negation” gesture (Figure 11.15): The movement affects the hands according to an extension and abduction, and then it is transferred to the forearms. Thus, this realization reverses the flow from proximal-distal (seen in the gesture in Figure 11.13) to distal-proximal by motion transfer. It should be noted that the movement of the forearms is much less extensive than that of the hands.

Figure 11.15 Political speech of Nicolas Sarkozy, April 5, 2012. While he says, “There is nothing we can do about it” (“On n’y peut rien”), we notice a movement of extension of the hand and a slight supination between frames 2 and 4 and an abduction of the hand for frames 4 and 5, finally an exterior rotation of the forearm and an abduction of the arm. Here it is a form of epistemic negation whose flow is inverted (distal-proximal) which is similar to that expected for a gesture of presentation.

The complete transfer of movement to a more distal segment can redeploy the course of the gesture or sign by modifying the flow of the movement. If the general shape of the gesture is not modified, the flow can be reversed. The flow does not define a gesture. In some cases, it makes it possible to distinguish between gestures (e.g. a gesture of presentation and a typical gesture of epistemic negation). The flow structures how the gesture unfolds, while encompassing, in the same entity of meaning, the forms captured on the dof or even the segments that the movement affects. Once the movement is extended through the segments, the meaning can be realized on a segment far from the origin of the gesture. Depending on the dof set in motion on this remote segment(s), the flow can be reversed. If this is the case, then the realization borrows the flow of the other gesture in the pair that has the opposing form of flow. For example, in Figure 11.15, there is an “epistemic negation” gesture transferred to the hand and forearm with an inverted flow – distal-proximal – which, therefore, will have a presentational color; the shape deployed on the hand and forearm is still similar to that of the “epistemic negation” gesture. The gesture can then be interpreted as “the presentation of an external reality about which I can do nothing.” The flow takes precedence over the form; it does not supplant it. The gestural form of epistemic negation persists but goes into the background in favor of the flow, which is indeed in this case that of “presentation.” Hybrid gestural forms therefore appear; borrowing from a flow and the form generated by the opposite flow, their meaning is a kind of composite.

4.5 Transfer of Movement, Transfer of Meaning

When the movements are transfers, then the dof affected by them establish by habit a formal genealogy; we can see that the transferred forms share a family resemblance with the initial forms. This family resemblance brings together achievements played out on different segments. They can appear between two dof, for example, one on the hand and the other on the forearm. Thus, a lateral gesture of exterior rotation of the forearm (Reference KendonKendon, 2004, pp. 262–263), sweeping up to its maximum amplitude along a horizontal plane at constant speed, expresses, according to Kendon, an interrupted line of action. The hand in pronation – the palm in the same horizontal plane – follows the movement of the forearm, and therefore does not have its own movement. The meaning ascribed to it, that we prefer to call “totality,” lies in this type of movement of the forearm with this particular quality of constant speed (Figure 11.16).

Figure 11.16 “So there was blood everywhere” (“donc y avait du sang partout”). The gesture begins with a movement of the forearm (exterior rotation of frames 1 to 3), then this movement affects the hand (frames 3 and 4) in the same direction, toward the outside. This gesture corresponds to the expression of totality. Note that the speed of it is important here (seen in the fuzzy image of the hand), remaining relatively constant and high.

(From Reference Boutet, Jégo and MeyrueisBoutet, Jégo, & Meyrueis, 2018, p. 147, Figure 5.2, with permission of John Benjamins Publishing Company.)

Continuing with pronation, if it is the hand that sweeps the horizontal plane of a necessarily shorter trajectory by a movement of adduction up to its maximum amplitude, then the same notion of “totality” emerges, on the express condition that this gesture is made at constant speed. This constant speed is a strong indication that the transfer of movement comes from the forearm. In the event that this hand adduction movement was performed with significant acceleration, then it would have responded to a distal-proximal flow and would have been generated on the hand, thus expressing a “negation” (Figure 11.17). We would then have left this genealogy of form, in favor of the expression of simple negation.

Figure 11.17 “Finally, there were not many of us” (“Enfin, on n’était pas nombreux”). This gesture begins with a movement of the hand (outward). This movement spreads to the forearm (frames 4 and 5) with the same external direction. This gesture corresponds to a negation. During the first three frames, when the hand is moving, the upper part of the right wrist remains in the same place (as can be seen in these frames, given the position of the left elbow in the background). In the sweeping movement to the outside of the hand, one can also see the expression of totality, the hand then covering the supposed extent that the people occupied.

(From Reference Boutet, Jégo and MeyrueisBoutet et al., 2018, p. 147, Figure 5.3, with permission of John Benjamins Publishing Company.)

4.6 What Is a Gestural Form?

The concept of gestural form as a carrier of a singular value remains to be defined. There is no constant form/meaning for gestures in the sense of a trajectory deployed in space, as we have seen. There is no single form for any meaning. Therefore, we cannot consider a form in itself. The system of values associated with shapes includes what the recurrent deployment of movements on the upper limb captures in the dof in terms of movement, but also in terms of position. If there is no form in itself independent of the degree(s) of freedom that generate(s) it, there is also no form without considering the flow of propagation of movement. A gestural form (or gestural unit) is constituted by the degree(s) of freedom on which a movement unfolds over all or part of the upper limb and where this movement is frozen, as well as by the order in which this movement unfolds over the dof it passes through.

A gestural form is based on kinesiological considerations of movements or positions transferred between segments. It is based on the stabilized unfolding of a gesture on the upper limb. Let us call this stabilization an action schema. It is a grouping of movements or positions governed by a gestural form or unit determined by the organized sequence of movements of one or more segments.

There are forty simple action schemas in embodied gestures. “Simple,” because they are constrained only by kinesiological principles, without any other external constraints, be they iconic (in the sense of dependent on the environment which they depict or designate) or material (manipulation of objects or any form of coupling with the body). They are also “simple” because they are based on a free transfer of movement to the upper limb and a voluntary movement pulse of two dof on the two extreme segments of the upper limb. Reference BoutetBoutet (2010) gives an inventory of the action schemas; here we provide the action schemas of just two gestural units as examples.

4.7 Circulation of Movement Transfers in Action Schemas

Figures 11.18 and 11.19 represent action schemas in boxes and movement transfers as lines between the boxes. We will look at this in two examples.

The first example (Figure 11.18) shows a distal-proximal flow. The gesture starts on the hand with an adduction and an extension. Any other movements can be considered as transfers of movement. In theory, only manual adduction (Figure 11.18.1)Footnote ³ and extension (Figure 11.18.3) are voluntary movements. The first transfer of movement spreads over the forearm and results in an extension movement of the forearm (Figure 11.18.2). The association of adduction and extension of the hand leads to a diadochal consequence: an involuntary pronation movement is quickly in place (Figure 11.18.4 bis). We can see it in the third photograph (frame 3), which shows an extension of the forearm at its maximum (first intersegmental transfer), a hand in an almost maximum extension position, a manual adduction movement already involved in a transfer in the form of an exterior rotation of the forearm (Figure 11.18.4) and a pronation at its quasi maximum (Figure 11.18.5).

As the palm changes orientation under the effect of pronation, which gradually changes it from an inward orientation to a downward orientation, the transfer of movement on the forearm is also modified: From an extension, it changes to an exterior rotation under the effect of manual adduction. When the extension of the hand is at its maximum (frame 4) and the exterior rotation of the forearm has itself reached the maximum position due to the transfer, then the transfer coming this time from the joint limit of the manual extension is reflected on the forearm by a flexion (Figure 11.18.6) that concludes the intersegmental transfer series.

Figure 11.18 Action schema of the gestural unit “Refusal.” The flow is distal-proximal and it comes from the two voluntary movements – adduction and extension – on the hand.

In the second action schema (Figure 11.19), the gesture is initiated on the arm by two voluntary movements: an abduction (distance in a frontal plane from the plane of sagittal symmetry) and a flexion (movement in the sagittal plane in front of a frontal plane). The gestural unit corresponds to the meaning of disinterest (je m’en fiche “I don’t care”). Abduction causes an interior rotation (Codman’s paradox, Figure 11.19.1) while brachial flexion is transferred to the forearm by flexion for inertial reasons (Figure 11.19.2). In turn, the interior rotation and flexion of the forearm are immediately released by a supination movement (diadochal transfer; Figure 11.19.3). This transfer is done at the same time as the combination of the two forearm movements. It appears in the first position in the line corresponding to the hand. The flexion of the forearm is transmitted by the hand in a flexion movement (Figure 11.19.4). The succession of supination and manual flexion results in a manual adduction movement (Figure 11.19.5) which, in turn, accentuates the interior rotation movement that was already in place (Figure 11.19.6).

Figure 11.19 Action schema of the Gestural Unit je m’en fiche “l don’t care about that.” The flow is proximal-distal and it comes from the two voluntary movements – abduction and flexion – on the arm.

We have thus defined action schemas according to two criteria: movement and position.

4.8 Position or Movement

First, let us define position from a kinematic point of view. A position results from an absence of movement, maintained either by the absence of any muscle contraction or by a contraction of antagonistic muscles that maintain balance. In the case of an absence of muscle contraction, we call it a rest position. In this case, the position is not marked within any degree of freedom; on the contrary, it remains in an intermediate or quasi-intermediate position. In the case of a balance between two antagonistic muscles, the position can be extreme. A movement in a specific direction is obviously different from a position. However, a position maintained which entails a dynamic at work is not so different from a movement. In this case, a position returns to a suspended or maintained movement. This maintained movement is documented in the form of a hold (Reference Kendon, Siegman and PopeKendon, 1972; Reference Kita, van Gijn, van der Hulst, Wachsmuth and FröhlichKita, van Gijn, & van der Hulst, 1998).

4.9 Semiotic Value and Actualization Modes

Another question is: In which cases does the position have the same semiotic value as the one carried by the movement?

The dynamic involved brings semiotic values to a degree of freedom, in the manner of Aristotelian material causality. The movement is then the realization of this dynamic – its implementation – whose value is updated in a coextensive way as it unfolds along the path traversed. Basically, a gestural value is reached by the movement; it does not reside in the movement, otherwise it cannot be present only in the final position. Therefore, this value is coextensive with the degree of freedom. Movement is a way of realizing it in real time. The position of a segment on the extent of its degree of freedom is another way. A position maintained in its movement amplitude is sufficient to carry a value. For the values staggered along a degree of freedom, the movement deployed and the position occupied are two ways to actualize a value.

The actualization modes – movement or position – do not have the same temporal repercussions. Movement cannot be considered outside a temporal sequence; it therefore updates a value within a given time period. Once reached, the position updates the value for the duration that it remains in place. The limits of joint movement are particularly useful for maintaining these meaningful positions. Thus, for the gesture of refusal, a palm forward position, fingers upward, reflects this meaning. The joint-limits rule, the diadochal rule (Add+Exten> Pro), maintains this extreme position and finally semiotizes the space of each segment. Thus, the positional mode of actualization of the values is persistent in time: it can be reached without precise ongoing movement. Its value lasts as long as the position is in place because it is marked by kinesiology.

Another type of position appears here: that of location, as in one of the four parameters that is traditionally used for Sign Language analysis. The particularity of these locations is that the segment that takes place there is not the one that creates the associated positional value. Thus, when the hand is above the shoulder and composes the absence of responsibility (incapacity + disinterest, see Figure 11.13, third image), the value of disinterest was not forged by the hand, but by the arm and forearm. The hand instantiates a position outside itself; this particular position of the arms and forearms becomes a location with a singular value within which the hands take on this particular color of disinterest. An important point, shown here, is that this parameter of location, easily readable in an egocentric reference frame, can perfectly derive from the notion of position, which is part of a so-called intrinsic framing, depending in fact on all the segments of the upper limb. One of the four manual parameters of signed languages thus emerges for gesture.

4.10 Outcomes of Tests Applied to 20 Action Schemas

Of the 40 gesture units (GU) based on action schemas, 20 were tested using independent judges (Reference BoutetBoutet, 2010). To summarize this work, two concepts were tested: the semantic validation of a label for the 20 GUs and the structuring of the meaning involved, based on kinesiological principles.

Semantic validation was performed for 90 percent of the GUs. The only two GUs whose label did not correspond to that expected were for a label whose gestural realization corresponded to a more proximal segmental origin (the arm). It seems that semantic confusion almost systematically operates in the sense of a hyperclass. The fact that it is attached to a proximal segment raises the question of the history of development of gestural semantics. Is the arm the primary substrate of gestural meaning, which then extends to the hand, whose appearance remains to be discovered? Is this semantic subordination to the gestures generated on the arm a historical fact or a kinesiological fact, linked, for example, to the upper inertia of the arm?

Tests of the kinesiological structuring of gestural meaning were conducted by varying the form that the realizations of the 20 action schemas take on one or more segments. Here again, the average recognition rate was very good: 90 percent. It should be noted that the GUs that are structured on the arm are better recognized than those that are structured on the hand. We also note that the choice of labels, when they are not the expected ones, responds to a gradient that follows formal proximity. When proximity is a matter of amplitude, it is coupled with strong semantic proximity. The polar opposite movements (e.g. abduction vs. adduction) show quite clearly that semantic values are attached to them. The updating of these data is possible through the successive cross-references offered by these 20 units. Semantic features associated with the poles and positions in the action schemas must emerge from the formation of minimal pairs and polar inversions,.

2.1 Methodological Overview

2.2 Summary

In this section, I have provided an overview of some of the major categories of motion tracking, such as markerless, markered, and video-based tracking, and I have described how different implementations compare within each category. Each category, in turn, has its own unique advantages and disadvantages. Markered tracking provides the highest precision but is the most invasive and expensive option. This approach is best when lab-based data collection is not an issue, and high precision for very small or fast movements is required. Markerless tracking sacrifices some precision but is much less invasive and much lower in cost. This approach is useful for capturing three-dimensional movements when multiple camera angles are not possible. Video-based tracking is unique in that it can be applied to data that have already been collected. While tracking based on single videos is restricted to two-dimensional images, multiple angles can provide higher precision and three-dimensional tracking. AI-based tracking with multiple camera angles therefore provides a good trade-off between precision and cost. Overall, most of these approaches currently have software available that makes capturing data relatively straightforward. As the number of tracked points increases, however, the more complex the data and thus the more technical skill will be required from the researcher in order to process the data. Section 3 provides information about some currently available software and toolkits that support the researcher in this. A flowchart for method selection can be found in Figure 12.1.

Figure 12.1 A flowchart providing some suggestions for method selection. Note that this is a simplified version of the arguments, and researchers should also carefully read the subsections devoted to their method of choice

3.1 Motion Tracking for Qualitative Coding

Gesture researchers typically have the daunting task of manually annotating video data in order to address the research question in which they are interested. As pointed out in Section 1, the process of manual annotation is highly time-intensive and often relies on many subjective decisions being made. In the case of poststroke holds (i.e. the pauses in movement immediately after the main stroke phase of a gesture), for instance, the hands may not remain perfectly still, as people do not tend to remain perfectly still. This can make the exact timing of holds difficult to determine consistently. Even when the features that the researcher is interested in can be reliably defined and annotated, the process remains very time-consuming. Utilizing motion tracking can provide a way to facilitate this annotation process.

Manual annotation may address abstract features not currently available to motion-tracking software, such as the semantic content of a gesture or its pragmatic role. However, any gesture-coding scheme necessarily rests on the gestures themselves first being accurately identified, often including the timing of the onset and offset of the gesture. Onset/offset identification thus relies on the ability to accurately and consistently identify changes in movement speed. Motion tracking provides an ideal tool for this task as the tracked coordinates (i.e. the position of a given articulator, such as the hand) can be easily converted into a measure of speed. By defining a threshold for this speed measure, one can use the objectively calculated speed as a reliable indicator of when a movement has started or stopped. This provides a distinct advantage over manual coding not only in terms of time invested by the researcher, but also in the fact that this measure is not disrupted by a slow “fading out” of the movement. This can happen if a speaker’s hand continues to move after a gesture, but at a continuously slower speed as it gradually returns to rest. The exact offset can be much more difficult to determine by manual coders. Using a speed-based or motion-based threshold, this is no longer an issue. One very simple implementation of this approach is seen in Figure 12.2, where the raw X, Y, and Z coordinates are loaded into ELAN as time series.

Figure 12.2 Three-dimensional coordinates of the right hand visualized in ELAN as individual time series. From bottom to top, the three lines in this figure depict X, Y, and Z coordinate time series of the right hand. Peaks in the time series can be used to find maximum extension or final position of a movement, while changes in the values (i.e. when the time series begins to peak) can be used to find movement onsets.

This technique has recently been developed into a full movement-detection application by Reference Ripperda, Drijvers and HollerRipperda, Drijvers, and Holler (2020 under the name SPUDNIG (Speeding Up the Detection of Non-Iconic and Iconic Gestures). The application has a high accuracy rate for detecting true hand movements (99.5 percent), although not all movements are what researchers might consider true gestures. However, such an approach only requires the gesture researcher to then “clean” the annotations, which are automatically imported into ELAN by SPUDNIG.Footnote ¹ As the name implies, manual annotation is thus much faster with this approach (Reference Ripperda, Drijvers and HollerRipperda et al., 2020). An additional advantage of this application is that it comes with OpenPose motion tracking built in, and is operated via a user-friendly graphical user interface (GUI). Similarly, Reference Ienaga, Scotney, Saito, Cravotta and BusàIenega, Scotney, Saito, Cravotta, and Busà (2018) provided a method for detecting and classifying certain basic gesture types, also using movement trajectories as calculated by OpenPose. These gestures included negation (e.g. lateral movements of the hand, with flat palm held downwards or towards the interlocutor), palm up open hand, and “me” (i.e. self-directed deictic gestures, including one or two hands placed over the heart, or pointing with the index finger towards the speaker him/herself). Accuracy for this approach ranged from 92 percent for negation and palm up to 75 percent for “me.” Taking advantage of the depth information in the Microsoft Kinect, Reference Juszczyk, Ciecierski, Brill, Gudberg and SchwabJuszczyk and Ciecierski (2016) developed an application that directly feeds NEUROGES annotations into ELAN, together with the simultaneously recorded video, from the Kinect sensor. This means that a researcher can collect their standard video data and the corresponding annotations simultaneously (Reference Juszczyk, Ciecierski, Brill, Gudberg and SchwabJuszczyk & Ciecierski, 2016). This allows data acquisition conditions (e.g. camera angle, lighting) to be optimized on the spot, potentially leading to higher quality data. Many other approaches for gesture detection and/or classification have been developed for the Microsoft Kinect. These methods have largely been described empirically, and can thus be seen as a proof of concept, but have not been developed as tools that can easily be implemented by gesture researchers. While the Microsoft Kinect V2 is no longer in production, it has been replaced by the Microsoft Kinect Azure. In principle, this proof of concept for Kinect-based gesture detection/classification thus shows that these applications can be similarly implemented with the Kinect, which has a similar functionality, but with improved spatial resolution and tracking capability. While the aforementioned approaches may not provide the end-goal information about a gesture that the researcher is interested in, they can be employed to reduce the burden of annotation by providing a starting point, and by providing timing information that is objectively captured.

Motion-tracking data can also inform more abstract levels of manual annotation, providing new routes for investigation. For example, Reference MittelbergMittelberg (2018, Reference Mittelberg2019) used optical motion tracking to visualize the spatial and temporal dynamics of hand gestures in three dimensions, which then allowed a more holistic analysis of the gestures’ form and dynamics. In this way, we are more able to move beyond the restrictions of viewing single video frames and go towards visualizing the more complex, imagistic aspects of gesture production. This could also be used, for example, for making more accurate comparisons between movement qualities of gestures produced by different individuals or within different contexts.

3.2 Motion Tracking for Quantitative Analysis

Motion tracking not only supports qualitative analysis but can also be used for more quantitative approaches to studying gesture. In this section, I will discuss three possible avenues for such quantitative analysis, which span three levels of investigation: fine-grained finger movements, whole body kinematics, and the relation between gesture and speech.

4.1 The Influence of Kinematics on Gesture Comprehension

Perhaps the most straightforward way to utilize motion tracking to study gesture comprehension is by taking advantage of the fine-grained calculations that can be performed on motion-tracking data and utilizing this to model gesture or multimodal language comprehension. This can be particularly informative for studies seeking to examine the role of gestures on addressee comprehension (e.g. Reference Driskell and RadtkeDriskell & Radtke, 2003; Reference Gullberg and KitaGullberg & Kita, 2009; Reference Holler, Shovelton and BeattieHoller, Shovelton, & Beattie, 2009). Some of these studies, such as that of Gullberg and Kita, specifically examined aspects of gesture that can also be captured by kinematic analysis. In this case, it was the use of gesture holds. Extracting features of the gestures of interest, as described in Reference Trujillo, Vaitonyte, Simanova and ÖzyürekTrujillo et al., (2019) or Reference Pouw and DixonPouw and Dixon (2020), would allow researchers to test the specific qualities of gestures on an addressee’s comprehension. This approach is relevant for gesture researchers because it allows us to better understand the qualitative and quantitative differences that we find in gesture production, depending on the speaker, context, or other factors.

A simple way to go about utilizing gesture kinematics to study comprehension is to test whether some kinematic feature is correlated with comprehension performance. For example, Reference Gullberg and KitaGullberg and Kita (2009) used short video clips of gestures taken from a larger corpus as stimuli, and the authors tested information uptake based on speaker behavior, finding that uptake of information from a gesture is primarily influenced by whether or not the speaker looks at their own gesture. In a similar type of experiment, Reference Trujillo, Simanova, Bekkering and ÖzyürekTrujillo et al. (2020) showed videos of people performing pantomime gestures to a naive set of participants and asked them to identify the gesture that they saw. The results showed that the size and number of holds produced in a gesture were related to the accuracy with which the gesture was identified. While these are quite tightly controlled experiments, one could utilize a similar approach for more naturalistic tasks, such as that described by Reference Holler, Shovelton and BeattieHoller et al. (2009). In this study, the authors investigated how iconic gestures influence addressee understanding in face-to-face compared to video-mediated communication. The authors found that gestures produced in the face-to-face condition were equally, if not more, informative than those in the video condition in communicating spatial information. Given the potential heterogeneity present in the form of the gestures themselves, an interesting follow-up would be to use motion tracking to determine the kinematics of these gestures. This would allow us to determine what kinds of gestures are most beneficial in certain kinds of communication, or what features contribute to successful communication.

The approaches discussed above largely relate to empirical studies using experimental approaches. However, utilizing the fine-grained movement information from motion tracking can also be applied to approaches such as Conversation Analysis (see Reference Sidnell and StiversSidnell & Stivers, 2012, for an overview), where video data are increasingly being employed so that movement of the head, hands and body together with speech, eye-gaze, and other relevant cues can be analyzed together as one holistic, multimodal utterance (Reference MondadaMondada, 2016). In addition, in this approach, methods to automatically annotate movements (e.g. Reference Ripperda, Drijvers and HollerRipperda & Drijvers, 2020 and extract relevant kinematic information (e.g. Reference Trujillo, Vaitonyte, Simanova and ÖzyürekTrujillo et al., 2019) can be utilized to potentially reduce the burden of manual annotation and allow the researcher to get at more fine-grained aspects of the movements themselves.

4.2 Motion Tracking for Stimulus Design

When studying gestures from a comprehension perspective, perfectly controlling particular aspects of the gesture or its linguistic context can be difficult, if not impossible, in most natural settings. However, such a degree of control is invaluable in empirical research as it allows us to test the specific effects of particular parameters of interest. In the case of gestures, this could be the speech (or the timing thereof) that is paired with a gesture, or something like the size or speed of a gesture. Controlling these features is possible by reducing the visual information available, or by mapping motion-tracking data onto virtual agents, such as avatars.

Multimodal communication is highly complex, involving a multiplex of sometimes very subtle signals (Reference Holler and LevinsonHoller & Levinson, 2019). Thus, trying to investigate any isolated aspect of it can prove difficult. One solution is to use the motion-tracking data directly as stimuli for comprehension experiments. Studies on human perception have long used simplified visual displays to study perceptual processes, allowing the researcher to examine specific phenomena without the presence of extraneous visual information. A classic example that is quite relevant to gesture researchers is the perception of biological motion (Reference JohanssonJohansson, 1973), whereby a human figure is easily identified even when reduced to single points representing the major joints of the body (i.e. point-light displays). Even in such reduced visual displays, people are able to recognize individual actions such as jumping or walking (Reference Manera, Becchio, Schouten, Bara and VerfaillieManera, Becchio, Schouten, Bara, & Verfaille, 2011) as well as communicative gestures (see Reference Manera, Becchio, Schouten, Bara and VerfaillieManera et al., 2011 and Reference Okruszek and ChrustowiczOkruszek & Chrustowicz, 2019, for open databases of point-light gesture videos). Using point-light-displays is useful because it allows the researcher to remove or control other visual features, such as background, body shape, facial expression, and so on. Besides being a useful experimental control, it also makes it easier to utilize such complex movements in neuroimaging studies, where visual information must be carefully controlled, which can be difficult with videos of natural gesture production. One example of such use is in Reference Trujillo, Simanova, Bekkering and ÖzyürekTrujillo et al. (2020), where stick-figures derived from Kinect motion-tracking data were used to study the neural response to communicatively intended gestures. In this case, the motion-tracking data served to quantify the features of gestures produced in an earlier study, as well as to create well-controlled stimuli. This allowed the authors to test brain responses to specific quantitative differences in the gestures.

Another approach is to take motion-tracking data and map it onto a virtual agent. This allows the researcher to control or manipulate each aspect of behavior individually (e.g. gesture form, motion, or linguistic context) without greatly reducing the visual information available. For example, Reference Hassemer and WinterHassemer & Winter (2018) created a continuum of hand shapes in order to empirically test the hypothesis that finger position influences our interpretation of a particular type of gesture. Even if the behavior is not manipulated, having an interaction with a virtual agent allows the experiment to be much more replicable across participants. This can be particularly relevant for more interactive experiments, allowing researchers to maximize the ecological validity (Reference Holler and LevinsonHoller & Levinson, 2019; Reference Valenti and GoodValenti & Good, 1991) of their findings while still maintaining experimental control. Virtual reality provides a powerful way to place participants in a fully immersive interaction with such experimentally controlled virtual agents. For an overview of how motion tracking can be used to create virtual agents for the study of speech–gesture integration, see Reference Nirme, Haake, Gulz and GullbergNirme, Haake, Gulz, & Gullberg (2020). For an in-depth overview of how virtual reality can be used to study social interaction more broadly, see Reference Pan and HamiltonPan and Hamilton (2018). This approach could allow researchers to test hypotheses about gesture or multimodal communication in an environment that is tightly controlled but could be replicated with other languages, different labs, and so on.

5.1 Synchronization

When working with multiple data streams, it may be necessary to synchronize them in time. For example, if we record motion-tracking data with the Kinect and wish to use this data to inform qualitative coding, or we wish to use qualitative coding to define specific gestures that we wish to analyze with the Kinect data, we need to know which data points in the motion-tracking data correspond to which frames in the video file. Similarly, if we are interested in speech–gesture coordination, we need to know which motion-tracking data points correspond to which time points in the audio file.

The process of synchronizing motion-tracking data will likely be different depending on how the data were collected. One advantage of video-based tracking is that no synchronization should be necessary, as the motion-tracking data are calculated from the video, meaning that the frame numbers will be identical. One notable exception to this is the possibility in OpenPose to specify the frame rate at which it should perform the tracking. This frame rate should be identical to the frame rate of the video recording.

When using device-based motion tracking, motion tracking and video recording must either be started simultaneously, or the two streams must be realigned after data collection. As timing is often needed to be quite precise, it is important that recordings start at nearly the exact same time. While this is practically impossible to do manually, a single computer connected to multiple recording devices can be used to send simultaneous start signals. This approach has been advocated by Reference Pouw, Trujillo and DixonPouw et al. (2020), who also provide working code for the simultaneous recording of Polhemus motion-tracking and audio recordings. With this, researchers can ensure that the two data streams are started at the same moment. As different recording devices likely use different frame rates, it is important to resample the data to ensure that each time point in one data stream has data at the same time point in the other data stream. This can be easily achieved by resampling the “slower” signal (typically motion tracking) to match the higher-rate signal (typically audio). If audio and motion tracking are synchronized in their onset, it is then additionally possible to synchronize both streams to a video recording, provided it also has an audio recording embedded in it. This “chaining synchronization” has also been fully described in a step-by-step fashion by Reference Pouw, Trujillo and DixonPouw et al. (2020). While this paper specifically refers to the Polhemus being synchronized to an audio recording, the code provided by the authors could be modified for other types of data streams and a similar procedure can be followed.

An alternative approach to synchronizing data streams is to provide a signal that can be used to align two or more streams. For example, many standard experiment button-boxes have a light that indicates when a button has been pressed. If this light is visible on camera, and the button sends a signal to the computer controlling the motion tracker, then the timestamp of the button press and the appearance of the light can be used to align the data streams. In this case, the two streams can have these two marks as a “time zero,” where everything is calculated as time after this point. Then, any calculations will be aligned with one another. This approach has been used by Trujillo and colleagues to synchronize Kinect data with video (Reference Trujillo, Simanova, Bekkering and ÖzyürekTrujillo et al., 2018). Other manual approaches include clapping at the beginning of data collection, allowing the motion-tracking frames to be aligned with the video frames.

Regardless of the method chosen, it is important that (1) any data streams being used (e.g. motion, audio, video) have the same starting point, or at least a point that is aligned across all data streams, and (2) that if analyses are directly comparing information in different streams (e.g. for speech–gesture synchrony), the sampling rates are also identical.

5.2 Reliability

Motion-tracking technology is constantly developing, and purely software-based approaches (i.e. video-based tracking) in particular are quickly improving. In general, however, it should be kept in mind that as one moves away from marker- or IMU-based tracking, tracking precision will decrease. This may or may not be an issue for every research question, but one should always ensure that the method chosen fits the research question and study design. Section 2.1 provides information about the precision of different methods and gives some suggestions for what types of data (e.g. precise trajectories, general gesture use) the different approaches are best suited.

Even under ideal recording conditions, there may be errors or artifacts present in the data. Depending on the type of analysis, these can be removed manually (e.g. within the statistical program the researcher is using) when the researcher is confident that a given frame is unreliable, or problematic sections of the data can be excluded from statistical analysis. Therefore, it is also important to check data quality before beginning analyses. Freely available software can be found allowing the researcher to view motion-tracking data as a video, which can be used to check the quality of recordings. An example of such software is Mokka (Reference Barré and ArmandBarré & Armand, 2014).

5.3 Limitations

Motion-tracking technology comes in many varieties, as was discussed in previous sections. This means that tracking can be performed with very high precision of even very small articulators, for example, the fingers. The main limitations in motion tracking are thus not necessarily related to the precision of tracking per se, but rather to the practicalities of data collection and analysis.

In terms of data collection, researchers must decide if they want high-definition tracking using optical or electromagnetic tracking, which may have psychological effects on the participant (e.g. by making participants aware of what is being studied and thus biasing their behavior [Reference Rosenthal and RubinRosenthal & Rubin, 1978], or by the wires and markers influencing how comfortably a participant is moving), or if lower precision tracking is sufficient, which is less costly to acquire and less likely to influence participant behavior. Furthermore, high-definition tracking is typically not feasible outside a lab setting, meaning that lower-precision tracking may also be more flexible for fieldwork or researchers working outside of lab settings. This may impact the ability to accurately capture fine-grained finger movements, although the latest implementation of OpenPose also captures finger movements, and the quality of this tracking seems promising, though rigorous validation tests are lacking. IMU-based tracking, such as Perception Neuron, also provides a promising possibility for high-quality tracking outside of the lab. Field-tests will be needed to determine how much Perception Neuron, or other IMUs, are affected by ambient magnetism, and thus to what extent the system can be used in more naturalistic settings.

The other potential limitation of motion-tracking technology for gesture research is that this type of data requires different skill sets than what typical gesture researchers often have. Unlike video data, motion data are not analyzed visually, unless they are being used as a visual tool for qualitative coding. Motion-tracking data are often highly complex and have many degrees of freedom, as each sensor or tracked joint has positional data for two or three dimensions, plus angle or orientation, and each of these variables extends as a time-series, leaving the researcher with many processing and analytic decisions to make. These processing and analysis steps are typically done using programming scripts written to extract relevant features, find relevant events in the data (e.g. gesture occurrences), or perform more complex analyses. This can be a daunting task, particularly for those who are less skilled in (or even unfamiliar with) scripting. However, this situation is beginning to change, as the move towards more open science has also generated more interest in open code and applications. In just the past couple of years, several open-source tools have become available that make motion tracking more accessible to gesture researchers. Several of these tools are highlighted in Section 6.2.

6.1 Summary

Motion tracking can provide a powerful tool to support current gesture-research protocols and to advance the way that we study gesture and multimodal communication. Particularly with the rise of lower-cost, mobile tracking systems, the primary hurdles for utilizing these technologies for gesture research at present are (1) selecting the most appropriate method or device based on what it can and cannot do, and (2) knowing how to work with the data once one has it. For the first issue, this chapter has attempted to provide a gentle introduction to the different methodologies and the questions that one can answer with each of them. For the second issue, the chapter indicates some of the currently available software and code packages that researchers can use for their own projects. These will be summarized again in this section, for easy reference, followed by a brief discussion on ways the research community can continue to develop these methodologies.

6.2 Resources

While many applications exist for visualizing or working with motion-tracking data, I will focus here on open-source applications or bundles of code that have been provided by the community. These are particularly valuable because they are available to all researchers and we can continue to shape their development. Table 12.1 gives a brief description of what each resource provides, what type of code is provided, and the reference.

6.3 Future Directions

As we continue to create and publish databases and corpora of motion-tracking data, it will be important to be able to take advantage of that wealth of data. Analyses of motion form (Reference Schüller, Beecks, Hassani, Hinnel, Brenger, Seidl and MittelbergSchüller et al., 2017) could be combined with recently developed algorithms that allow researchers to search motion-tracking databases for similar spatiotemporal patterns (Reference Beecks and GrassBeecks & Grass, 2018). This could open new avenues for gesture research to take advantage of big data analyses, for example, by finding patterns of similar gestures across many different contexts, cultures, and so on. Together with graph-based visualizations, as proposed by Reference Pouw and DixonPouw and Dixon (2020), this could be used to investigate how gestures are (dis)similar on an even larger scale. Future methodological work should also aim to bring together multiple streams of data for visualization in order to take advantage of a more holistic approach to data analysis, similar to the workflow proposed by Reference Jégo, Meyrueis and BoutetJégo and colleagues (2019).

While there are many new and exciting developments being made in applying motion-tracking methods to gesture research, many of these applications require highly specialized knowledge. Therefore, future research should also aim to produce applications and toolboxes that make these approaches more accessible. At the least, those working with motion-tracking approaches can make their analysis code available, allowing others to benefit from the collective knowledge. Overall, this will help to advance gesture research for everyone.

6.4 Conclusion

Motion-tracking technology comes in many forms, which can be intimidating when getting started. However, this variety also ensures that there are methods suited to many different topics and researchers. Whether for supporting manual annotation, analyzing kinematics, or generating more naturalistic virtual stimuli, these motion-tracking methodologies have great potential for further raising the quality and innovativeness of gesture research.