Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-76fb5796d-2lccl Total loading time: 0 Render date: 2024-04-25T14:03:19.229Z Has data issue: false hasContentIssue false

21 - Gesture and Prosody in Multimodal Communication

from Part II - Topics and Settings in Sociopragmatics

Published online by Cambridge University Press:  01 April 2021

By
Lucien Brown  and
Pilar Prieto
Edited by
Michael Haugh ,
Dániel Z. Kádár  and
Marina Terkourafi
Michael Haugh
Affiliation:
University of Queensland
Dániel Z. Kádár
Affiliation:
Hungarian Research Institute for Linguistics, and Dalian University of Foreign Languages
Marina Terkourafi
Affiliation:
Leiden University
Get access

Summary

Whereas sociopragmatics as a field has been dominated by the analysis of verbal elements, this chapter adopts the perspective that sociopragmatic meanings are communicated in a multimodal fashion that encompasses prosody, gesture and other forms of nonverbal expression. We provide an overview of how prosodic and gestural means are employed for signalling information status, for marking the internal organization of speech and for communicating epistemic stance, (im)politeness, irony and speaker identity. This overview shows that prosody is closely integrated with gesture both at the temporal level and in the kinds of pragmatic meanings that these two systems are used to encode. We thus adopt the position, following the tenets of audiovisual prosody, that prosody and gesture can be considered as sister systems in the marking of sociopragmatic meanings in human communication.

Keywords

multimodalitygestureprosodyidentity(im)politeness
Type
Chapter
Information
The Cambridge Handbook of Sociopragmatics , pp. 430 - 453
Publisher: Cambridge University Press
Print publication year: 2021

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Ambrazaitis, G. and House, D. (2017), Multimodal prominences: Exploring the patterning and usage of focal pitch accents, head beats and eyebrow beats in Swedish television news readings. Speech Communication, 95, 100–113.CrossRefGoogle Scholar
Attardo, S. (2000). Irony as relevant inappropriateness. Journal of Pragmatics, 32, 793–826.CrossRefGoogle Scholar
Attardo, S., Eisterhold, J., Hay, J. and Poggi, I. (2003). Multimodal markers of irony and sarcasm. International Journal of Humor Research, 16, 243–60.Google Scholar
Bavelas, J. B., Gerwing, J. and Healing, S. (2014). Including facial gestures in gesture-speech ensembles. In Seyfeddinipur, M. and Gullberg, M., eds., From Gesture in Conversation to Visible Action as Utterance: Essays in Honor of Adam Kendon. Amsterdam: John Benjamins, pp. 15–34.Google Scholar
Benitez-Quiroz, F. C., Wilburb, R. B. and Martinez, A. M. (2016). The not face: A grammaticalization of facial expressions of emotion. Cognition, 150, 77–84.CrossRefGoogle ScholarPubMed
Boersma, P. and Weenink, D. (2017). Praat: Doing Phonetics by Computer [Computer program]. Version 6.0.35.Google Scholar
Borràs-Comes, J., Kiagia, E. and Prieto, P. (2019). Epistemic intonation and epistemic gesture are mutually co-expressive: Empirical results from two intonation–gesture matching tasks. Journal of Pragmatics, 150, 39–52.CrossRefGoogle Scholar
Breen, M., Fedorenko, E., Wagner, M. and Gibson, E. (2010). Acoustic correlates of information structure. Language and Cognitive Processes, 25(7–9), 1044–98.CrossRefGoogle Scholar
Brown, L. and Winter, B. (2019). Multimodal indexicality in Korean: “Doing deference” and “performing intimacy” through nonverbal behavior. Journal of Politeness Research, 15(1), 25–54.Google Scholar
Brown, L., Winter, B., Idemaru, K. and Grawunder, S. (2014). Phonetics and politeness: Perceiving Korean honorific and non-honorific speech through phonetic cues. Journal of Pragmatics, 66, 45–60.CrossRefGoogle Scholar
Brown, P. and Levinson, S. C. (1987). Politeness: Some Universals in Language Usage. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Bryant, G. A. (2010). Prosodic contrasts in ironic speech. Discourse Processes, 47(7), 545–66.CrossRefGoogle Scholar
Bryant, G. A. and Fox Tree, J. E. (2002). Recognizing verbal irony in spontaneous speech. Metaphor and Symbol, 17(2), 99–119.Google Scholar
Bryant, G. A. and Fox Tree, J. E. (2005). Is there an ironic tone of voice? Language and Speech, 48(3), 257–77.Google Scholar
Bryant, G. A. (2011). Verbal irony in the wild. Pragmatics and Cognition, 19(2), 291–309.Google Scholar
Cavé, C., Guaïtella, I., Bertrand, R., Santi, S., Harlay, F. and Espesser, R. (1996). About the relationship between eyebrow movements and F0 variations. Proceedings of ICSLP, 96, 2175–9.Google Scholar
Crespo-Sendra, V., Kaland, K., Swerts, M. and Prieto, P. (2013). Perceiving incredulity: The role of intonation and facial gestures. Journal of Pragmatics, 47, 1–13.Google Scholar
Culpeper, J. (2011a). Impoliteness: Using Language to Cause Offence, Cambridge: Cambridge University Press.Google Scholar
Culpeper, J. (2011). “It’s not what you said, it’s how you said it!”: Prosody and impoliteness. In Linguistic Politeness Research Group, ed., Discursive Approaches to Politeness. Mouton Series in Pragmatics 8. Berlin: Mouton de Gruyter, pp. 57–83.CrossRefGoogle Scholar
Devís, E. and Cantero, F. (2014). The intonation of mitigating politeness in Catalan. Journal of Politeness Research, 10, 127–49.Google Scholar
de Haan, F. (2001). The relation between modality and evidentiality. In Müller, R. and Reis, M., eds., Modalität und Modalverben im Deutschen. Linguistische Berichte 9. Hamburg, Germany: H. Buske, pp. 201–16.Google Scholar
Dimitrova, D. V., Stowe, L. A., Redeker, G. and Hoeks, J. C. (2012). Less is not more: Neural responses to missing and superfluous accents in context. Journal of Cognitive Neuroscience, 24(12), 2400–2418.Google Scholar
Dimitrova, D., Chu, M., Wang, L., Özyürek, A. and Hagoort, P. (2016). Beat that word: How listeners integrate beat gesture and focus in multimodal speech discourse. Journal of Cognitive Neuroscience, 28(9), 1255–69.CrossRefGoogle ScholarPubMed
Escandell, V. (2017). Intonation and evidentiality in Spanish polar interrogatives. Language and Speech, 60(2), 224–41.Google Scholar
Esteve-Gibert, N., Borràs-Comes, J., Asor, E., Swerts, M. and Prieto, P. (2017). The timing of head movements: The role of prosodic heads and edges. The Journal of the Acoustical Society of America , 6(141), 4727–39.Google Scholar
Esteve-Gibert, N. and Guellaï, B. (2018). Prosody in the auditory and visual domains: A developmental perspective. Frontiers in Psychology, 9.Google Scholar
Gibbon, D. (2009). Gesture theory is linguistics: On modelling multimodality as prosody. Proceedings of the Twenty-third Pacific Asia Conference on Language, Information and Computation, pp. 9–18.Google Scholar
González-Fuente, S. (2017). Audiovisual Prosody and Verbal Irony. Universitat Pompeu Fabra, Departament of Translation and Language Sciences.Google Scholar
Goodwin, M., Goodwin, C. and Yaeger-Dror, M. (2002). Multi-modality in girls’ game disputes. Journal of Pragmatics, 34, 1621–49.Google Scholar
Gravano, A. and Hirschberg, J. (2011). Turn-taking cues in task-oriented dialogue, Computer Speech and Language, 25, 601–34.CrossRefGoogle Scholar
Guellaï, B., Langus, A. and Nespor, M. (2014). Prosody in the hands of the speaker. Frontiers in Psychology, 5, 700.Google Scholar
Gussenhoven, C. (2002). Intonation and interpretation: Phonetics and phonology. In Bel, B. and Marlien, I., eds., Proceedings of the Speech Prosody. Aix-en- Provence, France: Université de Provence, pp. 47–57.Google Scholar
Gussenhoven, C. (2004). The Phonology of Tone and Intonation, Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Haugh, M., Chang, W. and Kádár, D. (2015). Doing deference: Identities and relational practices in Chinese online discussion boards. Pragmatics, 25(1), 73–98.Google Scholar
Henton, C. G. and Bladon, R. A. W. (1985). Breathiness in normal female speech: Inefficiency versus desirability. Language and Communication, 5, 221–7.Google Scholar
Hillewaert, S. (2016). Tactics and tactility: A sensory semiotics of handshakes in coastal Kenya. American Anthropologist, 118(1), 49–66.Google Scholar
Hübscher, I. (2018). Preschoolers’ pragmatic development: How prosody and gesture lend a helping hand. Unpublished doctoral dissertation, Universitat Pompeu Fabra, Barcelona, Spain.Google Scholar
Hübscher, I., Borràs-Comes, J. and Prieto, P. (2017). Prosodic mitigation characterizes Catalan formal speech: The Frequency Code reassessed. Journal of Phonetics, 65, 145–59.CrossRefGoogle Scholar
Hübscher, I. and Prieto, P. (2019). Gestural and prosodic development act as sister systems and jointly pave the way for children’s sociopragmatic development. Frontiers in Psychology, 10.CrossRefGoogle ScholarPubMed
Hübscher, I., Sánchez-Conde, C., Borràs-Comes, J., Vincze, L. and Prieto, P. (submitted). Multimodal mitigation: How facial and body cues index social meaning in Catalan requests.Google Scholar
Idemaru, K., Winter, B. and Brown, L. (2019). Cross-cultural multimodal politeness: The phonetics of Japanese deferential speech in comparison to Korean. Intercultural Pragmatics, 16(5), 517–56.Google Scholar
Idemaru, K., Winter, B., Brown, L. and Oh, G. E. (2020). Loudness trumps pitch in politeness judgments: evidence from Korean deferential speech. Language and Speech, 60, 123–48.Google Scholar
Ito, M. (2004). Politeness and voice quality – The alternative method to measure aspiration noise. Proceedings of the Second International Conference on Speech Prosody. Nara, Japan: International Speech Communication Association pp. 213–16.Google Scholar
Ito, K. and Speer, S. R. (2008). Anticipatory effects of intonation: Eye movements during instructed visual search. Journal of Memory and Language, 58(2), 541–73.Google Scholar
Jun, S. (2005). Prosodic Typology: The Phonology of Intonation and Phrasing, Oxford: Oxford University Press.Google Scholar
Jun, S. (2014). Prosodic Typology 2: The Phonology of Intonation and Phrasing, Oxford: Oxford University Press.Google Scholar
Kádár, D. Z. and Haugh, M. (2013). Understanding Politeness. Cambridge: Cambridge University Press.Google Scholar
Kendon, A. (2004). Gesture: Visible Action as Utterance, Cambridge: Cambridge University Press.Google Scholar
Kita, S. and Essegbey, J. (2001). Pointing left in Ghana: How a taboo on the use of the left hand influences gestural practice. Gesture, 1(1), 73–95.Google Scholar
Krahmer, E. and Swerts, M. (2005). How children and adults produce and perceive uncertainty in audiovisual speech. Language and Speech, 48(1), 29–53.Google Scholar
Krahmer, E. and Swerts, M. (2007). The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception. Journal of Memory and Language, 57(3), 396–414.CrossRefGoogle Scholar
Krivokapic, J., Tiede, M. K., Tyrone, M. E. and Goldenberg, D. (2016). Speech and manual gesture coordination in a pointing task. Proceedings of the Eighth International Conference on Speech Prosody, pp. 1240–44.Google Scholar
R Core Team. (2016). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. www.R-project.org/.Google Scholar
Swerts, M. and Krahmer, E. (2005). Audiovisual prosody and feeling of knowing. Journal of Memory and Language, 53(1), 81–94.Google Scholar
Ladd, D. R. (2008). Intonational Phonology. 2nd ed. Cambridge: Cambridge University Press.Google Scholar
Lempert, M. (2011). Barack Obama, being sharp: Indexical order in the pragmatics of precision-grip gesture. Gesture, 11(3), 241–70.Google Scholar
Levinson, S. C. and Holler, J. (2014). The origin of human multi-modal communication. Philosophical Transactions of the Royal Society of London, Series B, 369(1651), 20130302.Google Scholar
Lima, C. F., Castro, S. L. and Scott, S. K. (2013). When voices get emotional: A corpus of nonverbal vocalizations for research on emotion processing. Behavior Research Methods, 45(4), 1234–45.Google Scholar
Lin, H., Kwock-Ping, J. T. and Fon, J. (2006). An acoustic study on the paralinguistic prosody in the politeness talk in Taiwan Mandarin. Proceedings from ISCA Tutorial and Research Workshop on Experimental Linguistics, pp. 173–6.Google Scholar
McNeill, D. (1992). Hand and Mind: What Gestures Reveal about Thought, Chicago: University of Chicago Press.Google Scholar
McKinnon, S. and Prieto, P. (2014). The role of prosody and gesture in the perception of mock impoliteness. Journal of Politeness Research, 10(2), 185–219.Google Scholar
Nadeu, M. and Prieto, P. (2011). Pitch range, gestural information, and perceived politeness in Catalan. Journal of Pragmatics, 43(3), 841–54.CrossRefGoogle Scholar
Ochs, E. (1993). Constructing social identity: A language socialization perspective. Research on Language and Social Interaction, 26(3), 287–306.Google Scholar
Ohala, J. J. (1984). An ethological perspective on common cross-language utilization of F0 of voice. Phonetica, 41, 1–16.Google Scholar
Ohara, Y. (2001). Finding one’s voice in Japanese: A study of the pitch levels of L2 users. In Pavlenko, A., Brackledge, A., Piller, I. and Teutsch-Dwye, M., eds., Multilingualism, Second Language Learning, and Gender. New York: Mouton de Gruyter, pp. 231–54.Google Scholar
Ola Orie, O. (2009). Pointing the Yoruba way. Gesture, 9(2), 237–61.Google Scholar
Orozco, L. (2008). Peticiones corteses y factores prosódicos. In Herrera, Z. E. and Martín Butragueño, P., eds., Fonología instrumental. Patrones fónicos y variación. México DF: El Colegio de México, pp. 335–55.Google Scholar
Podesva, R. J. (2007). Phonation type as a stylistic variable: The use of falsetto in constructing a persona. Journal of Sociolinguistics, 11(4), 478–504.Google Scholar
Podesva, R. J. and Callier, P. (2015). Voice quality and identity. Annual Review of Applied Linguistics, 35, 173–94.Google Scholar
Prieto, P., Borràs-Comes, J., Tubau, S. and Espinal, T. (2013). Prosody and gesture constrain the interpretation of double negation. Lingua, 131, 136–50.Google Scholar
Prieto, P. and Borràs-Comes, J. (2018). Question intonation contours as dynamic epistemic operators. Natural Language and Linguistic Theory, 36(2), 563–86.Google Scholar
Prieto, P., Cravotta, A., Kushch, O., Rohrer, P. L. and Vilà-Giménez, I. (2018). Deconstructing beat gestures: A labelling proposal. Proceedings of the International Conference on Speech Prosody 2018, Poznan, Poland, 13–16 June.Google Scholar
Prieto, P., Puglesi, C., Borràs-Comes, J., Arroyo, E. and Blat, J. (2015). Exploring the contribution of prosody and gesture to the perception of focus using an animated agent. Journal of Phonetics, 49(1), 41–54.Google Scholar
Rockwell, P. (2000). Lower, slower, louder: Vocal cues of sarcasm. Journal of Psycholinguistic Research, 29(5), 483–95.Google Scholar
Roseano, P., González, M., Borràs-Comes, J. and Prieto, P. (2016). Communicating epistemic stance: How speech and gesture patterns reflect epistemicity and evidentiality. Discourse Processes, 53(3), 135–74.Google Scholar
Shattuck-Hufnagel, S. and Ren, A. (2018). The prosodic characteristics of non-referential co-speech gestures in a sample of academic-lecture-style speech. Frontiers in Psychology, 9, 1514.Google Scholar
Silverstein, M. (2003). Indexical order and the dialectics of social life. Language and Communication, 23, 193–229.Google Scholar
Stadler, S. (2007). Multimodal (Im)politeness: The Verbal, Prosodic and Non-Verbal Realization of Disagreement in German and New Zealand English. Hamburg, Germany: Verlag Dr. Kovac.Google Scholar
Swerts, M. and Krahmer, E. (2005). Audiovisual prosody and feeling of knowing. Journal of Memory and Language, 53, 81–94.Google Scholar
Tessendorf, S. (2007). Pragmatic functions of gestures: The case of the ‘brushing aside gesture’ in Spanish conversation. Proceedings of the International Pragmatics Association Conference 2007, Gothenburg, Sweden, 8–12 July.Google Scholar
Van Bezooijen, R. (1995). Sociocultural aspects of pitch differences between Japanese and Dutch women. Language and Speech, 38(3), 253–65.Google Scholar
Vanrell, M. M., Stella, A., Gili-Fivela, B. and Prieto, P. (2013). Prosodic manifestations of the Effort Code in Catalan, Italian, and Spanish contrastive focus. Journal of the International Phonetics Association, 43(2), 195–220.Google Scholar
Vanrell, M. M., Armstrong, M. and Prieto, P. (2017). Experimental evidence for the role of intonation in evidential marking. Language and Speech, 60(2), 242–59.CrossRefGoogle ScholarPubMed
Wilkins, D. (2003). Why pointing with the index finger is not a universal (in sociocultural and semiotic terms). In Kita, S., eds., Pointing: Where Language, Culture, and Cognition Meet. Mahwah, NJ: Erlbaum, pp. 171–215.Google Scholar
Winter, B. and Grawunder, S. (2012). The phonetic profile of Korean formal and informal speech registers. Journal of Phonetics, 40(6), 808–15.Google Scholar
Yamazawa, H. and Hollien, H. (1992). Speaking fundamental frequency pattern of Japanese women. Phonetica, 49, 128–40.Google Scholar
Yuasa, I. P. (2010). Creaky voice: A new feminine voice quality for young urban-oriented upwardly mobile American women? American Speech, 85(3), 315–37.Google Scholar
Zellers, M., House, D. and Alexanderson, S. (2016). Prosody and hand gesture at turn boundaries in Swedish. Proceedings of the International Conference on Speech Prosody, pp. 831–5.Google Scholar

Save book to Kindle

To save this book to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×