Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-848d4c4894-p2v8j Total loading time: 0.001 Render date: 2024-05-17T13:20:07.287Z Has data issue: false hasContentIssue false

19 - Nonverbal Communication and Context: Multimodality in Interaction

from Part V - Advances in Multimodal and Technological Context-Based Research

Published online by Cambridge University Press:  30 November 2023

By
Pauline Madella  and
Tim Wharton
Edited by
Jesús Romero-Trillo
Jesús Romero-Trillo
Affiliation:
Universidad Autónoma de Madrid
Get access

Summary

Traditionally, the study of linguistics has focused on verbal communication. In the sense that linguistics is the scientific study of language, the approach is perfectly justified. Those working in the sub-discipline of linguistic pragmatics, however, are faced with something of a dilemma. The aim of a pragmatic theory is to explain how utterances are understood, and utterances, of course, have both linguistic and nonlinguistic properties. As well as this, current work in pragmatics emphasizes that the affective dimension of a speaker’s meaning is at least as important as the cognitive one, and it is often the nonlinguistic properties of utterances that convey information relating to this dimension. This chapter highlights the major role of nonverbal “modes” of communication (”multimodality”) in accounting for how meaning is achieved and explores in particular how the quasi-musical contours we impose on the words we say, as well as the movements of our face and hands that accompany speech, constrain the context and guide the hearer to our intended meaning. We build on previous exploration of the relevance of prosody (Wilson and Wharton 2006) and, crucially, look at prosody in relation to other nonverbal communicative behaviors from the perspective of Relevance Theory. In so doing, we also hope to shed light on the role of multimodality in both context construction and utterance interpretation and suggest prosody needs to be analyzed as one tool in a set of broader gestural ones (Bolinger 1983). Relevance Theory is an inferential model, in which human communication revolves around the expression and recognition of the speaker’s intentions in the performance of an ostensive stimulus: an act accompanied by the appropriate combination of intentions. This inferential model is proposed as a replacement for the traditional code-model of communication, according to which a speaker simply encodes into a signal the thought they wish to communicate and the hearer retrieves their meaning by decoding the signal they have provided. We will argue that much existing work on multimodality remains rooted in a code model and show how adopting an inferential model enables us to integrate multimodal behaviors more completely within a theory of utterance interpretation. As ostensive stimuli, utterances are composites of a range of different behaviors, each working together to form a range of contextual cues.

Keywords

non-verbal behaviours –inferential pragmatics –relevance –multimodal prosody –gesture –vocal pointing
Type
Chapter
Information
The Cambridge Handbook of Language in Context , pp. 419 - 435
Publisher: Cambridge University Press
Print publication year: 2023

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Abercrombie, D. (1968). Paralanguage. International Journal of Language & Communication Disorders, 3(1), 55–59.CrossRefGoogle ScholarPubMed
Birdwhistell, R. L. (1970). Kinesics and Context: Essays on Body Motion and Communication. Philadelphia, PA: University of Pennsylvania Press.Google Scholar
Blakemore, D. (1987). Semantic Constraints on Relevance. Oxford: Blackwell.Google Scholar
Blakemore, D. (2002). Relevance and Linguistic Meaning: The Semantics and Pragmatics of Discourse Markers. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Bolinger, D. (1964). Around the edge of language: Intonation. Harvard Educational Review, 34, 282–296.CrossRefGoogle Scholar
Bolinger, D. (1972). Accent is predictable (if you’re a mind-reader). Language, 48(3), 633–644.CrossRefGoogle Scholar
Bolinger, D. (1983a). Where does intonation belong? Journal of Semantics, 2(2), 101–120.CrossRefGoogle Scholar
Bolinger, D. (1983b). Intonation and gesture. American Speech, 58(2), 156–174.CrossRefGoogle Scholar
Bolinger, D. (1983c). The inherent iconism of intonation. In Haiman, J. (ed.), Iconicity in Syntax (pp. 97–109). Amsterdam: John Benjamins.Google Scholar
Brazil, D. (1975). The Communicative Value of Intonation in English. Birmingham: University of Birmingham.Google Scholar
Calhoun, S. (2009). What makes a word contrastive: Prosodic, semantic and pragmatic perspectives. In Barth-Weingarten, D., Dehé, N., and Wichmann, A. (eds.), Where Prosody Meets Pragmatics: Research at the Interface (pp. 53–78). Bingley: Emerald. CrossRefGoogle Scholar
Chen, A., and Gussenhoven, C. (2003). Language-dependence in signalling of attitude in speech. In N. Suzuki and C. Bartneck (eds.), Proceedings of Workshop on the Subtle Expressivity of Emotion, CHI 2003 Conference on Human and Computer Interaction.Google Scholar
Clark, B. (2007). Blazing a trail: Moving from natural to linguistic meaning in accounting for the tones of English. In Nilsen, R. A., Amfo, N. A. A., and Borthen, K. (eds.), Interpreting Utterances: Pragmatics and Its Interfaces: Essays in Honour of Thorstein Fretheim (pp. 69–81). Oslo: Novus.Google Scholar
Clark, B., and Lindsey, G. (1990). Intonation, grammar and utterance interpretation. University College London Working Papers in Linguistics, 2, 32–51.Google Scholar
Clark, H. (1996). Using Language. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Claudel, P. [1946] (1990). L’oeil ecoute. Collection Folio essais (127), Gallimard Education.Google Scholar
Ekman, P. (1999). Emotional and conversational nonverbal signals. In L. Messing, and R. Campbell, (eds.), Gesture, Speech and Sign (pp. 45–57). Oxford: Oxford University Press.Google Scholar
Escandell-Vidal, V. (1998). Intonation and procedural encoding: The case of Spanish interrogatives. In Rouchota, V. and Jucker, A. (eds.), Current Issues in Relevance Theory (pp. 169–203). Amsterdam: John Benjamins.Google Scholar
Escandell-Vidal, V. (2002). Echo-syntax and metarepresentations. Lingua, 112(11), 871–900.CrossRefGoogle Scholar
Forceville, C. (2021). Multimodality. In Xu, W. and Taylor, J. R. (eds.), The Routledge Handbook of Cognitive Linguistics (pp. 676–687). New York/London: Routledge.Google Scholar
Fretheim, T. (2002). Intonation as a constraint on inferential processing. In Bel, B. and Marlien, I. (eds.), Speech Prosody 2002: Proceedings of the First International Conference on Speech Prosody, Aix-en-Provence, 11–13 April 2002 (pp. 59–64). Aix-en-Provence: Laboratoire Parole et Langage.Google Scholar
Fridlund, A. (1994). Human Facial Expression: An Evolutionary View. San Diego: Academic Press.Google Scholar
Gigerenzer, G., Todd, P., and the ABC Research Group (1999). Simple Heuristics That Make Us Smart. Oxford: Oxford University Press.Google Scholar
Grice, H. P. (1957). Meaning. Philosophical Review, 66, 377–388.CrossRefGoogle Scholar
Grice, H. P. (1989). Studies in the Way of Words. Cambridge, MA: Harvard University Press.Google Scholar
Gussenhoven, C. (1984). On the Grammar and Semantics of Sentence Accents. Dordrecht: Foris.CrossRefGoogle Scholar
Gussenhoven, C. (2002). Intonation and interpretation: phonetics and phonology. In Bel, B. and Marlien, I. (eds.), Speech Prosody 2002: Proceedings of the First International Conference on Speech Prosody, Aix-en-Provence, 11–13 April 2002 (pp. 47–57). Aix-en-Provence: Laboratoire Parole et Langage.CrossRefGoogle Scholar
Gussenhoven, C. (2006). Semantics of prosody. In Brown, K. (ed.), Encyclopedia of Language and Linguistics, 2nd ed. (Vol. XI, pp. 170–172). Oxford: Elsevier.CrossRefGoogle Scholar
Hall, A. (2004). The meaning of but: a procedural reanalysis. UCL Working Papers in Linguistics, 16, 199–236.Google Scholar
Halliday, M. (1963). Explorations in the Function of Language. London: Arnold.Google Scholar
Halliday., M. (1967). Intonation and Grammar in British English. The Hague: Mouton.CrossRefGoogle Scholar
Hagoort, P., and Berkum, J. van (2007). Beyond the sentence given. Philos. Trans. R. Soc. B, 362, 801–811.CrossRefGoogle ScholarPubMed
Hauser, M. (1996). The Evolution of Communication. Cambridge, MA: MIT Press.Google Scholar
Hirschberg, J., and Ward, G. (1995). The interpretation of the high-rise question contour in English. Journal of Pragmatics, 24, 407–412.CrossRefGoogle Scholar
Hooff van, J. (1972). A comparative approach to the phylogeny of laughter and smiling. In Hinde, R. (ed.), Non-verbal Communication (pp. 209–238). Cambridge: Cambridge University Press.Google Scholar
House, J. (1990). Intonation structures and pragmatic interpretation. In Ramsaran, S. (ed.), Studies in the Pronunciation of English (pp. 38–57). London: Routledge.Google Scholar
House, J. (2006). Constructing a context with intonation. Journal of Pragmatics, 38(10), 1542–1558.CrossRefGoogle Scholar
House, J. (2007). The role of prosody in constraining context selection: A procedural approach. Nouveaux Cahiers de Linguistique Française, 28, 369–383.Google Scholar
Imai, K. (1998). Intonation and relevance. In Carston, R. and Uchida, S. (eds.), Relevance Theory: Applications and Implications (pp. 69–86). Amsterdam: John Benjamins.Google Scholar
Kahneman, D. (2011). Thinking Fast and Slow. New York: Farrar, Straus and Giroux.Google Scholar
Kendon, A. (1972). Some relationships between body motion and speech. In Siegman, A. and Pope, B. (eds.), Studies in Dyadic Communication (pp. 177–210). New York: Pergamon Press.Google Scholar
Kendon, A. (1980). Gesticulation and speech: Two aspects of the process of utterance. In Key, M. R. (ed.), The Relationship of Verbal and Nonverbal Communication (pp. 207–227). The Hague: Mouton and Co.Google Scholar
Kendon, A. (2004). Gesture: Visible Action as Utterance. New York: Cambridge University Press.CrossRefGoogle Scholar
Kendon, A. (2014a). The poly-modalic nature of utterances and its relevance for inquiring into language origins. In Dor, D., Knight, C., and Lewis, J. (eds.), The Social Origins of Language (pp. 67–76). Oxford: Oxford University Press.Google Scholar
Kendon, A. (2014b). Semiotic diversity in utterance production and the concept of “language.” Philos. Trans. R. Soc. Lond. B Biol. Sci., 369(1651), 20130293. doi: 10.1098/rstb.2013.0293.CrossRefGoogle ScholarPubMed
König, E. (1991). The Meaning of Focus Particles: A Comparative Perspective. London: Routledge.Google Scholar
Kress, G., and Leeuwen, T. van (2001). Multimodal Discourse: The Modes and Media of Contemporary Communication. Oxford: Oxford University Press.Google Scholar
Ladd, R. (1978). The Structure of Intonational Meaning. London: Indiana University Press.Google Scholar
Ladd, R. (1996). Intonational Phonology. Cambridge: Cambridge University Press. 2nd ed. 2008.Google Scholar
Ladd, R. (2008). Intonational Phonology, 2nd ed. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Madella, P. (2021). Prosodic pointing: From pragmatic awareness to pragmatic competence in Chinese hearers of L2 English. Ph.D. thesis, University of Brighton.Google Scholar
Madella, P. (forthcoming). Relevance and multimodal prosody: implications for L2 teaching and learning. In Wharton, T., Maillat, D., Jagoe, C., and Scott, K., Relevance in Mind, Front. Psychol.Google Scholar
Madella, P., and Romero-Trillo, J. (2019). Prosodic pointing in inferential comprehension: The application of relevance theory to L2 listening instruction. Letrônica, 12(4), 1–17.CrossRefGoogle Scholar
McNeill, D. (1992). Hand and Mind. Chicago: University of Chicago Press.Google Scholar
McNeill, D. (1985). So you think gestures are nonverbal? Psychological Review, 92(3), 350–371.CrossRefGoogle Scholar
McNeill, D., ed. (2000). Language and Gesture. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Mehrabian, A. (1971). Silent Messages: Implicit Communication of Emotion and Attitudes. Belmont, CA: Wadworth.Google Scholar
Noveck, I. A. (2004). Pragmatic inferences related to logical terms. In Noveck, I. A. and Sperber, D. (eds.), Experimental Pragmatics (pp. 301–321). London: Palgrave Macmillan. CrossRefGoogle Scholar
O’Connor, J., and Arnold, G. (1973). Intonation of Colloquial English. Harlow: Longman.Google Scholar
Pierrehumbert, J., and Hirschberg, J. B. (1990). The meaning of intonational contours in the interpretation of discourse. In Cohen, P.. Morgan, J., and Pollack, M. (eds.), Intentions in Communication (pp. 271–311). Cambridge MA: MIT Press.Google Scholar
Poyatos, F. (1983). New Perspectives in Nonverbal Communication. Oxford: Pergamon Press.Google Scholar
Riley, P. (1979). Viewing comprehension: L’oeil écoute. In The Teaching of Listening Comprehension: Papers Presented at the Goethe Institut Colloquium Held in Paris in 1979. London: The British Council.Google Scholar
Scott, K. (2017a). Ostension, Expectations and Non-encoded Meaning [Presentation]. International Pragmatics Association (IPrA) Conference, Belfast, UK.Google Scholar
Scott, K. (2017b). Prosody, procedures and pragmatics. In Depraetere, I. and Salkie, R. (eds.), Semantics and Pragmatics: Drawing a Line. Logic, Argumentation & Reasoning, 11 (pp. 323–341). Berlin: Springer.Google Scholar
Scott, K. (2021). Contrastive stress in English meaning, expectations, and ostension. In Ifantidou, E., de Saussure, L., and Wharton, T. (eds.), Beyond Meaning, Pragmatics & Beyond New Series 324 (pp. 29–42). Amsterdam: John Benjamins.CrossRefGoogle Scholar
Sperber, D. and Wilson, D. (1986/1995). Relevance: Communication and Cognition. Oxford: Blackwell.Google Scholar
Sperber, D., and Wilson, D. (2015). Beyond speaker’s meaning. Croatian Journal of Philosophy, 15(44), 117–149.Google Scholar
Stevick, E. W. (1982). Teaching and Learning Languages. Cambridge: Cambridge University Press.Google Scholar
Streeck, J., Goodwin, C., and Lebaron, C. (2011). Embodied Interaction: Language and Body in the Material World. Cambridge: Cambridge University Press.Google Scholar
Trager, G. L. (1958). Paralanguage: A first approximation. Studies in Linguistics, 13, 1–12.Google Scholar
Vandepitte, S. (1989). A pragmatic function of intonation. Lingua, 79, 265–297.CrossRefGoogle Scholar
Ward, G., and Hirschberg, J. (1988). Intonation and propositional attitude: The pragmatics of L*+H L H%. In Proceedings of the Fifth Eastern States Conference on Linguistics (pp. 512–522). Columbus: Ohio State University.Google Scholar
Wharton, T. (2003a). Interjections, language and the “showing”/”saying” continuum. Pragmatics and Cognition, 11(1), 39–91.CrossRefGoogle Scholar
Wharton, T. (2003b). Natural pragmatics and natural codes. Mind and Language, 18(5), 447–477.CrossRefGoogle Scholar
Wharton, T. (2009). Pragmatics and Non-verbal Communication. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Wichmann, A. (2002). Attitudinal intonation and the inferential process. In Bel, B. and Marlien, I. (eds.), Speech Prosody 2002: Proceedings of the First International Conference on Speech Prosody, Aix-en-Provence, 11–13 April 2002 (pp. 11–16). Aix-en-Provence: Laboratoire Parole et Langage. CrossRefGoogle Scholar
Wilson, D., and Carston, R. (2019). Pragmatics and the challenge of “non-propositional” effects. Journal of Pragmatics, 145, 31–38.CrossRefGoogle Scholar
Wilson, D., and Sperber, D. (1993). Linguistic form and relevance. Lingua, 90(1), 1–25.CrossRefGoogle Scholar
Wilson, D., and Sperber, D. (2004). Relevance theory. In Horn, L. R. and Ward, G. (eds.), The Handbook of Pragmatics (pp. 607–632). Oxford: Blackwell.Google Scholar
Wilson, D., and Wharton, T. (2006). Relevance and prosody. Journal of Pragmatics, 38, 1559–1579.CrossRefGoogle Scholar
Zubizarreta, M. L. (2016). Nuclear stress and information structure. In Féry, C. and Ishihara, S. (eds.), The Oxford Handbook of Information Structure (pp. 163–184). Oxford: Oxford University Press.Google Scholar

Save book to Kindle

To save this book to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×