Skip to main content Accessibility help
×
×
Home

Taking simulation semantics out of the laboratory: towards an interactive and multimodal reappraisal of embodied language comprehension

  • KASPER KOK (a1) and ALAN CIENKI (a2)

Abstract

Recent embodied theories of meaning known as ‘simulation semantics’ posit that language comprehension engages, or even amounts to, mental simulation. What is meant here by ‘language comprehension’, however, deviates from the perspectives on interpersonal communication adhered to by researchers in social psychology and interactional linguistics. In this paper, we outline four alternative perspectives on comprehension in spoken interaction, each of which highlights factors that have remained largely outside the current purview of simulation theories. These include perspectives on language comprehension in terms of (i) striving for inter-subjective conformity; (ii) recognition of communicative intentions; (iii) prediction and anticipation in a dynamic environment; and (iv) integration of multimodal cues. By contrasting these views with simulation theories of comprehension, we outline a number of fundamental differences in terms of the kind of process comprehension is assumed to be (passive and event-like versus active and continuous), as well as the kind of stimulus that language is assumed to be (comprising unimodal units versus being multimodal and distributed across conversational turns). Finally, we discuss potential points of connection between simulation semantics and research on spoken interaction, and touch on some methodological implications of an interactive and multimodal reappraisal of simulation semantics.

Copyright

References

Hide All
Altenberg, B. (1987). Prosodic patterns in spoken English: studies in the correlation between prosody and grammar for text-to-speech conversion. Lund: Lund University Press.
Austin, J. (1962). How to do things with words. London: Oxford University Press.
Aziz-Zadeh, L., Wilson, S., Rizzolatti, G., & Iacoboni, M. (2006). Congruent embodied representations for visually presented actions and linguistic phrases describing actions. Current Biology, 16(18), 18181823.
Bakhtin, M. (1981). The dialogic imagination. Austin: University of Texas.
Barsalou, L. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22(4), 577609.
Barsalou, L. (2003). Situated simulation in the human conceptual system. Language and Cognitive Processes, 18(5/6), 513562.
Barsalou, L. (2005). Situated conceptualization. In Cohen, H. & Lefebvre, C. (Eds.), Handbook of categorization in cognitive science (pp. 619650). St Louis: Elsevier.
Barsalou, L. (2009). Simulation, situated conceptualization, and prediction. Philosophical Transactions of the Royal Society B: Biological Sciences, 364, 12811289.
Barwise, J., & Perry, J. (1981). Situations and attitudes. Journal of Philosophy, 78(11), 668691.
Barwise, J., & Perry, J. (1983). Situations and attitudes. Cambridge, MA: MIT Press.
Beattie, G., & Shovelton, H. (1999). Mapping the range of information contained in the iconic hand gestures that accompany spontaneous speech. Journal of Language and Social Psychology, 18(4), 438462.
Bergen, B. K. (2012). Louder than words: the new science of how the mind makes meaning. New York: Basic Books.
Bergen, B. K., & Chang, . (2005). Embodied Construction Grammar in simulation-based language understanding. In Östman, J-O & Fried, M (Eds.), Construction Grammars: cognitive grounding and theoretical extensions (pp. 147190). Amsterdam & Philadelphia: John Benjamins.
Bergen, B. K., Lindsay, S., Matlock, T., & Narayanan, S. (2007). Spatial and linguistic aspects of visual imagery in sentence comprehension. Cognitive Science, 31(5), 733764.
Bergen, B. K., & Wheeler, K. (2005). Sentence understanding engages motor processes. Paper presented at the Proceedings of the 27th Annual Conference of the Cognitive Science Society, Mahwah, NJ.
Bernardis, P., & Gentilucci, M. (2006). Speech and gesture share the same communication system. Neuropsychologia, 44(2), 178190.
Bolinger, D. (1986). Intonation and its parts: melody in spoken English. Stanford: Stanford University Press.
Borghi, A., Glenberg, A., & Kaschak, M. P. (2004). Putting words in perspective. Memory & Cognition, 32(6), 863873.
Borghi, A., & Riggio, L. (2009). Sentence comprehension and simulation of object temporary, canonical and stable affordances. Brain Research, 1253, 117128.
Brazil, D. (1997). The communicative value of intonation in English. Cambridge: Cambridge University Press.
Brunyé, T., Ditman, T., Mahoney, C., Augustyn, J., & Taylor, H. (2009). When you and I share perspectives pronouns modulate perspective taking during narrative comprehension. Psychological Science, 20(1), 2732.
Carruthers, P., & Smith, P. (1996). Theories of theories of mind. Cambridge: Cambridge University Press.
Chafe, W. (1994). Discourse, consciousness, and time: the flow and displacement of conscious experience in speaking and writing. Chicago: University of Chicago Press.
Chafe, W., & Tannen, D. (1987). The relation between written and spoken language. Annual Review of Anthropology, 16, 383407.
Chao, L., Haxby, J., & Martin, A. (1999). Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects. Nature Neuroscience, 2(10), 913919.
Chwilla, D., Kolk, H., & Vissers, C. (2007). Immediate integration of novel meanings: N400 support for an embodied view of language comprehension. Brain Research, 1183, 109123.
Cienki, A. (2012). Usage events of spoken language and the symbolic units we (may) abstract from them. In Badio, J. & Kosecki, K. (Eds.), Cognitive processes in language (pp. 149158). Bern: Peter Lang.
Clark, H. (1973). Space, time, semantics, and the child. New York: Academic Press.
Clark, H. (1996). Using language. Cambridge: Cambridge University Press.
Clark, H. (1997). Dogmas of understanding. Discourse Processes, 23(3), 567598.
Clark, H. (1999). On the origins of conversation. Verbum, 2, 147161.
Clark, H., & Brennan, S. (1991). Grounding in communication. In Resnick, L. B. Levine, J. M., & Teasley, S. D. (Eds.), Perspectives on socially shared cognition (pp. 127149). Washington, DC: APA Books.
Clark, H., & Krych, M. (2004). Speaking while monitoring addressees for understanding. Journal of Memory and Language, 50(1), 6281.
Clark, H., & Marshall, C. (1981). Definite reference and mutual knowledge. In Joshi, A. K. Webber, B. L., & Sag, I. A. (Eds.), Elements of discourse understanding (pp. 1063). Cambridge: Cambridge University Press.
Connell, L. (2007). Representing object colour in language comprehension. Cognition, 102(3), 476485.
Croft, W. (2009). Towards a social cognitive linguistics. In Evans, V. & Pourcel, S. (Eds.), New directions in cognitive linguistics (pp. 395420). Amsterdam: John Benjamins.
Dale, R., Fusaroli, R., Duran, N., & Richardson, D. (2013). The self-organization of human interaction. In Ross, H. (Ed.), Psychology of Learning and Motivation, Vol. 59 (pp. 4395). Online: <http://www.sciencedirect.com/science/article/pii/B9780124071872000022>.
Desai, R., Binder, J., Conant, L., & Seidenberg, M. (2010). Activation of sensory–motor areas in sentence comprehension. Cerebral Cortex, 20(2), 468478.
Dove, G. (2010). On the need for embodied and dis-embodied cognition. Frontiers in Psychology, 1(242).
Egorova, N., Shtyrov, Y., & Pulvermüller, F. (2013). Early and parallel processing of pragmatic and semantic information in speech acts: neurophysiological evidence. Frontiers in Human Neuroscience, 7(86).
Evans, V. (2009). How words mean. Oxford: Oxford University Press.
Gallese, V. (2007). Before and below ‘theory of mind’: embodied simulation and the neural correlates of social cognition. Philosophical Transactions of the Royal Society B: Biological Sciences, 362(1480), 659669.
Gallese, V., & Goldman, A. (1998). Mirror neurons and the simulation theory of mind-reading. Trends in Cognitive Sciences, 2(12), 493501.
Garrod, S., & Anderson, A. (1987). Saying what you mean in dialogue: a study in conceptual and semantic co-ordination. Cognition, 27(2), 181218.
Garrod, S., & Pickering, M. (2004). Why is conversation so easy? Trends in Cognitive Sciences, 8(1), 811.
Geeraerts, D. (2010). Recontextualizing grammar: underlying trends in thirty years of Cognitive Linguistics. In Tabakowska, E., Choinski, M., & Wiraszka, L. (Eds.), Cognitive linguistics in action: from theory to application and back (pp. 71−102). Berlin: Mouton de Gruyter.
Gibbs, R. (2006). Metaphor interpretation as embodied simulation. Mind & Language, 21(3), 434458.
Glenberg, A., & Gallese, V. (2012). Action-based language: a theory of language acquisition, comprehension, and production. Cortex, 48(7), 905922.
Glenberg, A., & Kaschak, M. (2002). Grounding language in action. Psychonomic Bulletin & Review, 9(3), 558565.
Glenberg, A., & Robertson, D. (1999). Indexical understanding of instructions. Discourse Processes, 28(1), 126.
Goldman, A. (1992). In defense of the simulation theory. Mind & language, 7(1/2), 104119.
Gopnik, A. (1995). How to understand beliefs. Behavioral and Brain Sciences, 18(02), 398400.
Graesser, A., Millis, K., & Zwaan, R. (1997). Discourse comprehension. Annual Review of Psychology, 48(1), 163189.
Graesser, A., Singer, M., & Trabasso, T. (1994). Constructing inferences during narrative text comprehension. Psychological Review, 101(3), 371395.
Hagoort, P., Hald, L., Bastiaansen, M., & Petersson, K. (2004). Integration of word meaning and world knowledge in language comprehension. Science, 304, 438441.
Hauk, O., Johnsrude, I., & Pulvermüller, F. (2004). Somatotopic representation of action words in human motor and premotor cortex. Neuron, 41(2), 301307.
Hauk, O., & Pulvermüller, F. (2004). Neurophysiological distinction of action words in the fronto‐central cortex. Human Brain Mapping, 21(3), 191201.
Havas, D., Glenberg, A., & Rinck, M. (2007). Emotion simulation during language comprehension. Psychonomic Bulletin & Review, 14(3), 436441.
Holler, J., & Beattie, G. (2003). Pragmatic aspects of representational gestures: Do speakers use them to clarify verbal ambiguity for the listener? Gesture, 3(2), 127154.
Hostetter, A. (2011). When do gestures communicate? A meta-analysis. Psychological Bulletin, 137(2), 297315.
Hostetter, A., & Alibali, M. (2008). Visible embodiment: gestures as simulated action. Psychonomic Bulletin & Review, 15(3), 495514.
Kaschak, M., & Glenberg, A. (2000). Constructing meaning: the role of affordances and grammatical constructions in sentence comprehension. Journal of Memory and Language, 43(3), 508529.
Kaschak, M., Madden, C., Therriault, D., Yaxley, R., Aveyard, M., Blanchard, A., & Zwaan, R. (2005). Perception of motion affects language processing. Cognition, 94(3), B79B89.
Kelly, S., Özyürek, A., & Maris, E. (2010). Two sides of the same coin: speech and gesture mutually interact to enhance comprehension. Psychological Science, 21(2), 260267.
Kendon, A. (1994). Do gestures communicate? A review. Research on Language and Social Interaction, 27(3), 175200.
Kendon, A. (2004). Gesture: visible action as utterance. Cambridge: Cambridge University Press.
Klatzky, R., Pellegrino, J., McCloskey, B., & Doherty, S. (1989). Can you squeeze a tomato? The role of motor representations in semantic sensibility judgments. Journal of Memory and Language, 28(1), 5677.
Krauss, R., & Fussell, S. (1988). Other-relatedness in language processing: discussion and comments. Journal of Language and Social Psychology, 7(3/4), 263279.
Krauss, R., & Fussell, S. (1996). Social psychological models of interpersonal communication. In Higgins, E. T. & Kruglanski, A. (Eds), Social psychology: a handbook of basic principles (pp. 655701). New York: Guilford Press.
Langacker, R. (1987). Foundations of Cognitive Grammar: theoretical prerequisites. Volume 1. Stanford: Stanford University Press.
Langacker, R. (1997). The contextual basis of cognitive semantics. In Nuyts, J. & Pedersen, E. (Eds.), Language and conceptualization (pp. 229252). Cambridge: Cambridge University Press.
Langacker, R. (2008). Cognitive Grammar: a basic introduction. Oxford: Oxford University Press.
Linell, P. (1982). The written language bias in linguistics. Linkoping: Deptartment of Communication Studies, University of Linkoping. Republished in revised form in 2005 by Routledge (London).
Linell, P. (1998). Approaching dialogue: talk, interaction and contexts in dialogical perspectives. Amsterdam: John Benjamins
Linell, P. (2007). Dialogicality in languages, minds and brains: Is there a convergence between dialogism and neuro-biology? Language Sciences, 29(5), 605620.
Mahon, B., & Caramazza, A. (2008). A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content. Journal of Physiology, 102(1/3), 5970.
Marghetis, T., & Bergen, B. (in press). Embodied meaning, inside and out: the coupling of gesture and mental simulation. In Müller, C., Cienki, A., Fricke, E., Ladewig, S., McNeill, D., & Bressem, J. (Eds.), Body – language – communication: an international handbook on multimodality in human interaction, Vol. 2. Berlin: Mouton De Gruyter.
Masson, M., Bub, D., & Warren, C. (2008). Kicking calculators: contribution of embodied representations to sentence comprehension. Journal of Memory and Language, 59(3), 256265.
Matlock, T. (2004). Fictive motion as cognitive simulation. Memory & Cognition, 32(8), 13891400.
McNeill, D. (1992). Hand and mind: what gestures reveal about thought. Chicago: University of Chicago Press.
Menenti, L., Pickering, M., & Garrod, S. (2012). Toward a neural basis of interactive alignment in conversation. Frontiers in Human Neuroscience, 6(185), 19.
Molnar-Szakacs, I., Wu, A., Robles, F., & Iacoboni, M. (2007). Do you see what I mean? Corticospinal excitability during observation of culture-specific gestures. PLoS One, 2(7), e626.
Montgomery, K., Isenberg, N., & Haxby, J. (2007). Communicative hand gestures and object-directed hand movements activated the mirror neuron system. Social Cognitive and Affective Neuroscience, 2(2), 114122.
Ochs, E., Schegloff, E., & Thompson, S. (1996). Interaction and grammar. Cambridge: Cambridge University Press.
Özyürek, A., Willems, R., Kita, S., & Hagoort, P. (2007). On-line integration of semantic information from speech and gesture: insights from event-related brain potentials. Journal of Cognitive Neuroscience, 19(4), 605616.
Parker, I. (1992). Discourse dynamics: critical analysis for social and individual psychology. London: Routledge.
Pickering, M., & Garrod, S. (2004). Toward a mechanistic psychology of dialogue. Behavioral and Brain Sciences, 27(02), 169190.
Pickering, M., & Garrod, S. (2009). Prediction and embodiment in dialogue. European Journal of Social Psychology, 39(7), 11621168.
Pulvermüller, F., Mohr, B., & Schleichert, H. (1999). Semantic or lexico-syntactic factors: What determines word-class specific activity in the human brain? Neuroscience Letters, 275(2), 8184.
Rączaszek-Leonardi, J. (2009). Symbols as constraints: the structuring role of dynamics and self-organization in natural language. Pragmatics & Cognition, 17(3), 657676.
Richardson, D., Spivey, M., Barsalou, L., & McRae, K. (2003). Spatial representations activated during real-time comprehension of verbs. Cognitive Science, 27(5), 767780.
Robinson, E. (2000). The cognitive foundations of pragmatic principles: implications for theories of linguistic and cognitive representation. In Nuyts, J. & Pedersen, E. (Eds.), Language and conceptualization (pp. 253271). Cambridge: Cambridge University Press.
Sato, M., Mengarelli, M., Riggio, L., Gallese, V., & Buccino, G. (2008). Task related modulation of the motor system during language processing. Brain and Language, 105(2), 8390.
Shannon, C., & Weaver, W. (1948). The mathematical theory of communication. Urbana: University of Illinois Press.
Skipper, J., Goldin-Meadow, S., Nusbaum, H., & Small, S. (2007). Speech-associated gestures, Broca’s area, and the human mirror system. Brain and Language, 101(3), 260277.
Speer, N., Zacks, J., & Reynolds, J. (2007). Human brain activity time-locked to narrative event boundaries. Psychological Science, 18(5), 449455.
Sperber, D., & Wilson, D. (1986). Relevance: communication and cognition. Cambridge, MA: Harvard University Press.
Stanfield, R., & Zwaan, R. (2001). The effect of implied orientation derived from verbal context on picture recognition. Psychological Science, 12(2), 153156.
Stolorow, R., & Atwood, G. (1992). Contexts of being: the intersubjective foundations of psychological life. Hillsdale: Analytic Press.
Swerts, M., & Geluykens, R. (1994). Prosody as a marker of information flow in spoken discourse. Language and Speech, 37(1), 2143.
Tettamanti, M., Buccino, G., Saccuman, M., Gallese, V., Danna, M., Scifo, P., Fazio, F., Rizzolatti, G., Cappa, S. F., & Perani, D. (2005). Listening to action-related sentences activates fronto-parietal motor circuits. Journal of Cognitive Neuroscience, 17(2), 273281.
Thibault, P. (2005). The interpersonal gateway to the meaning of mind: unifying the inter- and intraorganism perspective on language. In Hasan, R., Matthiessen, C., & Webster, J. (Eds.), Continuing discourse on language: a functional perspective (pp. 117156). London: Equinox.
Van Berkum, J., Van Den Brink, D., Tesink, C., Kos, M., & Hagoort, P. (2008). The neural integration of speaker and message. Journal of Cognitive Neuroscience, 20(4), 580591.
Van Dam, W., Rueschemeyer, S., Lindemann, O., & Bekkering, H. (2010). Context effects in embodied lexical-semantic processing. Frontiers in Psychology, 1(150).
Van Dijk, T., & Kintsch, W. (1983). Strategies of discourse comprehension. New York: Academic Press.
Van Elk, M., Slors, M., & Bekkering, H. (2010). Embodied language comprehension requires an enactivist paradigm of cognition. Frontiers in Psychology, 1(234).
Varela, F., Thompson, E., & Rosch, E. (1991). The embodied mind: cognitive science and human experience. Cambridge, MA: MIT press.
Vigliocco, G., Warren, J., Siri, S., Arciuli, J., Scott, S., & Wise, R. (2006). The role of semantics and grammatical class in the neural representation of words. Cerebral Cortex, 16(12), 17901796.
Wallentin, M., Nielsen, A. H., Vuust, P., Dohn, A., Roepstorff, A., & Lund, T. E. (2011). BOLD response to motion verbs in left posterior middle temporal gyrus during story comprehension. Brain and Language, 119(3), 221225.
Willems, R., Benn, Y., Hagoort, P., Toni, I., & Varley, R. (2011). Communicating without a functioning language system: implications for the role of language in mentalizing. Neuropsychologia, 49(11), 31303135.
Willems, R., de Boer, M., de Ruiter, J., Noordzij, M., Hagoort, P., & Toni, I. (2010). A dissociation between linguistic and communicative abilities in the human brain. Psychological Science, 21(1), 814.
Willems, R., & Francken, J. (2012). Embodied cognition: taking the next step. Frontiers in Psychology, 3(582).
Willems, R., & Hagoort, P. (2007). Neural evidence for the interplay between language, gesture, and action: a review. Brain and Language, 101(3), 278289.
Willems, R., & Varley, R. (2010). Neural insights into the relation between language and communication. Frontiers in Human Neuroscience, 4(203).
Wilson, A., & Golonka, S. (2013). Embodied cognition is not what you think it is. Frontiers in Psychology, 4(58).
Winter, B., & Bergen, B. (2012). Language comprehenders represent object distance both visually and auditorily. Language and Cognition, 4, 116.
Yaxley, R., & Zwaan, R. (2007). Simulating visibility during language comprehension. Cognition, 105(1), 229236.
Zwaan, R. (2003). The immersed experiencer: toward an embodied theory of language comprehension. Psychology of Learning and Motivation, 44, 3562.
Zwaan, R. (2009). Mental simulation in language comprehension and social cognition. European Journal of Social Psychology, 39(7), 11421150.
Zwaan, R. (2014). Embodiment and language comprehension: reframing the discussion. Trends in Cognitive Sciences 18(5), 229234.
Zwaan, R., Madden, C., Yaxley, R., & Aveyard, M. (2004). Moving words: dynamic representations in language comprehension. Cognitive Science, 28(4), 611619.
Zwaan, R., Magliano, J., & Graesser, A. (1995). Dimensions of situation model construction in narrative comprehension. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21(2), 386397.
Zwaan, R., & Radvansky, G. (1998). Situation models in language comprehension and memory. Psychological Bulletin, 123(2), 162185.
Zwaan, R., Stanfield, R., & Yaxley, R. (2002). Language comprehenders mentally represent the shapes of objects. Psychological Science, 13(2), 168171.
Zwaan, R., & Taylor, L. (2006). Seeing, acting, understanding: motor resonance in language comprehension. Journal of Experimental Psychology: General, 135(1), 111.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Language and Cognition
  • ISSN: 1866-9808
  • EISSN: 1866-9859
  • URL: /core/journals/language-and-cognition
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×

Keywords

Metrics

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed