Skip to main content Accessibility help
×
Home

Types of iconicity and combinatorial strategies distinguish semantic categories in silent gesture across cultures

  • GERARDO ORTEGA (a1) and ASLI ÖZYÜREK (a2)

Abstract

In this study we explore whether different types of iconic gestures (i.e., acting, drawing, representing) and their combinations are used systematically to distinguish between different semantic categories in production and comprehension. In Study 1, we elicited silent gestures from Mexican and Dutch participants to represent concepts from three semantic categories: actions, manipulable objects, and non-manipulable objects. Both groups favoured the acting strategy to represent actions and manipulable objects; while non-manipulable objects were represented through the drawing strategy. Actions elicited primarily single gestures whereas objects elicited combinations of different types of iconic gestures as well as pointing. In Study 2, a different group of participants were shown gestures from Study 1 and were asked to guess their meaning. Single-gesture depictions for actions were more accurately guessed than for objects. Objects represented through two-gesture combinations (e.g., acting + drawing) were more accurately guessed than objects represented with a single gesture. We suggest iconicity is exploited to make direct links with a referent, but when it lends itself to ambiguity, individuals resort to combinatorial structures to clarify the intended referent. Iconicity and the need to communicate a clear signal shape the structure of silent gestures and this in turn supports comprehension.

Copyright

Corresponding author

*Address for correspondence: e-mail: g.ortega@bham.ac.uk

References

Hide All
Barsalou, L. W. (1999). Perceptual symbol systems. Behavioral and Brain Sciences 22(4), 577609.
Barsalou, L. W. (2008). Grounding symbolic operations in the brain’s modal systems. In Semin, G. & Smith, E. (eds.), Embodied grounding: social, cognitive, affective and neuroscientific approaches (pp. 942). Cambridge: Cambridge University Press.
Bell, A. (1984). Language style as audience design. Language in Society 13, 240250.
Brentari, D., Renzo, A. Di, Keane, J. & Volterra, V. (2015). Cognitive, cultural, and linguistic sources of a handshape distinction expressing agentivity. Topics in Cognitive Science 7(1), 95123.
Christensen, P., Fusaroli, R. & Tylén, K. (2016). Environmental constraints shaping constituent order in emerging communication systems: structural iconicity, interactive alignment and conventionalization. Cognition 146, 6780.
Chu, M. & Kita, S. (2016). Co-thought and co-speech gestures are generated by the same action generation process. Journal of Experimental Psychology: Learning, Memory, and Cognition 42(2), 257270.
Clark, H. & Murphy, G. (1982). Audience design in meaning and reference. In Le Ny, J. F. & Kintsch, W. (eds.), Language and comprehension (pp. 287300). Amsterdam: North-Holland.
Cook, S. W. & Tanenhaus, M. K. (2009). Embodied communication: speakers’ gestures affect listeners’ actions. Cognition 113(1), 98104.
Dingemanse, M., Blasi, D. E., Lupyan, G., Christiansen, M. H. & Monaghan, P. (2015). Arbitrariness, iconicity and systematicity in language. Trends in Cognitive Sciences 19(10), 603615.
Fay, N., Arbib, M. & Garrod, S. (2013). How to bootstrap a human communication system. Cognitive Science 37(7), 13561367.
Galantucci, B. & Garrod, S. (2011). Experimental semiotics: a review. Frontiers in Human Neuroscience, 5. doi.org/10.3389/fnhum.2011.00011.
Gibson, E., Piantadosi, S. T., Brink, K., Bergen, L., Lim, E. & Saxe, R. (2013). A noisy-channel account of crosslinguistic word-order variation. Psychological Science 24(7), 10791088.
Goldin-Meadow, S. & Brentari, D. (2017). Gesture, sign and language: the coming of age of sign language and gesture studies. Behavioral and Brain Sciences 40, E46. doi.org/10.1017/S0140525X15001247.
Goldin-Meadow, S., McNeill, D. & Singleton, J. (1996). Silence is liberating: removing the handcuffs on grammatical expression in the manual modality. Psychological Review 103(1), 3455.
Goldin-Meadow, S., So, W. C., Özyürek, A. & Mylander, C. (2008). The natural order of events: how speakers of different languages represent events nonverbally. Proceedings of the National Academy of Sciences of the United States of America 105(27), 91639168.
Hall, M. L., Ahn, Y. D., Mayberry, R. I. & Ferreira, V. S. (2015). Production and comprehension show divergent constituent order preferences: evidence from elicited pantomime. Journal of Memory and Language 81, 1633.
Hall, M. L., Mayberry, R. I. & Ferreira, V. S. (2013). Cognitive constraints on constituent order: evidence from elicited pantomime. Cognition 129(1), 117.
Hassemer, J. & Winter, B. (2018). Decoding gestural iconicity. Cognitive Science 42(8), 30343049.
Haviland, J. (2013). The emerging of nouns in a first generation sign language: specification, iconicity, and syntax. Gesture 13(3), 309353.
Hockett, C. (1960). The origin of speech. Scientific American 203, 88111.
Hostetter, A. B. & Alibali, M. W. (2008). Visible embodiment: gestures as simulated action. Psychonomic Bulletin & Review 15(3), 495514.
Imai, M. & Kita, S. (2014). The sound symbolism bootstrapping hypothesis for language acquisition and language evolution. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 369(1651). doi.org/10.1098/rstb.2013.0298.
Johnston, T. (2001). Nouns and verbs in Australian sign language: An open and shut case? Journal of Deaf Studies and Deaf Education 6(4), 235257.
Kendon, A. (2004). Gesture: visible action as utterance. Cambridge: Cambridge University Press.
Kimmelman, V., Klezovich, A. & Moroz, G. (2018). Iconicity patterns in sign languages. Retrieved from <https://sl-iconicity.shinyapps.io/iconicity_patterns/>.
Kita, S. & Özyürek, A. (2003). What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language 48(1), 1632.
Klima, E. & Bellugi, U. (1979). The signs of language. Cambridge, MA: Harvard University Press.
Mandel, M. A. (1977). Iconic devices in American Sign Language. In Friedman, A. (ed.), On the other hand: new pespectives on American Sign Language (pp. 57107). New York: Academic Press.
Masson-Carro, I., Goudbeek, M. & Krahmer, E. (2016). Can you handle this? The impact of object affordances on how co-speech gestures are produced. Language, Cognition and Neuroscience 31(3), 430440.
Meir, I., Aronoff, M., Börstell, C., Hwang, S., Ilkbasaran, D., Kastner, I., … Sandler, W. (2017). The origin of grammatical word order: insights from novel communication systems and young sign languages. Cognition 158, 189207.
Meir, I., Aronoff, M., Sandler, W. & Padden, C. A. (2010). Sign languages and compounding. In Scalise, S. & Vogel, I. (eds.), Cross-disciplinary issues in compounding (pp. 301322). Philadelphia, PA: John Benjamins Publishing Company.
Meir, I., Sandler, W., Padden, C. & Aronoff, M. (2012). Emerging sign languages. In Marschark, M. & Spencer, P. (eds.), The Oxford handbook of deaf studies, language, and education (Vol. 2, pp. 122). Oxford: Oxford University Press. doi.org/10.1093/oxfordhb/9780195390032.013.0018
Micklos, A. (2016). Interaction for facilitating conventionalization: negotiating the silent gesture communication of noun–verb pairs. In Roberts, S., Cuskley, C., McCrohon, L., Barcelo-Coblijn, L., Feher, O. & Verhoef, T. (eds.), The evolution of language: Proceedings of the 11th International Conference (EVOLANG11). New Orleans: EVOLANG. Retrieved from <http://evolang.org/neworleans/papers/143.html>.
Misyak, J., Noguchi, T. & Chater, N. (2016). Instantaneous conventions: the emergence of flexible communicative signals. Psychological Science 27(12). doi.org/10.1177/0956797616661199.
Motamedi, Y., Schouwstra, M., Culbertson, J., Smith, K. & Kirby, S. (2018). Evolving artificial sign languages in the lab: from improvised gesture to systematic sign. doi.org/10.31234/osf.io/be7qy.
Müller, C. (2013). Gestural modes of representation as techniques of depcition. In Müller, C., Cienki, A., Ladewig, S., McNeill, D. & Bressem, J. (eds.), Body – language – communication: an international handbook on multimodality in human interaction (pp. 16871701). Berlin: De Gruyter Mouton.
Müller, C. (2016). From mimesis to meaning: a systematics of gestural mimesis for concrete and abstract referenital gestures. In Zlatev, J., Sonesson, G. & Konderak, P. (eds.), Meaning, mind and communication: explorations in cognitive semiotics. Frankfurt am Main: Peter Lang.
Nyst, V. (2016). The depiction of size and shape in gestures accompanying object descriptions in Anyi (Côte d’Ivoire) and in Dutch (the Netherlands). Gesture 15(2), 156191.
Ortega, G., Sümer, B. & Özyürek, A. (2014). Type of iconicity matters: bias for action-based signs in sign language acquisition. In Bello, P., Guarini, M., McShane, M. & Scassellatie, B. (eds.), 36th Annual Conference of the Cognitive Science Society (pp. 11141119). Austin, TX: Cognitive Science Society.
Ortega, G., Sümer, B. & Özyürek, A. (2017). Type of iconicity matters in the vocabulary development of signing children. Developmental Psychology 53(1), 8999.
Özçalişkan, Ş., Lucero, C. & Goldin-Meadow, S. (2016). Does language shape silent gesture? Cognition 148, 1018.
Padden, C., Hwang, S.-O., Lepic, R. & Seegers, S. (2015). Tools for language: patterned iconicity in sign language nouns and verbs. Topics in Cognitive Science 7(1), 8194.
Padden, C., Meir, I., Hwang, S.-O., Lepic, R., Seegers, S. & Sampson, T. (2013). Patterned iconicity in sign language lexicons. Gesture 13(3), 287305.
Perlman, M. & Lupyan, G. (2018). People can create iconic vocalizations to communicate various meanings to naïve listeners. Scientific Reports 8(1), 2634. doi.org/10.1038/s41598-018-20961-6.
Perniss, P., Thompson, R. L. & Vigliocco, G. (2010). Iconicity as a general property of language: evidence from spoken and signed languages. Frontiers in Psychology 1. doi.org/10.3389/fpsyg.2010.00227.
Perniss, P. & Vigliocco, G. (2014). The bridge of iconicity: from a world of experience to the experience of language. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 369(1651). doi.org/10.1098/rstb.2013.0300.
Pettenati, P., Sekine, K., Congestrì, E. & Volterra, V. (2012). A comparative study on representational gestures in Italian and Japanese children. Journal of Nonverbal Behavior 36(2), 149164.
Pettenati, P., Stefanini, S. & Volterra, V. (2010). Motoric characteristics of representational gestures produced by young children in a naming task. Journal of Child Language 37(4), 887911.
Ramachandran, S. & Hubbard, E. M. (2001). Synaesthesia: a window to perception, thought and language. Journal of Consciosness Studies 8(12), 334.
Roomer, E. K., Hoogerwerf, A. C. & Linn, D. E. (2011). Boston benoem taak 2011. Utrecht.
Sloetjes, H. & Wittenburg, P. (2018). ELAN (version 5.2). Nijmegen: Max Planck Institute for Psycholinguistics. Retrieved from <https://tla.mpi.nl/tools/tla-tools/elan/>.
Smith, K. & Kirby, S. (2008). Cultural evolution: implications for understanding the human language faculty and its evolution. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 363(1509). doi.org/10.1098/rstb.2008.0145.
Sulik, J. (2018). Cognitive mechanisms for inferring the meaning of novel signals during symbolisation. PLoS ONE 13(1). doi.org/10.1371/journal.pone.0189540.
Supalla, T. & Newport, E. L. (1986). How many sits in a chair? The derivations of nouns and verbs in American Sign Language. In Siple, P. (ed.), Understanding language through sign language research (pp. 91132). New York: Academic Press.
Tkachman, O. & Sandler, W. (2013). The noun–verb distinction in two young sign languages. Gesture 13(3), 253286.
Tolar, T. D., Lederberg, A. R., Gokhale, S. & Tomasello, M. (2008). The development of the ability to recognize the meaning of iconic signs. Journal of Deaf Studies and Deaf Education 13(2), 225240.
van Nispen, K., van de Sandt-Koenderman, M., Mol, L. & Krahmer, E. (2014). Pantomime strategies: on regularities in how people translate mental representations into the gesture modality. In Bello, C., Guarini, M., McShane, M. & Scassellatti, B. (eds.), Proceedings of the 36th Annual Conference of the Cognitive Science Society (CogSci 2014) (pp. 30203026). Austin, TX: Cognitive Science Society.
van Nispen, K., van de Sandt-Koenderman, W. M. E. & Krahmer, E. (2017). Production and comprehension of pantomimes used to depict objects. Frontiers in Psychology 8. doi.org/10.3389/FPSYG.2017.01095.
Verhoef, T., Kirby, S. & de Boer, B. (2016). Iconicity and the emergence of combinatorial structure in language. Cognitive Science 40(8), 19691994.
Vigliocco, G., Perniss, P. & Vinson, D. (2014). Language as a multimodal phenomenon: implications for language learning, processing and evolution. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 369(1651). doi.org/10.1098/rstb.2013.0292.

Keywords

Types of iconicity and combinatorial strategies distinguish semantic categories in silent gesture across cultures

  • GERARDO ORTEGA (a1) and ASLI ÖZYÜREK (a2)

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed