Skip to main content
×
×
Home

OBSERVING AND PRODUCING PITCH GESTURES FACILITATES THE LEARNING OF MANDARIN CHINESE TONES AND WORDS

  • Florence Baills (a1), Nerea Suárez-González (a1), Santiago González-Fuente (a1) and Pilar Prieto (a2)
Abstract

This study investigates the perception and production of a specific type of metaphoric gesture that mimics melody in speech, also called pitch gesture, in the learning of L2 suprasegmental features. In a between-subjects design, a total of 106 participants with no previous knowledge of Chinese were asked to observe (Experiment 1) and produce (Experiment 2) pitch gestures during a short multimodal training session on Chinese tones and words. In both experiments they were tested on (a) tone identification and (b) word learning. Results showed the positive effect of a training session with pitch gesture observation compared to a training session without it (Experiment 1) and the benefits of producing gestures compared to only observing them and repeating the words aloud (Experiment 2). A comparison of the results of the two experiments revealed that there was no significant difference between the simple observation of pitch gestures and the production of speech accompanied by pitch gestures in facilitating lexical tone identification and word learning. Thus, both perception and production tasks with pitch gestures can be regarded as beneficial learning strategies for the initial stages of tones acquisition in the Chinese as a Second Language classroom.

Copyright
Corresponding author
*Correspondence concerning this article should be addressed to Florence Baills, Department of Translation and Language Sciences, Universitat Pompeu Fabra, Roc Boronat 138, 08018 Barcelona, Spain. E-mail: florence.baills@upf.edu
Footnotes
Hide All

Florence Baills and Santiago González-Fuente are predoctoral researchers at the Department of Translation and Language Sciences, Universitat Pompeu Fabra, Barcelona. Pilar Prieto is an ICREA researcher and a professor at the Department of Translation and Language Sciences, Universitat Pompeu Fabra.

This research has been funded by two research grants awarded by the Ministry of Science and Innovation (FFI2015-66533-P) and the Generalitat de Catalunya (2014 SGR-925), both to the Prosodic Studies Group. The first author has a predoctoral research grant awarded by the Department of Translation and Language Sciences, Universitat Pompeu Fabra. The third author also acknowledges a FPU 2012-05893 grant awarded by the Spanish Ministry of Science and Innovation.

We are deeply grateful to all the people who patiently participated in the creation of the material for this experiment: Rita Zaragoza Jové, Chenjie Yuan, Feifei Li, and Yuan Yuan. Many thanks also to Joan Carles Mora for his comments and suggestions on the first draft of this article and to Joan-Borràs Comes for his crucial help all along.

Footnotes
References
Hide All
Austin, E. E., & Sweller, N. (2014). Presentation and production: The role of gesture in spatial communication. Journal of Experimental Child Psychology, 122, 92103.
Barsalou, L. W. (2008). Grounded cognition. Annual Review of Psychology, 59, 617645.
Barsalou, L. W., Simmons, W. K., Barbey, A., & Wilson, C. D. (2003). Grounding conceptual knowledge in modality-specific systems. Trends in Cognitive Sciences, 7, 8491.
Bernardis, P., & Gentilucci, M. (2006). Speech and gesture share the same communication system. Neuropsychologia, 44, 178190.
Boersma, P., & Weenink, D. (2017). Praat: Doing phonetics by computer [Computer software]. Version 6.0.33. Retrieved from http://www.praat.org/.
Borghi, A. M., & Caruana, F. (2015). Embodiment theory. In Wright, J. D. (Ed.), International encyclopedia of the social and behavioral sciences (2nd ed., pp. 420426). Amsterdam, The Netherlands: Elsevier.
Bunting, M., Cowan, N., & Saults, J. S. (2006). How does running memory span work? Quarterly Journal of Experimental Psychology, 59, 16911700.
Burnham, D., Ciocca, V., & Stokes, S. (2001). Auditory-visual perception of lexical tone. In Dalsgaard, P., Lindberg, B., Benner, H., & Tan, Z.-H. (Eds.), Proceedings from Eurospeech 2001: 7th European Conference on Speech Communication and Technology (pp. 395398). Aalborg, Denmark: Center for Personkommunikation.
Casasanto, D., Phillips, W., & Boroditsky, L. (2003). Do we think about music in terms of space? Metaphoric representation of musical pitch. Proceedings of the Twenty-Fifth Annual Conference of the Cognitive Science, 10, 1323.
Cassidy, J. W. (1993). Effects of various sightsinging strategies on non-music majors’ pitch accuracy. Journal of Research in Music Education, 41, 293302.
Chao, Y. R. (1968). A grammar of spoken Chinese. Berkeley, CA: University of California Press.
Chen, C. M. (2013). Gestures as tone markers in multilingual communication. In Kecskes, I. (Ed.), Research in Chinese as a second language (pp. 143168). Boston, MA, and Berlin, Germany: De Gruyter Mouton.
Chen, T. H., & Massaro, D. W. (2008). Seeing pitch: Visual information for lexical tones of Mandarin-Chinese. The Journal of the Acoustical Society of America, 123, 23562366.
Cohen, R. L. (1981). On the generality of some memory laws. Scandinavian Journal of Psychology, 22, 267281.
Connell, L., Cai, Z. G., & Holler, J. (2013). Do you see what I’m singing? Visuospatial movement biases pitch perception. Brain and Cognition, 81, 124130.
Cook, S. W., Mitchell, Z., & Goldin-Meadow, S. (2008). Gesturing makes learning last. Cognition, 106, 10471058.
Dolscheid, S., Willems, R. M., Hagoort, P., & Casasanto, D. (2014). The relation of space and musical pitch in the brain. The 36th Annual Meeting of the Cognitive Science Society, 3, 421426.
Engelkamp, J., Zimmer, H. D., Mohr, G., & Sellen, O. (1994). Memory of self-performed tasks: Self-performing during recognition. Memory and Cognition, 22, 3439.
Francis, A. L., Ciocca, V., Ma, L., & Fenn, K. (2008). Perceptual learning of Cantonese lexical tones by tone and non-tone language speakers. Journal of Phonetics, 36, 268294.
Gluhareva, D., & Prieto, P. (2017). Training with rhythmic beat gestures favors L2 pronunciation in discourse-demanding situations. Language Teaching Research, 21, 609631.
Goldin-Meadow, S. (2003). Hearing gesture: How our hands help us think. Cambridge, MA: Harvard University Press.
Goldin-Meadow, S. (2011). Learning through gesture. Wiley Interdisciplinary Reviews: Cognitive Science, 2, 595607.
Goldin-Meadow, S. (2014). Widening the lens: What the manual modality reveals about language, learning and cognition. Philosophical Transactions of the Royal Society B: Biological Sciences, 369, 111.
Goldin-Meadow, S., Cook, S. W., & Mitchell, Z. A. (2009). Gesturing gives children new ideas about math. Psychological Science: A Journal of the American Psychological Society, 20, 267272.
Goldin-Meadow, S., Nusbaum, H., Kelly, S. D., & Wagner, S. (2001). Explaining math: Gesturing lightens the load. Psychological Science: A Journal of the American Psychological Society, 12, 516522.
González-Fuente, S., Escandell-Vidal, V., & Prieto, P. (2015). Gestural codas pave the way to the understanding of verbal irony. Journal of Pragmatics, 90, 2647.
Guasch, M., Boada, R., Ferré, P., & Sánchez-Casas, R. (2013). NIM: A web-based Swiss army knife to select stimuli for psycholinguistic studies. Behavior Research Methods, 45, 765771.
Gullberg, M. (1998). Gesture as a communication strategy in second language discourse: A study of learners of French and Swedish. Lund, Sweden: Lund University Press.
Gullberg, M. (2006). Some reasons for studying gesture and second language acquisition (Homage to Adam Kendon). IRAL—International Review of Applied Linguistics in Language Teaching, 44, 103124.
Gullberg, M. (2014). Gestures and second language acquisition. In Müller, C., Cienki, A., Fricke, E., Ladewig, S., McNeill, D., & Tessendorf, S. (Eds.), Body, language, communication: An international handbook on multimodality in human interaction (Vol. 2, pp. 18681875). Berlin, Germany, and Boston, MA: Mouton De Gruyter.
Gullberg, M., deBot, K., & Volterra, V. (2008). Gestures and some key issues in the study of language development. Gesture, 8, 149179.
Hao, Y. C. (2012). Second language acquisition of Mandarin Chinese tones by tonal and nontonal language speakers. Journal of Phonetics, 40, 269279.
Hannah, B., Wang, Y., Jongman, A., & Sereno, J. A. (2016). Cross-modal association between auditory and visual-spatial information in Mandarin tone perception. The Journal of the Acoustical Society of America, 140, 3225.
Hardison, D. M. (2003). Acquisition of second-language speech: Effects of visual cues, context, and talker variability. Applied Psycholinguistics, 24, 495522.
Hirata, Y., & Kelly, S. D. (2010). Effects of lips and hands on auditory learning of second-language speech sounds. Journal of Speech, Language, and Hearing Research, 53, 298310.
Hirata, Y., Kelly, S. D., Huang, J., & Manansala, M. (2014). Effects of hand gestures on auditory learning of second-language vowel length contrasts. Journal of Speech, Language, and Hearing Research, 57, 20902101.
IBM Corporation. (2015). IBM SPSS statistics for Windows, version 23.0 [Computer software]. Armonk, NY: IBM Corporation.
IBM Corporation. (2016). IBM SPSS statistics for Windows, version 24.0 [Computer software]. Armonk, NY: IBM Corporation.
Igualada, A., Esteve-Gibert, N., & Prieto, P. (2017). Beat gestures improve word recall in 3-to 5-year-old children. Journal of Experimental Child Psychology, 156, 99112.
Jia, L., & Wang, J. (2013a). On the effects of visual processing of tone production by English-speaking learners of Chinese. TCSOL Studies, 52, 63104.
Jia, L., & Wang, J. (2013b). The effects of visual processing on tone perception by native English-speaker learners of Chinese. Chinese Teaching in the World, 27, 548557.
Kelly, S. D., & Lee, A. L. (2012). When actions speak too much louder than words: Hand gestures disrupt word learning when phonetic demands are high. Language and Cognitive Processes, 27, 793807.
Kelly, S. D., Bailey, A., & Hirata, Y. (2017). Metaphoric gestures facilitate perception of intonation more than length in auditory judgments of nonnative phonemic contrasts. Collabra: Psychology, 3, 7.
Kelly, S. D., McDevitt, T., & Esch, M. (2009). Brief training with co-speech gesture lends a hand to word learning in a foreign language. Language and Cognitive Processes, 24, 313334.
Kelly, S. D., Hirata, Y., Manansala, M., & Huang, J. (2014). Exploring the role of hand gestures in learning novel phoneme contrasts and vocabulary in a second language. Frontiers in Psychology, 5, 111.
Kendon, A. (2004). Gesture: Visible action as utterance. New York, NY: New York University Press.
Kiefer, M., & Trumpp, N. M. (2012). Embodiment theory and education: The foundations of cognition in perception and action. Trends in Neuroscience and Education, 1, 1520.
Kiriloff, C. (1969). On the auditory perception of tones in Mandarin. Phonetica, 20, 6367.
Krauss, R. M., Chen, Y., & Chawla, P. (1996). Nonverbal behavior and nonverbal communication: What do conversational hand gestures tell us? Advances in Experimental Social Psychology, 28, 389450.
Krauss, R. M., Chen, Y., Gottesman, R. F., & McNeill, D. (2000). Lexical gestures and lexical access: A process model. In McNeill, D. (Ed.), Language and gesture (pp. 261283). New York: Cambridge University Press.
Kushch, O., & Prieto, P. (2016). The effects of pitch accentuation and beat gestures on information recall in contrastive discourse. In Barnes, J., Brugos, A., Shattuck-Hufnagel, S., & Veilleux, N. (Eds.), Proceedings of speech prosody 8 (pp. 922925). Boston: ICSA.
Kushch, O., Igualada, A., & Prieto, P. (2018). Prominence in speech and gesture favor second language novel word learning. Language, Cognition and Neuroscience. Available at https://doi.org/10.1080/23273798.2018.1435894.
Li, M., & DeKeyser, R. (2017). Perception practice, production practice, and musical ability in L2 Mandarin tone-word learning. Studies in Second Language Acquisition, 39, 593620.
Liu, Y., Wang, M., Perfetti, C. A., Brubaker, B., Wu, S., & MacWhinney, B. (2011). Learning a tonal language by attending to the tone: An in vivo experiment. Language Learning, 61, 11191141.
Macedonia, M., & Klimesch, W. (2014). Long-term effects of gestures on memory for foreign language words trained in the classroom. Mind, Brain, and Education, 8, 7488.
Macedonia, M., Müller, K., & Friederici, A. D. (2011). The impact of iconic gestures on foreign language word learning and its neural substrate. Human Brain Mapping, 32, 982998.
Madan, C. R., & Singhal, A. (2012). Using actions to enhance memory: Effects of enactment, gestures, and exercise on human memory. Frontiers in Psychology, 3, 20102013.
Masumoto, K., Yamaguchi, M., Sutani, K., Tsunetoa, S., Fujitaa, A., & Tonoike, M. (2006). Reactivation of physical motor information in the memory of action events. Brain Research, 1101, 102109.
McCafferty, S. G. (2002). Gesture and creating zones of proximal development for second language learning. Modern Language Journal, 86, 192203.
McNeill, D. (1992). Hand and mind: What gestures reveal about thought. Chicago, IL: University of Chicago Press.
McNeill, D. (2005). Gesture and thought. Chicago, IL: University of Chicago Press.
Morett, L. M., & Chang, L.-Y. (2015). Emphasising sound and meaning: Pitch gestures enhance Mandarin lexical tone acquisition. Language, Cognition and Neuroscience, 30, 347353.
Munhall, K. G., Jones, J. A., Callan, D. E., Kuratate, T., & Vatikiotis-Bateson, E. (2004). Visual prosody and speech intelligibility: Head movement improves auditory speech perception. Psychological Science, 15, 133137.
Paivio, A. (1990). Mental representations: A dual coding approach. New York, NY: Oxford University Press.
Post, L. S., Van Gog, T., Paas, F., & Zwaan, R. A. (2013). Effects of simultaneously observing and making gestures while studying grammar animations on cognitive load and learning. Computers in Human Behavior, 29, 14501455.
Prieto, P. (2004). Fonètica i fonologia. Els sons del català. Barcelona, Spain: EdiUOC.
Reid, A., Burnham, D., Kasisopa, B., Reilly, R., Attina, V., Rattanasone, N. X., et al. . (2015). Perceptual assimilation of lexical tone: The roles of language experience and visual information. Attention, Perception and Psychophysics, 77, 571–91.
Saltz, E., & Donnenwerth-Nolan, S. (1981). Does motoric imagery facilitate memory for sentences? A selective interference test. Journal of Verbal Learning and Verbal Behavior, 20, 322332.
Smith, D., & Burnham, D. (2012). Facilitation of Mandarin tone perception by visual speech in clear and degraded audio: Implications for cochlear implants. The Journal of the Acoustical Society of America, 131, 14801489.
So, W. C., Sim Chen-Hui, C., & Low Wei-Shan, J. (2012). Mnemonic effect of iconic gesture and beat gesture in adults and children: Is meaning in gesture important for memory recall? Language and Cognitive Processes, 27, 665681.
Stefan, K., Cohen, L. G., Duque, J., Mazzocchio, R., Celnik, P., Sawaki, L., et al. . (2005). Formation of a motor memory by action observation. The Journal of Neuroscience: The Official Journal of the Society for Neuroscience, 25, 93399346.
Szumilas, M. (2010). Explaining odd ratios. Journal of the Canadian Academy of Child and Adolescent Psychiatry, 19, 227229.
Tellier, M. (2008). The effect of gestures on second language memorisation by young children. Gesture, 8, 219235.
Thompson, L. A. (1995). Encoding and memory for visible speech and gestures: A comparison between young and older adults. Psychology and Aging, 10, 215228.
Wagner, S. M., Nusbaum, H., & Goldin-Meadow, S. (2004). Probing the mental representation of gesture: Is hand waving spatial? Journal of Memory and Language, 50, 395407.
Wang, Y., Behne, D. M., & Jiang, H. (2008). Linguistic experience and audio-visual perception of non-native fricatives. The Journal of the Acoustical Society of America, 124, 17161726.
Wang, Y., Jongman, A., & Sereno, J. A. (2003a). Acoustic and perceptual evaluation of Mandarin tone productions before and after perceptual training. The Journal of the Acoustical Society of America, 113, 10331043.
Wang, M., Perfetti, C. A., & Liu, Y. (2003b). Alphabetic readers quickly acquire orthographic structure in learning to read Chinese. Scientific Studies of Reading, 7, 183208.
Wang, Y., Spence, M. M., Jongman, A., & Sereno, J. A. (1999). Training American listeners to perceive Mandarin tones. The Journal of the Acoustical Society of America, 106, 36493658.
Wellsby, M., & Pexman, P. M. (2014). Developing embodied cognition: Insights from children’s concepts and language processing. Frontiers in Psychology, 5, 110.
Wong, P. C. M., & Perrachione, T. K. (2007). Learning pitch patterns in lexical identification by native English-speaking adults. Applied Psycholinguistics, 28, 565585.
Xu, Y. (1994). Production and perception of coarticulated tones. The Journal of the Acoustical Society of America, 95, 22402253.
Yuan, C., González-Fuente, S., Baills, F., & Prieto, P. (2018, in press). Observing pitch gestures favors the learning of Spanish intonation by Mandarin speakers. Studies in Second Language Acquisition, 128. Available at https://doi.org/10.1017/S0272263117000316.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Studies in Second Language Acquisition
  • ISSN: 0272-2631
  • EISSN: 1470-1545
  • URL: /core/journals/studies-in-second-language-acquisition
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed