Hostname: page-component-76d6cb85b7-xh428 Total loading time: 0 Render date: 2026-07-12T09:37:18.189Z Has data issue: false hasContentIssue false
  • English
  • Français

Multimodal input in second-language speech processing

Published online by Cambridge University Press:  17 November 2020

Debra M. Hardison
Debra M. Hardison*
Affiliation:
Michigan State University, East Lansing, USA
*
*Corresponding author. Email: hardiso2@msu.edu
Get access Rights & Permissions [Opens in a new window]

Extract

This timeline provides an update on research since 2009 involving auditory-visual (AV) input in spoken language processing (see Hardison, 2010 for an earlier timeline on this topic). A brief background is presented here as a foundation for the more recent studies of speech as a multimodal phenomenon (e.g., Rosenblum, 2005).

Information

Type
Research Timeline
Information
Language Teaching , Volume 54 , Issue 2 , April 2021 , pp. 206 - 220
Check for updates
Copyright
Copyright © The Author(s), 2020. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Benoît, C., Mohamadi, T., & Kandel, S. (1994). Effects of phonetic context on audio-visual intelligibility of French. Journal of Speech and Hearing Research, 37(5), 1195–1203.CrossRefGoogle ScholarPubMed
Bergeson, T. R., Pisoni, D. B., & Davis, R. A. O. (2003). A longitudinal study of audiovisual speech perception by children with hearing loss who have cochlear implants. Volta Review, 103(4), 347–370.Google ScholarPubMed
de Bot, K. (1983). Visual feedback of intonation I: Effectiveness and induced practice behaviour. Language and Speech, 26(4), 331–350.CrossRefGoogle Scholar
Dimitrova, D., Chu, M., Wang, L., Özyürek, A., & Hagoort, P. (2016). Beat that word: How listeners integrate beat gesture and focus in multimodal speech discourse. Journal of Cognitive Neuroscience, 28(9), 1255–1269.CrossRefGoogle ScholarPubMed
Gick, B., Bernhardt, B. M., Bacsfalvi, P., & Wilson, I. (2008). Ultrasound imaging applications in second language acquisition. In Hansen, J. & Zampini, M. (Eds.), Phonology and Second Language Acquisition (pp. 309–322). Amsterdam, Netherlands: John Benjamins.CrossRefGoogle Scholar
Gullberg, M. (2006). Some reasons for studying gesture and second language acquisition (Hommage à Adam Kendon). IRAL, 44(2), 103–124.CrossRefGoogle Scholar
Hardison, D. M. (1996). Bimodal speech perception by native and non-native speakers of English: Factors influencing the McGurk effect. Language Learning, 49(1), 213–283.CrossRefGoogle Scholar
Hardison, D. M. (2003). Acquisition of second-language speech: Effects of visual cues, context, and talker variability. Applied Psycholinguistics, 24(4), 495–522.CrossRefGoogle Scholar
Hardison, D. M. (2004). Generalization of computer-assisted prosody training: Quantitative and qualitative findings. Language Learning & Technology, 8(1), 34–52.Google Scholar
Hardison, D. M. (2005). Second-language spoken word identification: Effects of perception training, visual cues, and phonetic environment. Applied Psycholinguistics, 26(4), 579–596.CrossRefGoogle Scholar
Hardison, D. M. (2010). Visual and auditory input in second-language speech processing. Language Teaching, 43(1), 84–95.CrossRefGoogle Scholar
Hardison, D. M. (2019). Technology-based communication success for second-language learners. In Peters, M. A. & Heraud, R. (Eds.), Encyclopedia of Educational Innovation. Singapore: Springer: doi.org/10.1007/978-981-13-2262-4_85-2Google Scholar
Hickok, G. B., Buchsbau, B., Humphries, C., & Muftuler, T. (2003). Auditory-motor interaction revealed by fMRI: Speech, music, and working memory in area Spt. Journal of Cognitive Neuroscience, 15(5), 673–682.CrossRefGoogle ScholarPubMed
Hincks, R. (2015). Technology and learning pronunciation. In Reed, M. & Levis, J. M. (Eds.), The handbook of English pronunciation (pp. 505–519). Hoboken, NJ: Wiley.CrossRefGoogle Scholar
Leather, J. (1990). Perceptual and productive learning of Chinese lexical tone by Dutch and English speakers. In Leather, J. & James, A. (Eds.), New Sounds 90 (pp. 72–97). Amsterdam, Netherlands: University of Amsterdam.Google Scholar
MacLeod, A., & Summerfield, Q. (1990). A procedure for measuring auditory and audio-visual speech-reception thresholds for sentences in noise: Rationale, evaluation, and recommendations for use. British Journal of Audiology, 24(1), 29–43.CrossRefGoogle ScholarPubMed
Massaro, D. W., & Light, J. (2003). Read my tongue movements: Bimodal learning to perceive and produce non-native speech /r/ and /l/. In EUROSPEECH-2003, pp. 2249–2252.Google Scholar
McGrath, M., & Summerfield, Q. (1985). Intermodal timing relations and audio-visual speech recognition by normal-hearing adults. Journal of the Acoustical Society of America, 77(2), 678–685.CrossRefGoogle ScholarPubMed
McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746–748.CrossRefGoogle ScholarPubMed
Meltzoff, A., & Moore, M. K. (1993). Why faces are special to infants – on connecting the attraction of faces and infants’ ability for imitation and cross-modal processing. In de Boysson-Bardies, B., de Schonen, S., Jusczyk, P., McNeilage, P., & Morton, J. (Eds.), Developmental neurocognition: Speech and face processing in the first year of life (pp. 211–225). Dordrecht, Netherlands: Kluwer Academic.CrossRefGoogle Scholar
Miller, G. A., & Nicely, P. E. (1955). An analysis of perceptual confusions among some English consonants. Journal of the Acoustical Society of America, 27, 338–352.CrossRefGoogle Scholar
Motohashi-Saigo, M., & Hardison, D. M. (2009). Acquisition of L2 Japanese geminates: Training with waveform displays. Language Learning & Technology, 13(2), 29–47.Google Scholar
Munhall, K. G., Jones, J. A., Callan, D. E., Kuratate, T., & Vatikiotis-Bateson, E. (2004). Visual prosody and speech intelligibility: Head movement improves auditory speech perception. Psychological Science, 15(2), 133–137.CrossRefGoogle ScholarPubMed
Munhall, K. G., & Tohkura, Y. (1998). Audiovisual gating and the time course of speech perception. Journal of the Acoustical Society of America, 104, 530–539.CrossRefGoogle ScholarPubMed
O'Brien, M. G., Derwing, T. M., Cucchiarini, C., Hardison, D. M., Mixdorff, H., Thomson, R. L., … Levis, G. M. (2018). Directions for the future of technology in pronunciation research and teaching. Journal of Second Language Pronunciation, 4(2), 182–206.CrossRefGoogle Scholar
Reisberg, D., McLean, J., & Goldfield, A. (1987). Easy to hear but hard to understand: A lip-reading advantage with intact auditory stimuli. In Dodd, B. & Campbell, R. (Eds.), Hearing by eye: The psychology of lip-reading (pp. 97–113). London, UK: Erlbaum.Google Scholar
Rosenblum, L. (2005). The primacy of multimodal speech perception. In Pisoni, D. & Remez, R. (Eds.), Handbook of Speech Perception (pp. 51–78). Malden, MA: Blackwell.CrossRefGoogle Scholar
Sams, M., Aulanko, R., Hämäläinen, M., Hari, R., Lounasmaa, O. V., Lu, S.-T., & Simola, J. (1991). Seeing speech: Visual information from lip movements modifies activity in the human auditory cortex. Neuroscience Letters, 127(1), 141–145.CrossRefGoogle ScholarPubMed
Shams, L. A., & Seitz, A. R. (2008). Benefits of multisensory learning. Trends in Cognitive Sciences, 12(11), 411–417.CrossRefGoogle ScholarPubMed
Sueyoshi, A., & Hardison, D. M. (2005). The role of gestures and facial cues in second-language listening comprehension. Language Learning, 55(4), 661–699.CrossRefGoogle Scholar
Sumby, W. H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212–215.CrossRefGoogle Scholar
Walden, B. E., Prosek, R. A., Montgomery, A. A., Scherr, C. K., & Jones, C. J. (1977). Effects of training on the visual recognition of consonants. Journal of Speech and Hearing Research, 20(1), 130–145.CrossRefGoogle ScholarPubMed