Hostname: page-component-848d4c4894-75dct Total loading time: 0 Render date: 2024-06-08T12:36:04.991Z Has data issue: false hasContentIssue false
  • English
  • Français

The impact of audio on the reading of intralingual versus interlingual subtitles: Evidence from eye movements

Published online by Cambridge University Press:  03 December 2021

Sixin Liao Open the ORCID record for Sixin Liao [Opens in a new window] ,
Lili Yu ,
Jan-Louis Kruger Open the ORCID record for Jan-Louis Kruger [Opens in a new window]  and
Erik D. Reichle
Sixin Liao*
Affiliation:
Department of Linguistics, Macquarie University, Sydney, NSW, Australia Macquarie University Centre for Reading, Sydney, NSW, Australia
Lili Yu
Affiliation:
Department of Psychology, Macquarie University, Sydney, NSW, Australia Macquarie University Centre for Reading, Sydney, NSW, Australia
Jan-Louis Kruger
Affiliation:
Department of Linguistics, Macquarie University, Sydney, NSW, Australia Macquarie University Centre for Reading, Sydney, NSW, Australia
Erik D. Reichle
Affiliation:
Department of Psychology, Macquarie University, Sydney, NSW, Australia Macquarie University Centre for Reading, Sydney, NSW, Australia
*
*Corresponding author. Email: sixin.liao@students.mq.edu.au
Get access Rights & Permissions [Opens in a new window]

Abstract

This study investigated how semantically relevant auditory information might affect the reading of subtitles, and if such effects might be modulated by the concurrent video content. Thirty-four native Chinese speakers with English as their second language watched video with English subtitles in six conditions defined by manipulating the nature of the audio (Chinese/L1 audio vs. English/L2 audio vs. no audio) and the presence versus absence of video content. Global eye-movement analyses showed that participants tended to rely less on subtitles with Chinese or English audio than without audio, and the effects of audio were more pronounced in the presence of video presentation. Lexical processing of subtitles was not modulated by the audio. However, Chinese audio, which presumably obviated the need to read the subtitles, resulted in more superficial post-lexical processing of the subtitles relative to either the English or no audio. On the contrary, English audio accentuated post-lexical processing of the subtitles compared with Chinese audio or no audio, indicating that participants might use English audio to support subtitle reading (or vice versa) and thus engaged in deeper processing of the subtitles. These findings suggest that, in multimodal reading situations, eye movements are not only controlled by processing difficulties associated with properties of words (e.g., their frequency and length) but also guided by metacognitive strategies involved in monitoring comprehension and its online modulation by different information sources.

Keywords

audioeye movementsmultimodal integrated-language processingreadingsubtitles
Type
Original Article
Information
Applied Psycholinguistics , Volume 43 , Issue 1 , January 2022 , pp. 237 - 269
Check for updates
Copyright
© The Author(s), 2021. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Andrews, S. & Veldre, A. (2020). Wrapping up sentence comprehension: The role of task demands and individual differences, Scientific Studies of Reading, DOI: 10.1080/10888438.2020.1817028.CrossRefGoogle Scholar
Atkinson, R. C. & Shiffrin, R. M. (1968). Human memory: A proposed system and its control processes. Psychology of Learning and Motivation, 2, 89–195.CrossRefGoogle Scholar
Atkinson, R. C. & Shiffrin, R. M. (1971). The control of short-term memory. Scientific American, 225, 82–90.CrossRefGoogle ScholarPubMed
Awh, E., Belopolsky, A. V., & Theeuwes, J. (2012). Top-down versus bottom-up attentional control: a failed theoretical dichotomy. Trends in Cognitive Sciences, 16, 437–443.CrossRefGoogle ScholarPubMed
Baker, M. (1992). In other words: A course book on translation. London and New York: Routledge.Google Scholar
Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68, 255–278.CrossRefGoogle ScholarPubMed
Bates, D., Kliegl, R., Vasishth, S., & Baayen, H. (2015). Parsimonious mixed models. Retrieved from https://arxiv.org/pdf/1506.04967.pdf.Google Scholar
Bisson, M. J., van Heuven, W. J. B., Conklin, K., & Tunney, R. J. (2014). Processing of native and foreign language subtitles in films: An eye tracking study. Applied Psycholinguistics, 35, 399–418.CrossRefGoogle Scholar
Cauchard, F., Cane, J. E., & Weger, U. W. (2012). Influence of background speech and music in interrupted reading: An eye-tracking study. Applied Cognitive Psychology, 26, 381–390.CrossRefGoogle Scholar
Cohn, N. (2016). A multimodal parallel architecture: A cognitive framework for multimodal interactions. Cognition, 146, 304–323.CrossRefGoogle ScholarPubMed
d’Ydewalle, G. & De Bruycker, W. (2007). Eye movements of children and adults while reading television subtitles. European Psychologist, 12(3), 196–205.CrossRefGoogle Scholar
d’Ydewalle, G., Praet, C., Verfaillie, K., & Van Rensbergen, J. (1991). Watching subtitled television: Automatic reading behavior. Communication Research, 18(5), 650–666.CrossRefGoogle Scholar
d’Ydewalle, G., Van Rensbergen, J., & Pollet, J. (1987). Reading a message when the same message is available auditorily in another language: The case of subtitling. In O’Regan, J. K. & Lévy-Schoen, A. (Eds.), Eye movements: From physiology to cognition (pp. 313–321). Amsterdam: North-Holland.CrossRefGoogle Scholar
Fothergill, A. (Director). (2006). Planet Earth. Burbank, CA: BBC Video.Google Scholar
Ghia, E. (2012). The impact of translation strategies on subtitle reading. In Perego, E. (Ed.), Eye tracking in audiovisual translation (pp. 157–182). Roma: Aracne.Google Scholar
Graesser, A. C., McNamara, D. S., Cai, Z., Conley, M., Li, H., & Pennebaker, J. (2014). Coh-metrix measures text characteristics at multiple levels of language and discourse. The Elementary School Journal, 115, 210–228.CrossRefGoogle Scholar
Hirotani, M., Frazier, L., & Rayner, K. (2006). Punctuation and intonation effects on clause and sentence wrap-up: Evidence from eye movements. Journal of Memory and Language, 54, 425–443.CrossRefGoogle Scholar
Hyönä, J., & Ekholm, M. (2016). Background speech effects on sentence processing during reading: An eye movement study. PLoS ONE, 11, 1–25.CrossRefGoogle Scholar
Inhoff, A. W. & Rayner, K. (1986). Parafoveal word processing during eye fixations in reading: Effects of word frequency. Perception & Psychophysics, 40, 431–439.CrossRefGoogle ScholarPubMed
Kruger, J.-L. (2016). Psycholinguistics and audiovisual translation. Target, 28, 276–287.CrossRefGoogle Scholar
Kruger, J.-L. & Doherty, S. (2016). Measuring cognitive load in the presence of educational video: Towards a multimodal methodology. Australasian Journal of Educational Technology, 32, 19–31.CrossRefGoogle Scholar
Kuperman, V., Dambacher, M., Nuthmann, A., & Kliegl, R. (2010). The effect of word position on eye-movements in sentence and paragraph reading. Quarterly Journal of Experimental Psychology: Human Experimental Psychology, 63, 1838–1857.CrossRefGoogle ScholarPubMed
Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2017). lmerTest Package: Tests in Linear Mixed Effects Models. Journal of Statistical Software, 82, 1–26.CrossRefGoogle Scholar
Lång, J. (2016). Subtitles vs. narration: The acquisition of information from visual-verbal and audio-verbal channels when watching a television documentary. In Hansen-Schirra, S. and Grucza, S. (Eds.), Eye tracking and Applied Linguistics (pp. 59–82). Berlin: Language Science Press.Google Scholar
Liao, S., Kruger, J.-L., Doherty, S. (2020). The impact of monolingual and bilingual subtitles on visual attention, cognitive load, and comprehension. The Journal of Specialised Translation, 33, 70–89.Google Scholar
Liao, S., Yu, L., Reichle, E. D., & Kruger, J.-L. (2021). Using eye movements to study the reading of subtitles in video. Scientific Studies of Reading, 25, 417–435.CrossRefGoogle Scholar
Loschky, L. C., Larson, A. M., Smith, T. J., & Magliano, J. P. (2020). The scene perception & event comprehension theory (SPECT) applied to visual narratives. Topics in Cognitive Science, 12(1), 311–351.CrossRefGoogle ScholarPubMed
Lüdecke, D. (2020). sjPlot: data visualization for statistics in social science. https://cran.r-project.org/web/packages/sjPlot/.Google Scholar
Marsh, J. E., Hughes, R. W., & Jones, D. M. (2008). Auditory distraction in semantic memory: A process-based approach. Journal of Memory and Language, 58, 682–700.CrossRefGoogle Scholar
Marsh, J. E., Hughes, R. W., & Jones, D. M. (2009). Interference by process, not content, determines semantic auditory distraction. Cognition, 110, 23–38.CrossRefGoogle Scholar
Martin, R. C., Wogalter, M. S., & Forlano, J. G. (1988). Reading comprehension in the presence of unattended speech and music. Journal of Memory and Language, 27, 382–398.CrossRefGoogle Scholar
Mayer, R. E. (2005). Cognitive Theory of Multimedia Learning. In Mayer, R. E. (Ed.), Cambridge handbook of multimodal learning (pp. 31–48). New York: Cambridge University Press.CrossRefGoogle Scholar
Mayer, R. E. (Ed.). (2014). The Cambridge handbook of multimedia learning. New York: Cambridge University Press.CrossRefGoogle Scholar
Paivio, A. (1986). Mental representations: A dual coding approach. New York: Oxford University Press.Google Scholar
Paivio, A. (2007). Mind and its evolution: A dual coding theoretical approach. Mahwah, NJ: Erlbaum.Google Scholar
Pollatsek, A., Juhasz, B. J., Reichle, E. D., Machacek, D., & Rayner, K. (2008). Immediate and delayed effects of word frequency and word length on eye movements in reading: A reversed delayed effect of word length. Journal of Experimental Psychology: Human Perception and Performance, 34, 726–750.Google ScholarPubMed
Pylyshyn, Z. (2004). Some puzzling findings in multiple object tracking: I. Tracking without keeping track of object identities. Visual Cognition, 11, 801–822.CrossRefGoogle Scholar
Pylyshyn, Z. & Storm, R. W. (1988). Tracking multiple independent targets: Evidence for a parallel tracking mechanism. Spatial Vision, 3, 179–197.CrossRefGoogle ScholarPubMed
Ragni, V. (2020). More than meets the eye: An eye-tracking study of the effects of translation on the processing and memorisation of reversed subtitles. Journal of Specialised Translation, 33, 99–128.Google Scholar
Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124, 372–422.CrossRefGoogle ScholarPubMed
Rayner, K. (2009). Eye movements and attention in reading, scene perception, and visual search. The Quarterly Journal of Experimental Psychology, 62, 1457–1506.CrossRefGoogle ScholarPubMed
Rayner, K., Ashby, J., Pollatsek, A., & Reichle, E. D. (2004). The effects of frequency and predictability on eye fixations in reading: Implications for the E-Z Reader model. Journal of Experimental Psychology: Human Perception and Performance, 30, 720–732.Google ScholarPubMed
Rayner, K., Kambe, G., & Duffy, S. A. (2000). The effect of clause wrap-up on eye movements during reading. The Quarterly Journal of Experimental Psychology, 53A, 1061–1080.CrossRefGoogle Scholar
Rayner, K. & McConkie, G. W. (1976). What guides a reader’s eye movements? Vision Research, 16, 829–837.CrossRefGoogle ScholarPubMed
Reichle, E. D. (2021). Computational models of reading: A handbook. Oxford, UK: Oxford University Press.CrossRefGoogle Scholar
Reichle, E. D., Liversedge, S. P., Pollatsek, A., & Rayner, K. (2009). Encoding multiple words simultaneously in reading is implausible. Trends in Cognitive Sciences, 13, 115–119.CrossRefGoogle ScholarPubMed
Reichle, E. D., Pollatsek, A., & Rayner, K. (2006). E-Z Reader: A cognitive-control, serial-attention model of eye-movement behavior during reading. Cognitive Systems Research, 7, 4–22.CrossRefGoogle Scholar
Reichle, E. D., Pollatsek, A., & Rayner, K. (2012). Using E-Z Reader to simulate eye movements in nonreading tasks: A unified framework for understanding the eye-mind link. Psychological Review, 119, 155–185.CrossRefGoogle ScholarPubMed
Reichle, E. D., Yu, L., Liao, S., & Kruger, J.-L. (2021). Using simulations to understand the reading of rapidly displayed subtitles. Proceedings of the 43rd Annual Conference of the Cognitive Science Society.Google Scholar
Ross, N. M., & Kowler, E. (2013). Eye movements while viewing narrated, captioned, and silent videos. Journal of Vision, 13, 1–19.CrossRefGoogle ScholarPubMed
Schilling, H. E. H., Rayner, K., & Chumbley, J. I. (1998). Comparing naming, lexical decision, and eye fixation times: Word frequency effects and individual differences. Memory & Cognition, 26, 1270–1281.CrossRefGoogle ScholarPubMed
Schnotz, W. (2005). Integrated model of text and picture comprehension. In Mayer, R. E. (Ed.), Cambridge handbook of multimodal learning (pp. 49–70). New York: Cambridge University Press.CrossRefGoogle Scholar
Stowe, L. A., Kann, E., Sabourin, L., & Taylor, R. C. (2018). The sentence wrap-up dogma. Cognition, 176, 232–247.CrossRefGoogle ScholarPubMed
Szarkowska, A. & Gerber-Morón, O. (2018). Viewers can keep up with fast subtitles: Evidence from eye movements. PLoS ONE, 13, 1–30.CrossRefGoogle ScholarPubMed
Tiffin-Richards, S. P. & Schroeder, S. (2018). The development of wrap-up processes in text reading: a study of children’s eye movements. Journal of Experimental Psychology: Learning, Memory, and Cognition, 44, 1051–1063.Google ScholarPubMed
van Heuven, W. J. B., Mandera, P., Keuleers, E., & Brysbaert, M. (2014). SUBTLEX-UK: A new and improved word frequency database for British English. Quarterly Journal of Experimental Psychology: Human Experimental Psychology, 67, 1176–1190. DOI: 10.1080/17470218.2013.850521.CrossRefGoogle ScholarPubMed
Vasilev, M. R., Kirkby, J. A., & Angele, B. (2018). Auditory distraction during reading: A Bayesian meta-analysis of a continuing controversy. Perspectives on Psychological Science, 13, 567–597.CrossRefGoogle ScholarPubMed
Vasilev, M. R., Liversedge, S. P., Rowan, D., Kirkby, J. A., & Angele, B. (2019). Reading is disrupted by intelligible background speech: Evidence from eye-tracking. Journal of Experimental Psychology: Human Perception and Performance, 45, 1484–1512.Google ScholarPubMed
von der Malsburg, T., & Angele, B. (2017). False positives and other statistical errors in standard analyses of eye movements in reading. Journal of Memory and Language, 94, 119–133.CrossRefGoogle ScholarPubMed
Warren, T., White, S.J., & Reichle, E. D. (2009). Investigating the causes of wrap-up effects: Evidence from eye movements and E-Z Reader. Cognition, 111, 132–137.CrossRefGoogle ScholarPubMed
Wu, G. (2010). Translation difference – A hybrid model for translation training. The International Journal for Translation & Interpreting Research, 2, 24–37.Google Scholar
Yan, G., Meng, Z., Liu, N., He, L., & Paterson, K. B. (2018). Effects of irrelevant background speech on eye movements during reading. Quarterly Journal of Experimental Psychology: Human Experimental Psychology, 71, 1270–1275.CrossRefGoogle ScholarPubMed
Zhang, H., Miller, K., Cleveland, R., & Cortina, K. (2018). How listening to music affects reading: Evidence from eye tracking. Journal of Experimental Psychology: Learning Memory and Cognition, 44, 1778–1791.Google ScholarPubMed