Hostname: page-component-8448b6f56d-c47g7 Total loading time: 0 Render date: 2024-04-23T21:04:14.092Z Has data issue: false hasContentIssue false
  • English
  • Français

Partial and synchronized captioning: A new tool to assist learners in developing second language listening skill

Published online by Cambridge University Press:  02 March 2017

Maryam Sadat Mirzaei ,
Kourosh Meshgi ,
Yuya Akita  and
Tatsuya Kawahara
Maryam Sadat Mirzaei
Affiliation:
Kyoto University, Japan (email: maryam@sap.ist.i.kyoto-u.ac.jp)
Kourosh Meshgi
Affiliation:
Kyoto University, Japan (email: meshgi-k@sys.i.kyoto-u.ac.jp)
Yuya Akita
Affiliation:
Kyoto University, Japan (email: akita@econ.kyoto-u.ac.jp)
Tatsuya Kawahara
Affiliation:
Kyoto University, Japan (email: kawahara@i.kyoto-u.ac.jp)
Get access Rights & Permissions [Opens in a new window]

Abstract

This paper introduces a novel captioning method, partial and synchronized captioning (PSC), as a tool for developing second language (L2) listening skills. Unlike conventional full captioning, which provides the full text and allows comprehension of the material merely by reading, PSC promotes listening to the speech by presenting a selected subset of words, where each word is synched to its corresponding speech signal. In this method, word-level synchronization is realized by an automatic speech recognition (ASR) system, dedicated to the desired corpora. This feature allows the learners to become familiar with the correspondences between words and their utterances. Partialization is done by automatically selecting words or phrases likely to hinder listening comprehension. In this work we presume that the incidence of infrequent or specific words and fast delivery of speech are major barriers to listening comprehension. The word selection criteria are thus based on three factors: speech rate, word frequency and specificity. The thresholds for these features are adjusted to the proficiency level of the learners. The selected words are presented to aid listening comprehension while the remaining words are masked in order to keep learners listening to the audio. PSC was evaluated against no-captioning and full-captioning conditions using TED videos. The results indicate that PSC leads to the same level of comprehension as the full-captioning method while presenting less than 30% of the transcript. Furthermore, compared with the other methods, PSC can serve as an effective medium for decreasing dependence on captions and preparing learners to listen without any assistance.

Keywords

listening comprehensionpartial and synchronized captionword frequencyspeech rateautomatic speech recognition
Type
Regular papers
Information
ReCALL , Volume 29 , Issue 2 , May 2017 , pp. 178 - 199
Check for updates
Copyright
Copyright © European Association for Computer Assisted Language Learning 2017 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Baddeley, A. (1992) Working memory. Science, 255(5044): 556–559.CrossRefGoogle ScholarPubMed
Bailly, G. and Barbour, W. S. (2011) Synchronous reading: Learning French orthography by audiovisual training. In: Cosi, P., De Mori, R., Di Fabbrizio, G. and Pieraccini, R. (eds.), Proceedings of the 12th Annual Conference of the International Speech Communication Association ( Interspeech 2011 ). Florence, Italy, 1153–1156.CrossRefGoogle Scholar
Bird, S. A. and Williams, J. N. (2002) The effect of bimodal input on implicit and explicit memory: An investigation into the benefits of within-language subtitling. Applied Psycholinguistics, 23(4): 509–533.CrossRefGoogle Scholar
Bird, S., Klein, E. and Loper, E. (2009) Natural language processing with Python. Sebastopol, CA: O’Reilly Media.Google Scholar
Bloomfield, A., Wayland, S. C., Rhoades, E., Blodgett, A., Linck, J. and Ross, S. (2010) What makes listening difficult? Factors affecting second language listening comprehension. College Park, MD: University of Maryland, Center for Advanced Study of Language.CrossRefGoogle Scholar
Braunschweiler, N., Gales, M. J. and Buchholz, S. (2010) Lightly supervised recognition for automatic alignment of large coherent speech recordings. In: Kobayashi, T., Hirose, K. and Nakamura, S. (eds.), Proceedings of the 11th Annual Conference of the International Speech Communication Association ( Interspeech 2010 ). Makuhari, Japan, 2222–2225.CrossRefGoogle Scholar
Buck, G. (2001) Assessing listening. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Chang, A. C. S. (2009) Gains to L2 listeners from reading while listening vs. listening only in comprehending short stories. System, 37(4): 652–663.CrossRefGoogle Scholar
Coxhead, A. (2000) A new academic word list. TESOL Quarterly, 34(2): 213–238.CrossRefGoogle Scholar
Danan, M. (1992) Reversed subtitling and dual coding theory: New directions for foreign language instruction. Language Learning, 42(4): 497–527.CrossRefGoogle Scholar
Danan, M. (2004) Captioning and subtitling: Undervalued language learning strategies. META, 49(1): 66–77.CrossRefGoogle Scholar
Davies, M. (2008) The Corpus of Contemporary American English: 450 million words, 1990–present. http://corpus.byu.edu/coca/ (accessed October, 2014).Google Scholar
Diao, Y., Chandler, P. and Sweller, J. (2007) The effect of written text on comprehension of spoken English as a foreign language. The American Journal of Psychology, 120(2): 237–261.CrossRefGoogle ScholarPubMed
Ellis, N. C. (2003) Constructions, chunking, and connectionism: The emergence of second language structure. In Doughty, C. J. and Long, M. H. (eds.), The handbook of second language acquisition. Oxford: Blackwell, 63–103.CrossRefGoogle Scholar
Gardner, D. and Davies, M. (2013) A new academic vocabulary list. Applied Linguistics, 35(3): 305–327.CrossRefGoogle Scholar
Garza, T. J. (1991) Evaluating the use of captioned video materials in advanced foreign language learning. Foreign Language Annals, 24(3): 239–258.CrossRefGoogle Scholar
Gilmore, A. (2007) Authentic materials and authenticity in foreign language learning. Language Teaching, 40(2): 97–118.CrossRefGoogle Scholar
Goh, C. (2000) A cognitive perspective on language learners’ listening comprehension problems. System, 28(1): 55–75.CrossRefGoogle Scholar
Griffiths, R. (1992) Speech rate and listening comprehension: Further evidence of the relationship. TESOL Quarterly, 26(2): 385–390.CrossRefGoogle Scholar
Guillory, H. G. (1998) The effects of keyword captions to authentic French video on learner comprehension. Calico Journal, 15(1–3): 89–108.CrossRefGoogle Scholar
Inhoff, A. W. and Rayner, K. (1986) Parafoveal word processing during eye fixations in reading: Effects of word frequency. Perception & Psychophysics, 40(6): 431–439.CrossRefGoogle ScholarPubMed
King, J. (2002) Using DVD feature films in the EFL classroom. Computer Assisted Language Learning, 15(5): 509–523.CrossRefGoogle Scholar
Korat, O. (2010) Reading electronic books as a support for vocabulary, story comprehension and word reading in kindergarten and first grade. Computers & Education, 55(1): 24–31.CrossRefGoogle Scholar
Krashen, S. D. (1985) The input hypothesis: Issues and implications. Harlow: Longman.Google Scholar
Leveridge, A. N. and Yang, J. C. (2013) Testing learner reliance on caption supports in second language listening comprehension multimedia environments. ReCALL, 25(2): 199–214.CrossRefGoogle Scholar
Lund, R. J. (1991) A comparison of second language listening and reading comprehension. The Modern Language Journal, 75(2): 196–204.CrossRefGoogle Scholar
Markham, P. (1989) The effects of captioned television videotapes on the listening comprehension of beginning, intermediate, and advanced ESL students. Educational Technology, 29(10): 38–41.Google Scholar
Markham, P. and Peter, L. (2003) The influence of English language and Spanish language captions on foreign language listening/reading comprehension. Journal of Educational Technology Systems, 31(3): 331–341.CrossRefGoogle Scholar
Mayer, R. E. and Moreno, R. (2003) Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38(1): 43–52.CrossRefGoogle Scholar
Mayer, R. E., Lee, H. and Peebles, A. (2014) Multimedia learning in a second language: A cognitive load perspective. Applied Cognitive Psychology, 28(5): 653–660.CrossRefGoogle Scholar
Medwell, J. (1998) The talking books project: Some further insights into the use of talking books to develop reading. Reading, 32(1): 3–8.CrossRefGoogle Scholar
Montero Perez, M., Van den Noortgate, W. and Desmet, P. (2013) Captioned video for L2 listening and vocabulary learning: A meta-analysis. System, 41(3): 720–739.CrossRefGoogle Scholar
Montero Perez, M., Peters, E. and Desmet, P. (2014a) Is less more? Effectiveness and perceived usefulness of keyword and full captioned video for L2 listening comprehension. ReCALL, 26(1): 21–43.CrossRefGoogle Scholar
Montero Perez, M., Peters, E., Clarebout, G. and Desmet, P. (2014b) Effects of captioning on video comprehension and incidental vocabulary learning. Language Learning & Technology, 18(1): 118–141.Google Scholar
Moran, S. (2010) The effect of linguistic variation on subtitle reception. In Perego, E. (ed.), Eye tracking in audiovisual translation, Roma: Aracne Editrice, 183–222.Google Scholar
Nation, I. S. P. (2006) How large a vocabulary is needed for reading and listening? Canadian Modern Language Review, 63(1): 59–82.CrossRefGoogle Scholar
Nation, I. S. P. and Beglar, D. (2007) A vocabulary size test. The Language Teacher, 31(7): 9–13.Google Scholar
Nation, I. S. P. and Webb, S. A. (2011) Researching and analyzing vocabulary. Boston: Heinle Cengage Learning.Google Scholar
Nissan, S., DeVincenzi, F. and Tang, K. L. (1996) An analysis of factors affecting the difficulty of dialogue items in TOEFL listening comprehension. TOEFL Research Report , 51. Princeton, NJ: Educational Testing Service.Google Scholar
Nogami, Y. and Hayashi, N. (2010) A Japanese adaptive test of English as a foreign language: Developmental and operational aspects. In: Van der Linden, W. J. and Glas, C. A. W. (eds.), Elements of adaptive testing, New York: Springer, 191–211.Google Scholar
Osada, N. (2004) Listening comprehension research: A brief review of the past thirty years. Dialogue, 3: 53–66.Google Scholar
Paivio, A. (1990) Mental representations: A dual coding approach. Oxford: Oxford University Press.CrossRefGoogle Scholar
Pimsleur, P., Hancock, C. and Furey, P. (1977) Speech rate and listening comprehension. In Burt, M. K., Dulay, H. B. and Finocchiaro, M. C. (eds.) Viewpoints on English as a second language. New York: Regents, 27–34.Google Scholar
Pujolà, J. T. (2002) CALLing for help: Researching language learning strategies using help facilities in a web-based multimedia program. ReCALL, 14(2): 235–262.CrossRefGoogle Scholar
Révész, A. and Brunfaut, T. (2013) Text characteristics of task input and difficulty in second language listening comprehension. Studies in Second Language Acquisition, 35(1): 31–65.CrossRefGoogle Scholar
Rost, M. (2005) L2 listening. In: Hinkel, E. (ed.) Handbook of research in second language teaching and learning. Mahwah, NJ: Erlbaum, 503–527.Google Scholar
Schmitt, N. and McCarthy, M. (eds.) (1997) Vocabulary: Description, acquisition and pedagogy. Cambridge: Cambridge University Press.Google Scholar
Sweller, J. (1994) Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction, 4(4): 295–312.CrossRefGoogle Scholar
Sydorenko, T. (2010) Modality of input and vocabulary acquisition. Language Learning & Technology, 14(2): 50–73.Google Scholar
Tauroza, S. and Allison, D. (1990) Speech rates in British English. Applied Linguistics, 11(1): 90–105.CrossRefGoogle Scholar
Taylor, G. (2005) Perceived processing strategies of students watching captioned video. Foreign Language Annals, 38(3): 422–427.CrossRefGoogle Scholar
Trancoso, I., Serralheiro, A., Viana, C., Caseiro, D. and Mascarenhas, I. (2007) Digital talking books in multiple languages and varieties. Proceedings of the 3rd Language & Technology Conference. Poznan, Poland.Google Scholar
Vandergrift, L. (2004) Listening to learn or learning to listen? ARAL, 24(1): 3–25.Google Scholar
Vandergrift, L. (2007) Recent developments in second and foreign language listening comprehension research. Language Teaching, 40(3): 191–210.CrossRefGoogle Scholar
Vandergrift, L. (2011) Second language listening: Presage, process, product, and pedagogy. In: Hinkel, E. (ed.), Handbook of research in second language teaching and learning. New York/London: Routledge, 455–471.Google Scholar
Vanderplank, R. (1988) The value of teletext sub-titles in language learning. ELT Journal, 42(4): 272–281.CrossRefGoogle Scholar
Vanderplank, R. (2010) Déjà vu? A decade of research on language laboratories, television and video in language learning. Language Teaching, 43(1): 1–37.CrossRefGoogle Scholar
Wang, D. and Narayanan, S. (2005) An unsupervised quantitative measure for word prominence in spontaneous speech. In: IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP ’05). Philadelphia, PA: IEEE, 377–380. doi: 10.1109/ICASSP.2005.1415129.Google Scholar
Webb, S. (2010) Using glossaries to increase the lexical coverage of television programs. Reading in a Foreign Language, 22(1): 201–221.Google Scholar
Winke, P., Gass, S. and Sydorenko, T. (2010) The effects of captioning videos used for foreign language listening activities. Language Learning & Technology, 14(1): 65–86.Google Scholar
Winke, P., Gass, S. and Sydorenko, T. (2013) Factors influencing the use of captions by foreign language learners: An eye-tracking study. The Modern Language Journal, 97(1): 254–275.CrossRefGoogle Scholar
Zhao, Y. (1997) The effects of listeners’ control of speech rate on second language comprehension. Applied Linguistics, 18(1): 49–68.CrossRefGoogle Scholar
Supplementary material: File

Mirzaei supplementary material

Mirzaei supplementary material 1

Download Mirzaei supplementary material(File)
File 265.3 KB