Studies on Lexical Coverage and L2 Reading Comprehension

The Lexical Quality Hypothesis outlines word knowledge as central to reading comprehension (Perfetti & Stafura, Reference Perfetti and Stafura2014). Although the relationship between lexical coverage and L2 reading comprehension has been widely explored (Laufer, Reference Laufer, Lauren and Nordman1989, Reference Laufer1992; Laufer & Sim, Reference Laufer and Sim1985; Qian Reference Qian1999; Hu and Nation Reference Hu and Nation2000), four studies are particularly relevant to the present investigation because of their findings and methodology.

Hu and Nation (Reference Hu and Nation2000) examined the percentage of lexical coverage needed for unassisted L2 reading for pleasure. They created four versions of a fictional text by replacing words with nonwords:

1. 1. The 100% coverage version containing no nonwords,

2. 2. The 95% coverage version with 5% nonwords,

3. 3. The 90% coverage version with 10% nonwords, and

4. 4. The 80% coverage version with 20% nonwords.

The authors ensured that all the remaining words belonged to the 2000 most frequent word families to avoid vocabulary difficulties. To assess comprehension, a multiple-choice and a written recall test were used. Results showed the following:

• • The density of unknown words affected comprehension;

• • No learners gained adequate comprehension at 80% lexical coverage; and

• • Although it was possible for a few learners to gain adequate comprehension at a level as low as 95% and even 90%, most learners did not.

Due to the limited number of learners obtaining adequate comprehension at 90% to 95%, Hu and Nation (Reference Hu and Nation2000) asserted that, although they did not specifically investigate the 98% coverage level, it would be the most reliable lexical coverage for most learners to attain unassisted comprehension of a fiction text. Therefore, it is the coverage suggested by the authors. It should be noted that Hu and Nation defined adequate comprehension as obtaining 12 of 14 correct answers on the multiple-choice test and 70 of 124 on the written recall test, which was the most recurrent score on the comprehension tests (mode).

Research by Laufer and Ravenhorst-Kalovski (Reference Laufer and Ravenhorst-Kalovski2010) suggested a coverage level of 95% to 98% for comprehension of written texts. In their study, the authors explored the relationship between adequate reading comprehension, lexical coverage, and vocabulary knowledge. Reading comprehension was assessed by means of the English part of a standardized university entrance test. Unlike Hu and Nation (Reference Hu and Nation2000), Laufer and Ravenhorst-Kalovski (Reference Laufer and Ravenhorst-Kalovski2010) did not replace words with nonwords to manipulate the degree of lexical coverage. Instead, they estimated lexical coverage by comparing participants’ results on the revised version of Nation’s (Reference Nation1983) vocabulary levels text (VLT; Schmitt, Schmitt, & Clapham, Reference Schmitt, Schmitt and Clapham2001) with the text vocabulary profiling. That is to say, if participants were familiar with the 3000 most frequent word families, they were expected to have 90% lexical coverage. Results showed that the higher the lexical coverage, the better the reading scores. The authors, then, suggest an optimal threshold reached at 98% lexical coverage, which corresponds to knowledge of the 6000 to 8000 most frequent word families, and a minimal threshold obtained at 95% lexical coverage with knowledge of the 4000- to 5000-word families. Adequate comprehension was defined as obtaining 134 of 150 on the standardized entrance test.

Schmitt et al. (Reference Schmitt, Jiang and Grabe2011) argued that the findings from Hu and Nation (Reference Hu and Nation2000) clearly showed that comprehension increases when vocabulary knowledge increases. The authors explored the relationship between the percentage of known words in a text and the degree of reading comprehension. Instead of manipulating a text (Hu & Nation, Reference Hu and Nation2000) or using a VLT (Laufer & Ravenhorst-Kalovski, Reference Laufer and Ravenhorst-Kalovski2010) to determine lexical coverage, the authors tested participants on a large sample of the words occurring in two academic texts. To that end, they developed a checklist vocabulary test. A learner’s lexical coverage was determined based on their score on the checklist test. That is to say, scores were split into 1% coverage bands (viz. 99%, 98%, 97%, etc.). Comprehension was measured through a multiple-choice test and a graphic organizer. Participants’ mean comprehension scores were calculated for each coverage percentage. Results of the study indicated that there was a positive, medium-sized linear correlation (r = .41), between lexical coverage and reading comprehension. The lexical coverage needed would depend on the degree of comprehension required. If 60% was considered adequate, then 95% lexical coverage should be targeted, whereas when 70% is necessary, then 98% to 99% is required. Schmitt et al. (Reference Schmitt, Jiang and Grabe2011) suggest that 98% coverage is a reasonable target for reading academic texts.

The most recent research on the relationship between lexical coverage and reading comprehension is the one conducted by Song and Reynolds (Reference Song and Reynolds2022). In their study, the authors examined the effect of two language learning variables (i.e., lexical coverage and topic familiarity) on L2 comprehension of expository texts by controlling for L2 reading ability and vocabulary size. Like Hu and Nation (Reference Hu and Nation2000), the authors manipulated two texts by replacing words with nonwords to create six levels of lexical coverage: 100%, 99%, 98%, 97, and 96%. Comprehension was measured through two reading comprehension tests, one for a familiar topic and one for an unfamiliar topic. Results revealed a nonsignificant interaction between topic familiarity and lexical coverage on comprehension scores. Although topic familiarity exhibited a significant main effect, lexical coverage did not. This suggests that, with controlled L2 vocabulary size and reading ability, topic familiarity has a larger effect on the expository comprehension of L2 learners than lexical coverage. These findings shed more light on the lexical coverage–comprehension relationship, adding to the results from previous studies that primarily focused on the role of lexical coverage in L2 reading comprehension. Regardless of lexical coverage, topic familiarity enhanced expository reading comprehension. This underscores the vital role of subject matter in text comprehension, enabling readers to filter out irrelevant information and potentially facilitating the inference of unknown words’ meanings. While acknowledging the importance of lexical coverage, the study supports the claim that it is just one of the many factors influencing L2 reading comprehension. These findings align with recent research that suggests that the relationship may be more complex than initially thought.

Two recent meta-analyses about the relationship between vocabulary knowledge and reading comprehension have found a correlation ranging from .57 (unattenuated correlation) to .67 (corrected correlation for attenuation) in Zhang and Zhang (Reference Zhang and Zhang2020) and .79 in Jeon and Yamashita (Reference Jeon and Yamashita2014). Results of the meta-analyses indicate that vocabulary knowledge accounts for more than a 31% variance in L2 reading comprehension (Zhang & Zhang, Reference Zhang and Zhang2020).

In summary, previous research has agreed that 95% to 98% lexical coverage is necessary to comprehend a text adequately. Although more recent research has found that topic familiarity may be a better predictor of text comprehension, the two meta-analyses have shown that vocabulary knowledge can still predict at least 31% of the variance in L2 reading.

Studies on Lexical Coverage and L2 Listening Comprehension

Although more studies have been conducted on the relationship between lexical coverage and reading than on listening, the number of studies focusing on listening is on the rise. Two approaches to the study of lexical coverage and listening are discussed in this section: studies using VLTs matched with text profiling to estimate whether learners have the lexical coverage necessary to understand the input (Noreillie et al., Reference Noreillie, Kestemont, Heylen, Desmet and Peters2018; Stæhr, Reference Stæhr2009) and studies manipulating the lexical coverage by replacing words with nonwords (Giordano, Reference Godfroid2020; van Zeeland & Schmitt, Reference van Zeeland and Schmitt2013).

Stæhr (Reference Stæhr2009) examined the correlation between vocabulary size and depth and adequate listening comprehension at an advanced proficiency level. Danish English as a foreign language participants took the second version of Schmitt et al.’s (Reference Schmitt, Schmitt and Clapham2001) VLT, a vocabulary depth test (an adapted version of the Word Associates Test), and a listening comprehension test (Cambridge Certificate of Proficiency in English). To estimate participants’ lexical coverage, the listening passages were submitted to a lexical frequency analysis, matched to the participants’ results on the VLT, and related to the listening comprehension scores. Therefore, if learners were familiar with the 3K level in the VLT, they were assumed to know 94% of the words in the text and had a mean comprehension score of 59.1%, whereas knowledge of the 5K would correspond to the understanding of 98% of the text and lead to a mean comprehension score of 72.9%. Adequate comprehension was operationalized as obtaining at least 70% on the comprehension test. Results indicated that vocabulary size and depth were significantly correlated with listening comprehension (.70 and .65, respectively), with vocabulary size a major predictor of listening comprehension. Furthermore, the author explains that a lexical coverage of 98% and a vocabulary target of 5000 words would largely facilitate listening comprehension and would allow learners to reach a mean comprehension score of 73% on an advanced listening test.

Whereas Staehr focused on an advanced level, Noreillie et al. (Reference Noreillie, Kestemont, Heylen, Desmet and Peters2018) looked at the relationship between lexical coverage and listening comprehension at an intermediate level. In their adapted design, they conducted two studies. In the first, learners took the Cambridge Preliminary English Test (PET) and a self-designed vocabulary test (VocabLab test; Peters Velghe, & Van Rompaey, Reference Peters, Velghe and Van Rompaey2015). In the second study, participants took a French vocabulary test and the Diplôme d’études en langue française (DELF), a listening comprehension test for French. Findings indicated that there was a positive, strong correlation, r_s = .63, between lexical coverage and listening comprehension at an intermediate level. Additionally, they found that a lower coverage of approximately 90% may be sufficient to achieve adequate comprehension. Adequate comprehension was operationalized as obtaining 70% on the PET and 62% on the DELF test. One limitation of the study was that written VLTs are designed to measure written word form, which might not directly reflect the participants’ knowledge of the spoken form of words.

Van Zeeland and Schmitt (Reference van Zeeland and Schmitt2013) used a different approach to studying lexical coverage and listening. They drew on Hu and Nation’s (Reference Hu and Nation2000) method and used nonwords to create texts with different degrees of lexical coverage. Furthermore, they included first-language (L1) participants in addition to L2 participants to investigate how lexical coverage affected L1 learners’ listening comprehension skills. The authors manipulated four spoken, informal narrative passages by replacing nonwords to create four different coverage levels: 100%, 98%, 95%, and 90%. Results indicated that there was no statistical difference between comprehension scores at 95% and 90% coverage levels as both led L1 and L2 speakers to adequately comprehend the spoken texts. However, because L2 speakers showed greater variation in comprehension scores at the 90% level than at the 95% level, the authors concluded that 95% coverage may be needed for comprehension of informal texts. If detailed comprehension is needed, then 98% coverage may be the optimal coverage level. Given that the texts used for this study were everyday storytelling, the results may not be applicable to other genres.

One of the latest studies on the relationship between lexical coverage and listening comprehension was conducted by Giordano (Reference Giordano2021). In his quasi-experimental research, the author inserted nonwords in five dialogs to create different levels of lexical coverage (viz. 98%, 95%, 90%, 85%, and 83%) at an intermediate level of English. The author also explored other variables that might affect comprehension, such as speech rate, topic familiarity, and discourse structure. Adequate comprehension was defined as obtaining 7 of 10 correct answers. Results supported previous findings on listening with learners having adequate comprehension at 90% lexical coverage. However, learners rated the 90% lexical coverage dialogue as easier than the other dialogs, which may explain why that lexical coverage level had the higher comprehension scores.

A recent meta-analysis found a correlation ranging from .56 (unattenuated) to .67 (corrected for attenuation; Zhang & Zhang, Reference Zhang and Zhang2020) between vocabulary knowledge and listening comprehension. The reason for the slightly lower correlation between listening and reading comprehension (Jeon & Yamashita, Reference Jeon and Yamashita2014; Zhang & Zhang, Reference Zhang and Zhang2020) may lie in the differences between spoken and written texts. Listening comprehension entails phonological knowledge and strategic competence to process the incoming stream of speech quickly and automatically (Stæhr, Reference Stæhr2008). Additionally, listening contains nonverbal aspects, such as rhythm, emphasis, tone, etc., that may facilitate comprehension (Durbahn et al., Reference Durbahn, Rodgers and Peters2020). Therefore, learners may not only rely on vocabulary knowledge but also on the other aspects of spoken language to understand listening texts.

Studies on Lexical Coverage and L2 Viewing Comprehension

The relationship between lexical coverage and viewing comprehension remains underexplored. Given that the imagery present in videos may help understand the meaning of the input, the relationship between lexical coverage and viewing comprehension may be different than with reading or listening. Mayer’s (Reference Mayer2009) Cognitive Theory of Multimedia Learning postulates that “People learn better from words and pictures than from words alone” (p. 1). Multimedia learning, also referred to as “dual-mode, dual-format, dual-code, or dual-channel learning” (p. 5), attempts to explain how the human mind processes information. Three assumptions underlie the theory (Mayer, 2014): dual channels, limited capacity, and active processing.

The dual-channel assumption, rooted in Paivio’s (Reference Paivio1986) Dual-Coding Theory, explains that humans process information through two separate channels (viz. auditory and visual channels). Although separate, learners may be able to convert the information represented in one channel and process it in the other channel. When information is presented in a more engaging and multisensory manner, it can enhance learning and, therefore, comprehension.

The Limited Capacity assumption is that humans can effectively process a finite amount of information in each channel, such as a few words (auditory channel) or a few images (visual channel). Well-constructed materials can enhance comprehension by reducing cognitive load, avoiding overload, focusing attention, and prioritizing essential information. This ensures that learners can assimilate and internalize the information effectively, leading to improved comprehension of the presented input.

The Active Processing Assumption is that humans take an active role in mentally processing information to form a cohesive representation of their experiences. This involves actively selecting pertinent incoming data, organizing it into a coherent cognitive framework, and integrating it with existing knowledge. Mayer’s Cognitive Theory of Multimedia Learning can be closely related to viewing comprehension since watching a video would represent a multimedia experience where words and images are presented in close proximity. That proximity was found to be more recurrent in the case of documentaries (Rodgers, Reference Rodgers2018).

In light of the definitions provided above, Mayer (2014) proposes that multimedia instruction should be designed aiming at taking full advantage of how the human mind works (i.e., assuming that humans have two information processing systems; that humans can process a limited amount of information in each channel, but that the information of each channel can be combined; and that humans actively engage in cognitive processes to make sense of multimedia presentations). When material is presented using only one channel, as in reading only and listening only, the potential contribution of the other channel is ignored (Mayer, Reference Mayer2009). Although Mayer’s theory is primarily focused on learning, enhanced learning processes often result in better comprehension of the input.

Less research has been carried out on viewing comprehension. Two corpus studies have been conducted in which the authors performed a lexical frequency analysis of a large corpus of TV programs and movie scripts (Webb & Rodgers, Reference Webb and Rodgers2009a, Reference Webb and Rodgers2009b). The aim was to determine the vocabulary demands needed to reach 95% and 98% coverage of the words in TV shows and movies. Those figures were borrowed from the percentages suggested in reading research to achieve minimal and adequate comprehension (Laufer & Ravenhorst-Kalovski, Reference Laufer and Ravenhorst-Kalovski2010), respectively; results revealed that learners need from 4000- to 5000-word families to achieve 95% lexical coverage and 8000 to attain 98% coverage. However, it is not clear whether the coverage figures for reading may indeed be transferred to viewing. Like listening, factors inherent to multimedia input, such as the presence of the imagery, may also affect viewing comprehension.

To date, only one study has empirically explored the relationship between lexical coverage and L2 viewing comprehension (Durbahn et al., Reference Durbahn, Rodgers and Peters2020). Like Schmitt et al. (Reference Schmitt, Jiang and Grabe2011), the authors tested the words that occurred in the input. To that end, they sampled 146 (56%) words occurring in a 30-min documentary and tested them in an aural meaning-recall test. Learners’ lexical coverage was calculated based on their results on the meaning-recall test. Additionally, the study also addressed the effect of imagery on viewing comprehension by assessing comprehension through questions designed to be answered by listening to the audio (audio-based), by paying attention to the imagery, and by listening to the audio (audio- and imagery-based) and viewing the imagery (imagery-based). One hundred and fourteen learners participated in the study. Results showed:

1. 1. An almost medium-sized correlation between knowledge of the words in the documentary and viewing comprehension (r_s [94] = .39);

2. 2. A positive, almost medium-sized correlation between audio-based questions and lexical coverage (r_s [94] = .36);

3. 3. A small-sized correlation between imagery-based questions and lexical coverage (r_s [94] = .29), and

4. 4. No significant correlation between lexical coverage and audio- plus imagery-based questions.

Findings indicated that lexical coverage for viewing documentaries is lower than the lexical coverage for unassisted reading but similar to informal listening (Schmitt et al., Reference Schmitt, Jiang and Grabe2011). The nonsignificant correlation between audio- plus imagery-based questions and viewing comprehension may also suggest that vocabulary plays a different role in viewing comprehension than in listening comprehension because of the presence of imagery. Imagery may help learners access word meaning, guess the meaning of unfamiliar words from context, and complete the meaning of partially known words (Peters, Reference Peters2019; Webb & Rodgers, Reference Webb and Rodgers2009a). Additionally, viewing has been found to have lower lexical demands than reading (Webb & Rodgers, Reference Webb and Rodgers2009a,Reference Webb and Rodgersb). Webb (Reference Webb2021), based on his discussion about lexical coverage and lexical profiling, indicates that 90% would be sufficient to understand audiovisual input. Finally, TV viewing may be easier to understand than listening because of the presence of imagery (Webb & Rodgers, Reference Webb and Rodgers2009b). All the reasons provided above lead us to assume that transferring coverage figures from reading or listening to viewing comprehension may not be appropriate. One of the limitations of Durbahn et al.’s (Reference Durbahn, Rodgers and Peters2020) study is that they only examined the correlation between lexical coverage and viewing comprehension. Further research examining the effect that different levels of lexical coverage have on viewing comprehension is needed to better understand how the knowledge of words affects comprehension when the input is supported by imagery.

In summary, lexical coverage has been investigated by means of three methodological approaches:

1. 1. By matching learners’ scores on vocabulary (levels) tests with the profile of the text they are supposed to read or listen to;

2. 2. Manipulating text by replacing words with nonwords to create different lexical coverage levels (Giordano, Reference Giordano2021; Hu & Nation, Reference Hu and Nation2000) and

3. 3. Corpus analyses (Webb & Rodgers, Reference Webb and Rodgers2009a, Reference Webb and Rodgers2009b).

The first approach can, in turn, be divided into: estimating the mastery of different frequency levels (Laufer & Ravenhorst-Kalovski, Reference Laufer and Ravenhorst-Kalovski2010; Noreillie et al., Reference Noreillie, Kestemont, Heylen, Desmet and Peters2018) and assessing the knowledge of words that appear in target written or spoken texts (Schmitt et al., Reference Schmitt, Jiang and Grabe2011; Durbahn et al., Reference Durbahn, Rodgers and Peters2020). The use of VLTs and corpus analyses can be considered indirect ways of determining lexical coverage, whereas text manipulation can more accurately give an account of learners’ lexical coverage. Therefore, more direct methods, such as text manipulation, are needed to investigate the effect of the degree of lexical coverage on viewing comprehension. Furthermore, it is worth noting that there is no consensus on what “adequate comprehension” is. What is adequate in one study may not be so in another (Nurmukhamedov & Webb, Reference Nurmukhamedov and Webb2019) because adequate “may refer to different levels of comprehension in different contexts” (Laufer & Ravenhorst-Kalovski, Reference Laufer and Ravenhorst-Kalovski2010, p. 16). Consequently, results cannot always be compared.

Importantly, there may be factors involved in comprehension other than vocabulary knowledge, such as background knowledge (Pulido, Reference Pulido2007; Song & Reynolds, Reference Song and Reynolds2022; Schmitt et al., Reference Schmitt, Jiang and Grabe2011) or inferring ability (Laufer, Reference Laufer2020), which make comprehension vary among learners when facing written and spoken input even if they have the same coverage. In several studies (Hu & Nation, Reference Hu and Nation2000; Schmitt et al., Reference Schmitt, Jiang and Grabe2011; van Zeeland & Schmitt, Reference van Zeeland and Schmitt2013, among others), students with even 100% coverage had poor comprehension scores. That is why Webb (Reference Webb2021) stresses that knowing 100% of the words in the input does not ensure 100% comprehension scores and that spoken and written texts are not always thoroughly understood at that level. Therefore, “while reaching target lexical coverage figures might indicate that text might be comprehensible, it does not ensure comprehension” (Webb, Reference Webb2021, p. 281).

Participants

Ninety-seven Spanish speakers whose L2 was English participated in the study. They were all undergraduate students from a university in Chile from the following programs: Bachelor in Language, English Language Pedagogy, English Literature and Linguistics, and Bachelor of Arts and Humanities. All participants were taking either English Language II or English Language IV courses, which means they had more than 5 hours a week of formal English language instruction plus other English language-related courses. They were selected on the basis of their scores on Webb et al.’s (Reference Webb, Sasao and Balance2017) updated Vocabulary Levels Tests (uVLT) for two reasons:

• • To identify whether they were familiar with at least the 2000 words and

• • To match participants’ vocabulary test scores with the text profile and ensure that the coverage levels were indeed 100%, 95%, 90%, and 80% and not lower.

Results from the uVLT indicated that 17 participants were not familiar with 1K or 2K; 2 participants had an L1 other than Spanish; and 2 participants’ comprehension scores in all the video clips were too low (one or two correct answers), meaning that they either did not have the proficiency level adequate to understand the clips, which was not the case according to the uVLT, or that they were not taking the study seriously. Data from those learners were excluded, resulting in 76 L2 participants. The remaining participants were 61 females, 12 males, 2 agender, and 1 nonbinary. Their ages ranged from 18 to 40 years old (M = 21.30; SD = 4.60). Table 1 presents the descriptive statistics of the uVLT results after the exclusion of the participants mentioned above.

Table 1.

Descriptive statistics of the updated vocabulary levels test (N = 76)

As in van Zeeland and Schmitt, data was also collected with native speakers of English. Sixty L1 learners took part in this study. However, only 40 L1 participants completed the experiment. They were invited via the Prolific platform, an online platform that finds and invites participants who match a specific study’s requirements and pays them after they have completed all the tasks. Only L1 English speakers who were currently studying at university were invited to make the sample of L1 speakers as comparable as possible with L2 speakers. No specific regions were targeted. The L1 speaker participants were 25 women and 15 men whose ages ranged from 18 to 40 years old (M = 22.53; SD = 5.14). There were 39 undergraduate students and 1 graduate student. All participants were from the United Kingdom living in the United Kingdom, except for one who was living in Spain.

Video Excerpts and Viewing Comprehension Test

The audiovisual texts used were four clips of approximately 2 min each (Table 2) of the first season of the documentary Planet Earth (2006), episodes: Deep Water, Forests, Fresh Water, and Mountains. The same documentary as Rodgers (Reference Rodgers2018) and Peters (Reference Peters2019) was selected for comparability purposes. Documentaries were chosen because they have been found to contain more word-imagery matching than narrative TV genres (Rodgers, Reference Rodgers2018). Additionally, in the background questionnaire, 100% of the L2 participants reported not having seen the documentary before.

Table 2.

Lexical frequency profile of the clips unaltered

As this study’s purpose was to explore the extent to which lexical coverage affects viewing comprehension, it was important to determine if the clips were similar in terms of comprehensibility. We asked six highly proficient L2 and eight L1 speakers to rate the clips. They watched the unaltered clips and ranked them according to their level of comprehensibility. Eight raters indicated that the videos were equally comprehensible. Five of the 13 participants indicated that the clips had different levels of comprehensibility. However, there was no agreement on the hierarchy of comprehensibility, which led us to assume that none of the clips was particularly more challenging than the others.

As the clips needed reworking by recording the voice over to create the different levels of lexical coverage, they were transcribed with the help of the online tool Happy Scribe (www.happyscribe.com). They were cleaned by transforming all the contractions to their full forms (Rodgers, Reference Rodgers2013). Next, they were submitted to the free software, AntWordProfiler (Anthony, Reference Anthony2014; Table 2) to compute the lexical profile of the clips. Because the clips were short, some sentences were added so that they had a similar number of tokens. Sentences were also added to clarify or complement some of the information. For example, in the Fresh Water video, it said: The skin of some species contains a powerful poison; these salamanders tend to be slow-moving and have bright warning coloration to advertise their harmfulness. Free from competition, these giants can dine alone. The sentence about their warning coloration was added to complement the information about the poison in their skins. Following Hu and Nation (Reference Hu and Nation2000), the texts were submitted to a process of simplification to make sure all the words were within the 2K most frequent word families. Words that fell over the 2K level were replaced with words of a higher frequency. For example, pine needles was replaced with pine’s sharp pins. A native speaker checked the clips for the naturalness of the language. Results from the lexical frequency profile after the modifications showed that approximately 99% of the tokens in the videos belonged to the 2000 most frequent word families (provided proper names, such as Japan or Indonesia, and English to Spanish cognates, such as opponent or coral, are known). A list of potential cognates was created. A native English speaker, the same person who did the voice-over, was asked to record the words. Subsequently, a group of five native Spanish speakers was informally asked to recognize the words from the list. A word was considered a cognate when it was recognized by all five Spanish speakers. The rating procedure plus the lexical frequency profile showed that the four clips were very similar in terms of topic difficulty and vocabulary load, respectively.

We then created 100%, 95%, 90%, and 80% coverage levels by replacing 0%, 5%, 10%, and 20% of the words in the clips with nonwords, respectively (Table 3). The replaced words were generally words from the 3K to 22K or off-the-list words and were the words that could not be replaced in the process of simplification. As there were not enough items of this type, words were taken randomly, using the free online software www.random.org, from 2K to 1K (in the case of 80%, e.g.). As in Hu and Nation (Reference Hu and Nation2000), if a word occurred more than once in the text, it was replaced with the same no-word on every occasion. The nonwords were taken from the ARC Nonword Database (Rastle, Harrington, & Coltheart, Reference Rastle, Harrington and Coltheart2002), which is the same database that was used by van Zeeland and Schmitt (Reference van Zeeland and Schmitt2013). All nonwords replaced into the passages complied with the phonological and orthographic rules of English, which is the purpose of the database, and were five to eight phonemes long. They sounded or looked like the part of speech of the words they replaced, and participants could easily identify their part of speech, for example: contain → shaint (Appendix A). As the words replaced were mostly low-frequency words and the ones that were high-frequency words were selected at random, each version of the clip would realistically represent the problems that L2 learners may encounter when viewing videos.

Table 3.

Number of words and nonwords in the four clips

Once the items were replaced, the narration for the videos was read and recorded from the scripts by a voice talent at a rate of 118.85 words per minute. To control for individual differences such as pronunciation, accent, and speaking style, the four clips were narrated by the same speaker in standard English (a Canadian L1 English male speaker). The original sound editing was carried out using the software Audacity, and the video editing was done using the software Cyberlink PowerDirector 18.

The present study employed a counterbalanced within-subjects design, where four levels of lexical coverage, 100%, 95%, 90%, and 80%, were created for each topic (Table 4). Each clip was recorded four times for each level of lexical coverage. Later, the videos were combined in such a way that four playlists were created containing one video of each topic with one different level of lexical coverage, all of them in a different order, as shown in Table 4. The playlists were saved as an mp4 file and uploaded on the testing platform Playposit, where the questions were added. Participants were then assigned to one of the four playlists.

Table 4.

Lexical coverage distribution per video topic

The questions were created as follows: A large set of questions was created based on the information in the clips, which resulted in 80 multiple-choice questions. Next, the questions were classified according to the information needed to answer them: verbal information and/or visual information. This resulted in 36 audio-based and 44 audio- plus imagery-based questions. Because of the length of the clips, all the questions were literal wh-questions relating to the central topics of the clips organized in chronological order. They contained three distractors that were related to the topics of the clips, a correct answer, and an “I don’t know” option to avoid guessing. The questions were written in the participants’ first language (English for L1 speakers and Spanish for L2 speakers). They were added after each clip to the testing platform Playposit. After the questions were created, two researchers rated the questions as audio- or audio- plus imagery-based questions. The questions were piloted with a group of 12 volunteers that included researchers, teachers, and proficient L2 speakers. Results revealed that some questions needed rewording, and questions with low scores were excluded. The final set of questions included seven audio-based and seven audio- plus imagery-based questions per clip (N = 56) constituted the comprehension test. A second pilot study with a representative sample of 10 EFL university students was conducted to ensure that the questions were clear, had no obvious answers, had only one correct answer, and that the format on the platform was appropriate. Results showed that pilot participants had, on average, 82.78% (SD = 9.77) correct answers.

We examined Cronbach’s α estimate of the internal consistency of each playlist. Reliability coefficients were calculated with the actual scores from L1 and L2 participants. A Cronbach’s α of .70 is often put forward for acceptable reliability (Cronbach, Reference Cronbach1951). Given the low number of questions, the playlists had an acceptable level of reliability at 100%, albeit with a lower coefficient for playlist D, as can be seen in Table 5. Taber (Reference Taber2017) explains that Cronbach’s α should be interpreted according to the context of a particular study considering the total number of items. In the case of this study, each clip had 14 questions, which may be the reason for the medium reliability coefficient. When the reliability coefficient was computed for each playlist, the α increased and each playlist was reliable as a whole. It is worth mentioning that reliability coefficients for listening are generally low compared to other skills (Plonsky & Derrick, Reference Plonsky and Derrick2016). Despite the lower reliability coefficient in Playlist D, a simple one-way analysis of variance revealed that the comprehension scores from the four different playlists were not significantly different, p >.05. To find the full report of reliability coefficients per clip and coverage levels, see Appendix C.

Table 5.

Reliability coefficients (Cronbach’s α) at 100% coverage and total per playlist

Note: Clips or playlists were considered reliable when α was >.70

Updated Vocabulary Levels Test

To ensure that the L2 participants were familiar with the 2000 most frequent word families and would be able to understand the words occurring in the videos, they took an online version of Webb et al.’s (Reference Webb, Sasao and Balance2017) updated VLTs created on Google Forms (Figure 1). The test contains 50 discrete, selective, context-independent clustered questions aiming at assessing the form-meaning link. Each cluster contains six words and three definitions that have to be matched with the words. In all, 150 words were assessed, 30 per level, from the first five frequency bands sampled from Nation’s (Reference Nation2012) British National Corpus/Corpus of Contemporary American English word lists. The test designers suggest a cutoff score of 29 of 30 correct answers in the first three and 24 in the last two levels. As seen in Table 1, all learners were familiar with the 1K and 2K levels, which suggests they could understand the input from the clips. The test had a medium reliability coefficient (α = .680).

Figure 1.

Example of an item of the updated vocabulary levels test.

Procedure

Due to the COVID-19 pandemic, the entire study was designed to be conducted via the Zoom platform. L2 students from eight intact classes were informed about the purpose of the study and asked to participate. They were also informed about the instructions for the activities. They received the links to the background questionnaire that also contained the informed consent and the online updated VLT. Approximately 1 month later, they received the link to the intervention, which consisted of the video clips and comprehension test. They were warned that some clips contained nonwords to prevent them from taking extra time trying to guess the meaning of the words. Each clip was played twice. After the first clip was played for the first time, the video was paused automatically, and all questions were displayed on the left part of the screen (Figure 2). At this point, participants could only read the questions but not answer them so that they could have an idea of what was going to be asked. The video was paused until the participants read all the questions and clicked “Continue.” Once the video was played for the second time, it was paused again, and the questions popped out one by one. Participants could not rewind or skip questions. When all the questions were answered, the video resumed. This process was repeated with each clip until they completed the comprehension test. Each playlist with the four clips plus the pauses lasted approximately 20 min. The participants could not fast-forward the video, and the video automatically stopped if they changed the tab or minimized the window. Participants were connected to Zoom throughout both sessions to ask questions, and their progress was tracked through the Playposit platform. They took, on average, 64 min as indicated on the platform.

Figure 2.

Example of an item of the comprehension test.

The procedure for the L1 participants was identical to that for the L2 participants, except that they did not take the updated VLT. Their consent form was embedded in the participants’ demographic information. On average, L1 speakers took 43 min and 42 s.

Scoring

All the tests were scored dichotomously. If an answer was correct, it was awarded 1 point and 0 if not. Both platforms, Google Forms, and Playposit, allowed for inputting the answer keys before participants took the test, and consequently, the test results could be downloaded in CSV format.

L1 speakers

Results from native speaker participants indicated that, even with 100% coverage, only six L1 participants obtained the maximum score of 14. The most recurring score (mode) was 12 points on the comprehension test, with 13 of 40 participants obtaining that score. Table 6 shows the number of participants in each comprehension score at each lexical coverage percentage, whereas Figure 3 shows the distribution of participants’ scores per lexical coverage level.

To answer whether lexical coverage affected the comprehension of L1 speakers, we ran the non-parametric Friedman test because the data was not normally distributed. The results showed that the degree of lexical coverage had a significant effect on comprehension, χ²(3) = 13.077, p = .004, w = .019, but with a small effect size. Pairwise comparisons were performed with a Bonferroni correction for multiple comparisons. Post hoc analysis revealed statistically significant differences in comprehension levels between 100% and 90% coverage (p = .044) and between 100% and 80% coverage (p = .019). All other comparisons were not statistically significant (Figure 4).

Figure 4.

Mean comprehension scores by L1 speakers on the four coverage levels with a Bonferroni correction.

L2 speakers

Results from non-native speaker participants indicated that, in the 100% coverage level, the most recurring score (mode) was 13 of 14 points on the comprehension test, with 20 of 76 participants obtaining that score. Most participants obtained between 12 and 14 correct answers. This indicates that with 100% lexical coverage, most participants were able to obtain between 85.7% and 100% correct answers on the comprehension test. In the 95% coverage, the highest scores were between 11 and 13. Comprehension trended downward in the other two lower levels. Table 7 shows the number of participants in each comprehension score at each lexical coverage percentage, whereas Figure 5 shows the distribution of participants’ scores per lexical coverage level.

Figure 5.

Non-native speakers’ distribution of results per lexical coverage level.

We also ran a Friedman test for the L2 data. The results showed that the degree of lexical coverage had a significant effect on comprehension, χ²(3) = 30.759, p <.001, w = .135, but with a small effect size. Pairwise comparisons were performed with a Bonferroni correction for multiple comparisons. Post hoc analysis revealed statistically significant differences in comprehension levels between 100% and 90% lexical coverage (p = .007), between 100% and 80% coverage (p <.001), and between 95% and 80% coverage (p = .015). All other comparisons were not significant (Figure 6).

Figure 6.

Mean comprehension scores by L2 speakers on the four coverage levels with a Bonferroni correction.

Because non-native speakers’ comprehension scores were surprisingly higher than native speakers, an additional statistical analysis was conducted to examine whether the difference was statistically significant. Comprehension scores were not normally distributed as assessed Shapiro-Wilk’s test (p <.05). Therefore, a Mann-Whitney U test was run. Distributions of the comprehension scores for native and non-native speakers were similar, as assessed by visual inspection. Results indicated that comprehension scores were not statistically significantly different between L1 and L2 speakers at the different levels of lexical coverage, 100%, U = 1372.5, z = –.873, p = .383; 95%, U = 1386, z = –.789, p = .430; 90%, U = 1479, z = –.241, p = .810; 80%, U = 1417.5, z = –.603, p = .546, using an exact sampling distribution for U (Dineen & Blakesley, Reference Dineen and Blakesley1973).

An additional analysis was carried out to compare the scores of the two different types of questions. Because the analysis is beyond the scope of this study, we did not include the results in this section. However, they can be found in Appendix B.

Considering the results from this study and following Laufer and Ravenhorst-Kalovski (Reference Laufer and Ravenhorst-Kalovski2010), the cutoff score for adequate comprehension was conceptualized as the point at which more learners are likely to comprehend well. In this study, the most recurrent score (mode) in the native speakers’ group at 100% lexical coverage (baseline) was 12 of 14 points. At this cutoff score, more than half of the native-speaker participants (25 of 40) had adequate comprehension. Regarding L2 participants, most learners received 13 at 100% coverage, but 12 is the point at which more than 50% of the participants reached adequate comprehension, altogether 51 of 76 participants. Additionally, it was observed that at 80% lexical coverage, the mean comprehension score was 10 of 14. At this point, a good proportion of participants could understand the clips. With a more lenient criterion, the threshold of adequate comprehension could also be set at 10 points, which is the lowest mean comprehension score of all the coverage percentages. In conclusion, we suggest two cutoff scores for viewing comprehension: one optimal reflected in the score of 12 and one minimal reflected in the score of 10 of 14 points.