Roles of vocabulary knowledge in real-life L2 listening proficiency

L2 listening is a complex and demanding task that requires a combination of linguistic and non-linguistic knowledge. Unlike L1 listening, where listeners can rely on automatic, subconscious, and effortless access to their native language system, L2 listening involves the use of L2 representations and processing skills that learners are gradually, partially, and concurrently developing (Vandergrift, Reference Vandergrift2007). For bottom-up processing, learners first need to detect prosodic structures in order to segment the speech stream into words (Cutler, Dahan, & Van Donselaar, Reference Cutler, Dahan and Van Donselaar1997). They further rely on segmental details of words to distinguish phonologically similar words, such as those with higher lexical density (e.g., Bradlow & Pisoni, Reference Bradlow and Pisoni1999) and minimal pairs (Munro & Derwing, Reference Munro and Derwing2006), and attend to the morphological attributes of each word to comprehend the grammatical patterns in a sentence. Additionally, L2 learners need to understand a speaker’s intention in accordance with conversational, societal, and cultural backgrounds (Taguchi, Reference Taguchi2011) and extralinguistic factors, such as vocal tone, facial cues, and bodily gestures (Kamiya, Reference Kamiya2022).

Regarding top-down processing, certain listeners can not only use a range of strategies, such as attending to the main points of L2 listening discourse, evaluating what they have understood versus what they have not, and inferring meaning from context, but also have high-level awareness of their own listening abilities and available strategies (Vandergrift & Goh, Reference Vandergrift and Goh2012). Successful L2 comprehension is greatly related to more precise auditory processing for better capturing of acoustic details of speech (Vandergrift & Baker, Reference Vandergrift and Baker2015) and greater working memory for holding and sustaining linguistic information for further linguistic analyses (Linck et al., Reference Linck, Hughes, Campbell, Silbert, Tare, Jackson and Doughty2013).

While all the abilities above are essential for the successful comprehension of L2 speech, ample studies have examined what factors account for the variances in global L2 listening proficiency, typically measured via high-stakes L2 listening proficiency tests (e.g., IELTS and TOEFL). These studies have consistently shown that learners’ vocabulary knowledge could be by far the most important correlate of successful L2 listening (r = .5–.7; see Zhang & Zhang, Reference Zhang and Zhang2022 for a meta-analysis) and that developing robust L2 vocabulary thus needs to be prioritized in an L2 listening syllabus (e.g., Stæhr, Reference Stæhr2009; Vafaee & Suzuki, Reference Vafaee and Suzuki2020; Vandergrift & Baker, Reference Vandergrift and Baker2015; Wallace, Reference Wallace2022).

When we look at the existing literature, such vocabulary knowledge has been exclusively assessed via tests designed to tap into learners’ form-meaning mappings, such as meaning recognition (Milton & Hopkins, Reference Milton and Hopkins2006; McLean et al., Reference McLean, Kramer and Beglar2015) and meaning recall (Cheng et al., Reference Cheng, Matthews, Lange and McLean2022; González-Fernández & Schmitt, Reference González-Fernández and Schmitt2020). According to Nation’s (Reference Nation2013) widely cited framework, word knowledge essential for successful L2 listening can be considered to involve not only recognizing the sound and meaning of target words (form-meaning mapping) but also understanding how these words interact with other words in semantically, collocationally, and grammatically appropriate ways (use-in-context). The former pertains to recognition of single or two-to-three words, while the latter highlights the ability to comprehend words on broader levels (e.g., clauses and sentences). While much discussion concerns the measurements and training of form-meaning mappings (e.g., recognition vs. recall), surprisingly little is known about how to define, measure, and train the use-in-context aspects of word knowledge (Schmitt, Reference Schmitt2019).

From a developmental perspective, the skill acquisition account of instructed L2 learning distinguishes three key types of knowledge: declarative, procedural, and automatized. These types reflect different stages of L2 phonological vocabulary learning, as outlined below:

• • Declarative knowledge involves learners’ explicit associations between word forms and meanings (knowing what). When a target word (or word strings) is presented in isolation, learners can recognize it in a list (McLean et al., Reference McLean, Kramer and Beglar2015) and/or recall its definition (Cheng et al., Reference Cheng, Matthews, Lange and McLean2022). However, this does not necessarily mean they can use the word appropriately in more global contexts (e.g., efficient retrieval of word meanings at the sentence level). The development of declarative knowledge can be achieved through explicit, form-focused, and controlled practice activities (e.g., multiple-choice quizzes, flashcards; Nakata et al., Reference Nakata, Tada, Mclean and Kim2021).

• • Procedural knowledge refers to the ability to use vocabulary in a context-appropriate manner without thinking about rules or meanings so explicitly (knowing how). It represents a gradual shift from more to less conscious and effortful processing in retrieving word meanings. Procedural knowledge develops through practice and involves using vocabulary in more extended, sentence-level contexts during listening (and speaking) activities (Masrai, Reference Masrai2020).

• • Automatized knowledge corresponds to the most advanced stage, where learners’ use of vocabulary (i.e., meaning retrieval) becomes fast, stable, and effortless (i.e., automaticity). Achieving automatized knowledge is often considered the ultimate goal, particularly in communicatively authentic listening (and speaking) contexts. With repetitive practice, and through the proceduralization of declarative knowledge via sustained exposure and engagement with target words in extended contexts, learners can eventually access this knowledge seamlessly in relation to surrounding words, forming chunks of linguistic constructions (Ellis, Reference Ellis2006).

By aligning Nation’s framework of word knowledge with skill acquisition theory, form-meaning mapping is associated with the declarative dimension of phonological vocabulary, while use-in-context corresponds to the procedural and automatized dimensions of phonological vocabulary. Since our conceptualization of vocabulary learning focuses on achieving advanced L2 listening proficiency, both the proceduralization and automatization of declarative knowledge involve the processing of vocabulary at the sentence level in this study. To avoid redundancy, we use “automatization” to encompass both proceduralization and automatization, indicating that the ultimate goal of instructed L2 learning is the automatization of proceduralized routines (Suzuki, Reference Suzuki2023). In the present investigation, we adopted these paradigms to develop assessment and training materials, employing meaning recognition tasks for the form-meaning/declarative dimension and lexicosemantic judgment tasks for the use-in-context/automatized dimension.

It is important to emphasize that neither Nation’s word knowledge framework nor the skill acquisition theory advocates discarding simple form-meaning mapping activities. Instead, these activities, which focus on memorizing patterns, are recognized as an essential initial step (for the suggested sequence of form-meaning and meaningful exposure, see Schmitt, Reference Schmitt2008 and Nation, Reference Nation2013). While form-meaning activities are integral and included in our current training paradigms (see the Method section), it is crucial to provide follow-up support that helps learners proceduralize and automatize the knowledge gained from these activities. This enables learners to access the knowledge more accurately, fluently, and subconsciously at more global sentence levels.

Measurement and training of automatized (vs. declarative) L2 knowledge

Within the skill acquisition framework, a range of single-modal tests have been adopted such as multiple-choice and fill-in-the-blanks, to assess participants’ declarative dimensions of morphosyntactic knowledge. In such tests, participants can fully focus on their abilities to comprehend and use only target structures (e.g., past tense markers, articles). In contrast, to assess automatized knowledge, dual-modal tests have been devised to examine how L2 learners can access the target structures accurately, promptly, and subconsiciously. In such test formats, participants’ specific linguistic knowledge is tested, while at the same time they are using language for meaning conveyance. One most oft-used test is grammaticality judgment tests (GJTs).

In GJTs, learners read or listen to sentences featuring manipulated target morphosyntactic structures and must quickly and intuitively decide the correctness of these sentences. While their main test is to grasp the overall meaning of the sentences (i.e., reading or listening processing), they simultaneously need to check whether any grammatical features are used correctly or incorrectly in any parts of the sentences (i.e., morphosyntax processing; see Plonsky et al., Reference Plonsky, Marsden, Crowther, Gass and Spinner2020 for a review). Ample evidence shows that when L2 learners take metalinguistic tests and GJTs, their test scores cluster into two different factors. This suggests that metalinguistic tests and GJTs measure fundamentally different aspects of L2 morphosyntactic proficiency, with GJTs being assumed to capture automatized knowledge and metalinguistic tests reflecting declarative knowledge (Ellis, Reference Ellis2005; Gutiérrez, Reference Gutiérrez2013).

While highly form-oriented instruction impacts only declarative knowledge, L2 learners can attain both declarative and automatized knowledge when they learn morphosyntactic accuracy from form-oriented lessons and also practice it repetitively via meaning-oriented instruction (Spada & Tomita, Reference Spada and Tomita2010). Additionally, performance on GJTs has been shown to correlate with key variables affecting high-level L2 proficiency, such as age of acquisition (Abrahamsson & Hyltenstam, Reference Abrahamsson and Hyltenstam2009) and length of immersion (Faretta-Stutenberg & Morgan-Short, Reference Faretta-Stutenberg and Morgan-Short2018).

According to Suzuki and Elgort’s (Reference Suzuki, Elgort and Suzuki2023) comprehensive review on automatized dimensions of various linguistic skills and training effects, much has been documented about L2 morphosyntax acquisition and about L2 phonetic acquisition to a lesser degree (cf. Saito & Plonsky, Reference Saito and Plonsky2019). However, very few studies have examined the automatized dimensions of phonological vocabulary knowledge (the main focus of the current investigation).

Elgort (Reference Elgort2011) investigated how learners can automatize their knowledge of pseudowords through intentional training involving meaning recognition and feedback. The study used lexical decision tests in which participants were shown strings of letters and asked to make an intuitive judgment about whether each string was a word or a non-word. The results showed that participants could recognize target words more quickly and accurately after being exposed to similar forms with related semantic meanings. These findings highlight the effectiveness of deliberate learning strategies in enhancing automatic lexical processing. For insights into the incidental approach to automaticity in lexical knowledge, see Elgort and Warren (Reference Elgort and Warren2014).

Foster et al. (Reference Foster, Bolibaugh and Kotula2014) employed a unique approach to examining L2 learners’ highly advanced, automatized collocational knowledge. A total of 160 experienced Polish users of English (in the UK and Poland) were asked to read narratives and highlight non-nativelike word combinations embedded in the text (e.g., “a young man was strolling his way in the street”). The methodological rationale was that given that advanced L2 learners are assumed to store a range of multiword units as lexical chunks, the degree of nativelikeness in their collocational knowledge can be clearly observed when these word combinations deviate from what they are accustomed to. The results indicated that more advanced L2 participants, particularly those with an earlier age of arrival, demonstrated greater accuracy in identifying non-nativelike word selections.

Building on the methodological paradigm underlying GJTs, scholars have introduced the LJT to assess automatized phonological vocabulary knowledge (Saito et al., Reference Saito, Uchihara, Takizawa and Suzukida2023; Uchihara et al., Reference Uchihara, Saito, Kurokawa, Takizawa and Suzukida2024). In the LJT, learners are aurally presented with grammatically correct sentences composed solely of high-frequency words. These sentences are categorized either “semantically appropriate” or “semantically inappropriate” based on the use of a target word (e.g., “estate”). Some sentences use the target word appropriately (e.g., “My grandfather bought an estate”), while others use it in a semantically incongruous manner (e.g., “My friend’s estate was very kind”). Learners are asked to judge the semantic appropriateness of each sentence upon hearing it, with limited time to deliberate on the correct or incorrect use of the word.

In previous projects, over 400 Japanese L2 learners of English with varied experience and proficiency profiles took a general L2 listening proficiency test (TOEIC) as well as a set of tests designed to measure both declarative (meaning recognition and meaning recall) and automatized dimensions of phonological vocabulary knowledge (LJT). The results showed that their vocabulary test scores were factored into two latent variables (meaning recognition and recall being distinguishable from LJT). This led us to speculate (a) that LJT taps into a construct of lexical knowledge fundamentally different from declarative phonological knowledge (typically measured via meaning recognition and recall) and (b) that the construct could be the automatized dimension of phonological knowledge (Uchihara et al., Reference Uchihara, Saito, Kurokawa, Takizawa and Suzukida2024).

Furthermore, the results also indicated that the learner factors predicting the attainment of L2 phonological vocabulary knowledge differed depending on the test formats. Those with more extensive L2 learning experience in classroom settings and greater working memory likely showed greater declarative phonological vocabulary knowledge (assessed via meaning recognition). In contrast, those who engaged in more extracurricular activities and exposure to communicatively authentic materials beyond classrooms (e.g., study abroad, viewing, and conversations) likely showed greater automatized vocabulary knowledge (assessed via LJT; Saito & Uchihara, Reference Saito and Uchihara2024). Finally, global L2 listening proficiency was more strongly predicted by LJT scores (r = .6–.7) than by scores for meaning recognition and recall (r = .4–.5; Saito et al., Reference Saito, Uchihara, Takizawa and Suzukida2023).

The accumulated cross-sectional evidence has thus far suggested that there are two overlapping but essentially different dimensions of phonological vocabulary knowledge: form-meaning mappings and use-in-context. These dimensions can be measured via declarative vocabulary tests (meaning recognition) and automatized vocabulary tests (LJT). From the skill acquisition perspective, vocabulary learning progresses from explicit, rule-based knowledge (declarative) to fluent, context-appropriate use (procedural), and ultimately to automatic, effortless application (automatized) as learners practice and reinforce their skills (DeKeyser, Reference DeKeyser2017; Suzuki, Reference Suzuki2023). Given the relative importance of vocabulary knowledge in global L2 listening proficiency (Wallace, Reference Wallace2022), the attainment of automatized vocabulary knowledge can help learners access words more spontaneously in real-life contexts, thereby enhancing their global L2 listening proficiency in the long run. By conducting an intervention study with a pre- and post-test design, the current study took the first step toward testing this hypothesis.

Enhancing global listening proficiency via vocabulary training

To help learners attain L2 listening proficiency, numerous studies have focused on top-down approaches via metacognitive strategy training. In this training, students are encouraged to engage in strategic planning prior to listening through an awareness of their available strategies, self-monitoring during listening, and reflecting on post-listening outcomes (e.g., Kobayashi, Reference Kobayashi2018; Milliner & Dimoski, Reference Milliner and Dimoski2024; Yeldham, Reference Yeldham2022). A meta-analysis by Dalman and Plonsky (Reference Dalman and Plonsky2022) has confirmed that such studies can eventually impact L2 learners’ global listening proficiency with medium effects (d = 0.69). Surprisingly, very few studies have explored how teaching phonological forms of a set of key vocabulary items can first impact vocabulary proficiency and then global L2 listening proficiency.

Zhang and Graham (Reference Zhang and Graham2020) provided different types of phonological vocabulary training (e.g., explanations in L1 vs. L2) to a total of 137 young Chinese learners of English over the course of six weeks. The pre- and post-tests showed significant enhancement in their form-meaning mapping of the 60 target items introduced during the sessions (measured via meaning recall). Similarly, Uchihara et al. (Reference Uchihara, Webb, Saito and Trofimovich2022) provided one hour of phonological vocabulary training on 40 target items to 80 Japanese college students of English. The pre- and post-tests (via picture naming test) demonstrated significant gains in form recall and prosodic accuracy after a brief training session. However, neither of these studies explored how such vocabulary gains ultimately impact L2 listening proficiency.

In Saito and Akiyama’s (Reference Saito and Akiyama2018) longitudinal investigation, Japanese college students of English engaged in individual interaction with native interlocutors over the course of 12 weeks. Whenever any linguistic errors (including lexical ones) interfered with overall understanding, the native interlocutors provided corrective feedback to L2 participants’ erroneous utterances and encouraged them to self-correct during conversational interactions with the aim of more easy-to-understand speech (i.e., negotiation for comprehensibility). The results showed that not only did the participants’ L2 speech become more comprehensible and intelligible, but their global L2 listening proficiency (measured via TOEIC) also improved with large effects. Yet, the study did not isolate and link the participants’ vocabulary development to their L2 listening proficiency development. The current study corresponded to these methodological concerns.

Target words

For the pre- and post-tests and training, a total of 80 target words were chosen to reflect real-life L2 vocabulary use. A speech corpus was developed using scripts from the TOEIC Listening test (Version B) for the post-tests, as the TOEIC test includes various forms of L2 discourse such as short sentences, conversations, and monologues. From this corpus, which contained 2,731 tokens, the top 80 words posing the most phonological challenges for Japanese EFL learners were carefully selected. These words were chosen based on their lower frequency profiles according to the BNC/COCA word family lists (Nation, Reference Nation2012). Priority was given to words that Japanese learners might find difficult, including iambic words with multiple syllables, challenging segmentals like English [r] and [l], and consonant clusters (Saito, Reference Saito2014). Additionally, loanwords were excluded as they could potentially facilitate L2 understanding (Uchihara et al., Reference Uchihara, Webb, Saito and Trofimovich2022).

Outcome measures

In the current project, participants practiced on the 80 target words (derived from TOEIC Test Version B at post-tests). Whereas the impact of vocabulary training on participants’ general L2 listening proficiency was evaluated based on their performance on TOEIC Test Version B, their pre-existing L2 listening proficiency was measured in a different version of TOEIC (Version A) at pre-tests. To examine the extent to which participants had enhanced the declarative and automatized dimensions of the target words, they took both meaning recognition tests (MRTs) and LJTs. Since the participants took the same format of the test for global L2 listening proficiency (TOEIC) and the same materials for the vocabulary tests (MRT and LJT) at pre- and post-tests, the performance of the comparison group served the presence of test–retest effects if any. The participants took the tests in the following order—(1) general L2 listening proficiency test (TOEIC), (2) LJT, and (3) MRT. All the test instructions were prepared in Japanese to facilitate participants’ understanding of the procedures. To explain both methodological and conceptual details of the tests in a logically reasonable manner, the tests were described as follows in the following order: (3) MRT → (2) LJT → (1) TOEIC. The test materials are deposited in L2 Speech Tools (Mora-Plaza et al., Reference Mora-Plaza, Saito, Suzukida, Dewaele and Tierney2022). The online versions of the tests are available as open materials in Gorilla and shared in Supporting Information S1.

Meaning recognition test

As operationalized in existing studies (e.g., McLean et al., Reference McLean, Kramer and Beglar2015), the declarative dimension of participants’ phonological vocabulary knowledge (i.e., form-meaning mapping of the target words) was measured via two different audio versions of MRT. One key aspect of form-meaning mapping in L2 speech development concerns generalization. Once learners become capable of recognizing words within training, they are believed to start generalizing their phonological representations from the contexts of trained talkers to other talker contexts (i.e., more real-life L2 listening contexts; Zhang, Cheng, & Zhang, Reference Zhang, Cheng and Zhang2021). To capture the robustness of generalization, participants took two versions of MRT, one featuring a trained voice (a male of General American; Talker A) and the other featuring an untrained voice (a female of General American; Talker B). A total of 160 trials were featured in MRT (n = 80 target words produced by Talkers A and B, respectively). In the Gorilla platform, the 160 stimuli were played in a randomized order. As shown in Figure 1, as soon as they listened to a target word, the participants were asked to choose the correct meaning from four options (one correct answer and three distractors). All four answer options were presented in Japanese, and they served the same part of speech. The distractors were selected from a list of words frequently found in TOEIC test materials. They were also matched in terms of same parts of speech. Total scores were calculated out of a maximum of 160 points. The pre- and post-test MRT materials demonstrated high reliability, α = .93 and .94, respectively.

Figure 1.

Screenshot of the meaning recognition test (in Japanese). Participants listened to a target word without spelling (e.g., “appliance”) and selected its meaning from four different choices (e.g., “領収書” [receipt], “電化製品” [appliance], “運送会社” [carriers], “開催地” [venue]).

Lexicosemantic judgment test

Following the precursor projects on the development and validation of the test designed to assess the automatized dimension of phonological vocabulary knowledge (Saito et al., Reference Saito, Uchihara, Takizawa and Suzukida2023; Uchihara et al., Reference Uchihara, Saito, Kurokawa, Takizawa and Suzukida2024), the same LJT was adopted in the current study. A total of 160 short sentences were prepared. Half of the sentences used the target words in a semantically appropriate manner, while the other half used them in a semantically inappropriate manner. For instance, for the word “estate,” participants heard “My grandfather bought an estate” (appropriate) and “My friend’s estate was very kind” (inappropriate). All other parts of the sentences were clear and easy to understand (comprising 1K word families)

The sentences were kept brief, ranging from 3 to 8 words, to minimize the strain on participants’ working memory during the test. To ensure that the test did not mainly evaluate syntactic knowledge or speech perception skills, we used simple syntax without complex subordination. Given that syntactic position can influence the salience of unfamiliar words, efforts were made to minimize such priming effects. First, we avoided placing target words at the beginning of sentences, as words appearing at the beginning or end of a sentence are typically more salient than those embedded in the middle (Moravcsik & Healy, Reference Moravcsik and Healy1998). Second, we avoided using subordinate clauses, as syntactic complexity affects the salience of words; words in main clauses are generally more salient than those in subordinate clauses (McKoon et al., Reference McKoon, Ward, Ratcliff and Sproat1993). Finally, all sentences were simple (without phrase boundaries) to prevent syntactic complexity from reducing word salience, as complex structures can lead to shorter eye fixations or cause words to be skipped altogether during sentence processing (Birch & Rayner, Reference Birch and Rayner2010). After drafting the sentences, three English experts reviewed them to ensure they conveyed the intended semantic appropriateness or inappropriateness. Participants listened to 160 short sentences spoken by the untrained talker (Talker B).

After hearing each sentence, they were asked to determine whether it was “semantically appropriate” or “semantically inappropriate” based on a single word (for a screenshot of the test, see Figure 2). To ensure participants listened to the entire sentence, the target word was placed somewhere other than at the beginning. The sentences were kept simple, ranging from 4 to 8 words in length, and were always grammatically correct. Most of the words (93%) came from the 1,000 most common words in English. As a measure of automatized vocabulary knowledge, participants were tested on their ability to process the conceptually appropriate (or inappropriate) use of target words produced by the new talker (Talker B) while at the same time processing other aspects of language on a sentence level (e.g., phonological, morphosyntactic, and pragmatic processing). The 160 sentences were then played to participants in a random order. Participants earned 1 point for each correctly judged sentence, with a maximum possible score of 160 points. The pre- and post-test LJT materials demonstrated high reliability, α = .92 and .90, respectively.

Figure 2.

Screenshot of the lexicosemantic judgement test (in Japanese). Participants listened to a sentence with an embedded target word (without captions) and determined whether the sentence was semantically appropriate or inappropriate.

In the current investigation, consistent with the earlier validation studies demonstrating the strong predictive power of the LJT for L2 listening proficiency (Saito et al., Reference Saito, Uchihara, Takizawa and Suzukida2023; Uchihara et al., Reference Uchihara, Saito, Kurokawa, Takizawa and Suzukida2024), we opted to use the untimed version of the LJT. Participants were informed that the test assessed both accurate and fluent processing of L2 English sentences, and they were encouraged to respond as quickly as possible, though no time limit was imposed for each trial. This approach was justified on both empirical and theoretical grounds.

Turning to the existing literature on L2 morphosyntax, researchers generally acknowledge that acceptability judgment tasks (GJTs) can measure the degree of automatization in L2 morphosyntactic knowledge. However, the role of time pressure remains a subject of debate. Some scholars advocate for timed tasks to better access automatized, potentially implicit, L2 knowledge by minimizing explicit, controlled, and monitored processing (e.g., Ellis, Reference Ellis2005; Gutiérrez, Reference Gutiérrez2013). However, Plonsky et al.’s (Reference Plonsky, Marsden, Crowther, Gass and Spinner2020) comprehensive review of GJT methodologies highlights that the effects of time pressure are particularly evident in written formats, where learners can revisit texts multiple times, which may lead to more deliberate, controlled processing. In contrast, such effects are not deemed necessary in auditory tasks, where participants hear a stimulus only once. The fleeting, continuous nature of auditory tasks inherently simulates real-life, relatively automatic language processing, reducing the need for external time constraints. As Plonsky et al. (Reference Plonsky, Marsden, Crowther, Gass and Spinner2020) rightly pointed out, “by definition, aural language happens at a fixed pace for all participants, so is, in a sense ‘timed’, and so, generally with these comparisons between aural and written, ‘modality’ is inevitably conflated with timing” (p. 598).

It is important to note that existing research suggests time pressure may not reliably distinguish between controlled and automatic processing across all learners, raising concerns about the use of time-limited tasks as a sole measure of automatization. For instance, Hulstijn et al. (Reference Hulstijn, Van Gelderen and Schoonen2009) argue that faster reaction times under time pressure do not necessarily indicate automatization but could instead reflect more efficient retrieval without full automaticity. This makes it challenging to differentiate true automatization from mere speed enhancement even if both reaction time and coefficient of variation are considered. Additionally, Maie and Godfroid (Reference Maie and Godfroid2022) demonstrated that time pressure affects both controlled and automatic processes, particularly for L2 learners with slower lexical decoding and processing speeds, complicating its use as a reliable measure of automatization. Learners with processing difficulties may respond more slowly regardless of time pressure. Their eye-tracking study showed that time pressure might be more closely related to participants’ lexical decoding skills and may not function as clearly as previously thought in distinguishing controlled from automatic processes, especially for learners with varying cognitive profiles.

To prepare for the current investigation and to strengthen the construct validity of LJT as a measure of automatized phonological vocabulary knowledge, we conducted an additional study to further examine the role of time pressure (Saito et al., Reference Saito, Hosaka, Macmillan, Takizawa, Suzukida and Uchiharaforthcoming). While many GJT studies ambiguously set time limits based on native speaker norms, we carefully established an optimal response speed based on the LJT performance of advanced Japanese learners of English (CEFR = C1, C2). Although considerably slower than native speakers (600–700 ms), these advanced L2 learners were considered to have fully automatized phonological vocabulary knowledge, and their processing speed (2000–2400 ms) served as a benchmark for the university-level Japanese students in this study, who had relatively basic-to-intermediate L2 proficiency levels (CEFR = A2, B1, B2). Our findings indicated that participants’ timed and untimed LJT scores similarly predicted their L2 listening proficiency (r = .64 and .73, respectively), with the difference in correlation coefficients failing to reach statistical significance (z = 1.081, p = .140). This lack of significant time pressure effects suggests that the construct of LJT as a measure of automatized phonological vocabulary knowledge is not substantially influenced by time constraints.

In essence, we find the timed approach problematic because enforcing a fixed time limit may disadvantage learners with weaker lexical decoding speeds (Maie & Godfroid, Reference Maie and Godfroid2022) and may not accurately reflect the complex processes underlying the automatization of L2 knowledge, where the development of accuracy does not necessarily align with the development of fast and stable processing skills (Hulstijn et al., Reference Hulstijn, Van Gelderen and Schoonen2009). Therefore, we emphasized to participants the importance of both accuracy and fluency in the LJT, encouraging them to respond as quickly as possible without imposing specific time limits. To ensure participants adhered to these instructions and to identify those who might have spent excessive time responding to each stimulus, we defined slow outliers as individuals with an average reaction time of 6000 ms or more. This threshold was calculated as 1.5 standard deviations beyond the mean reaction time among the similar population—that is, n = 126 Japanese EFL participants in Saito et al. (Reference Saito, Uchihara, Takizawa and Suzukida2023). None of the participants reached this threshold, and thus no data were excluded on this basis.

Training

The participants were randomly assigned to one of the three training conditions—Automatization, Declarative, and Comparison. The training was implemented via Gorilla with a 24-hour timer. If participants did not complete each session within 24 hours, they would be eliminated from the project. As such, the timing they can access the training was strictly controlled by the researchers. The length/number of training sessions (30-minute × 6 sessions) was determined in accordance with previous similar studies (e.g., Elgort, Reference Elgort2011). To facilitate participants’ understanding of the training procedure, a Japanese brochure was provided prior to the training, all the training instructions were displayed in Japanese, and research assistants were on standby to respond to any queries in Japanese. The online versions of the training materials are available as open materials in Gorilla and shared in Supporting Information S2. The content of training for the experimental groups (Automatization and Declarative) was summarized in Table 1 and detailed as follows.

Table 1.

Summary of automatization and declarative training

Note: MRT for meaning recognition task; LJT for lexicosemantic judgement task; mild for 10.5 sec of lenient timeout; mild for 4.5 sec of timeout.

Automatization group

For the automatization group, the training program consisted of a total of six 30-minute sessions, with the first three sessions devoted to the development of declarative knowledge (Sessions 1–3) and the latter three sessions devoted to the development of automatized knowledge (Sessions 4–6).

Between Sessions 1 and 3, learners initially worked on form-meaning mappings of linguistic items. Following the notion of the skill acquisition theory for instructed L2 acquisition (DeKeyser, Reference DeKeyser2017; Suzuki, Reference Suzuki2023), this process was promoted under a single-task condition wherein they could fully focus on target areas of linguistic processing without much attention to other aspects of language. As operationalized in previous studies (Zhang & Graham, Reference Zhang and Graham2020), participants engaged in meaning recognition tasks to learn the 80 target words. Unlike the MRT in the pre- and post-tests, each trial in the training sessions was followed by feedback (see Figure 3).

Figure 3.

Screenshot of feedback in the meaning recognition training (in Japanese). The feedback displays the spelling of the target word (e.g., “appliance”) and its L1 translation (e.g., “電化製品”).

Participants completed two rounds of MRT within each session, with certain time thresholds. In Session 1, participants engaged in the first round of MRT without any time limits, allowing them to take as much time as needed (i.e., no timeout). However, during the second round, a lenient timeout of 10.5 seconds was applied. In Sessions 2 and 3, participants again completed the first round of MRT without time constraints, followed by a second round with the lenient timeout (10.5 seconds) and a third round with a mild timeout (4.5 seconds).

To prepare for the current project, we conducted a series of pilot training sessions involving five experienced Japanese teachers of English, who engaged in meaning recognition and lexicosemantic judgment tasks under various timeout conditions. This helped us determine the optimal time windows for the participants in the current study (CEFR: A2, B1, B2). The purpose of implementing timeouts was to prevent L2 learners from spending excessive time on each trial, ensuring that each session could be completed within 30 minutes without undue delay or memory burden. Additionally, the timeouts aimed to facilitate a gradual shift from focusing solely on accuracy to increasing attention to fluency, thereby encouraging the proceduralization of any partially acquired vocabulary knowledge. Importantly, the goal of introducing timeouts was not to impose time pressure, as we consider the effects of time pressure problematic, particularly for cognitively diverse L2 learners. This rationale aligns with our adoption of an untimed approach, as previously justified.

To this end, we established 10.5 seconds as the lenient timeout and 4.5 seconds as the mild timeout. For the lenient timeout, our assumption was that none of the participants would fully use the 10.5 seconds; rather, the presence of the timeout would give them sufficient time to process each item, while also fostering an awareness that taking more than 10 seconds per item might not be ideal. Regarding the mild timeout, we assumed that 4.5 seconds would provide just enough time for participants to process each stimulus, instilling a slight sense of urgency to prioritize fluency alongside accuracy. Notably, this timeout approach differs from traditional time pressure methods. We ensured that participants (CEFR: A2, B1, B2) would not feel pressured by time constraints during the training but would be encouraged to balance accuracy and fluency in their phonological vocabulary knowledge and processing.

After participants established robust form-meaning mappings of target words, they proceeded to the automatization phase (Sessions 4–6). In each 30-minute session, the participants focused on the use-in-context aspects of the 80 learned vocabulary items via one round of lexicosemantic judgment tasks followed by feedback (160 trials). As stated in the skill acquisition theory (DeKeyser, Reference DeKeyser2017; Suzuki, Reference Suzuki2023), automatization can be enhanced when learners engage in such dual-task conditions. During the LJT, learners worked on accessing the target lexical items as a primary focus while using all other linguistic information (grammar, collocational, and pragmatic processing) at a sentence level. Each trial was followed by feedback (see Figure 4). Unlike MRT, participants did not have any timeouts.

Figure 4.

Screenshot of feedback in the lexicosemantic judgement training (in Japanese). The feedback indicates whether the sentence was semantically appropriate or inappropriate (e.g., “不適切” [inappropriate]) and provides the L1 translation for the target word in the sentence (e.g., “賞賛の言葉” [praise]).

Declarative group

The participants worked only on the form-meaning mappings of the target words via MRT for all the sessions. In Session 1, they completed two rounds of MRT (without timeouts and with mild timeouts). In Sessions 2 to 6, they completed three rounds of MRT (with no, mild, and strict timeouts).

Comparison group

The purpose of the comparison group was to determine if there was any test–retest effect resulting from taking the tests twice (MRT, LJT, and TOEIC). As developed and piloted in another precursor project (Kachlicka et al., Reference Kachlicka, Symons, Ruan, Saito, Dick and Tierneyforthcoming), participants engaged in 30 minutes of adaptive vocabulary training. The training comprised picture matching tasks wherein participants read a target word prompt and chose a matching one out of four pictures. Word prompts and corresponding (and distractor) pictures were randomly selected from the word and picture banks. There were a total of 60 words randomly selected from each of the five vocabulary bands (n = 12 from 2000-, 3000-, 5000-, 10000-word-frequency families, and the academic vocabulary list, respectively) as set up in the BNC-COCA corpus in Vocab Profiler. Within each session (30 minutes), participants worked on two rounds of picture matching of 60 words.