Introduction
Since the development of corpus linguistics as a methodology in the 1990s, a large body of research has emerged concerning formulaic language, such as idioms (e.g., Carrol & Conklin, Reference Carrol and Conklin2017; Siyanova-Chanturia et al., Reference Siyanova-Chanturia, Conklin and Schmitt2011a), binomials (e.g., Du et al., Reference Du, Elgort and Siyanova-Chanturia2021; Siyanova-Chanturia et al., Reference Siyanova-Chanturia, Conklin and van Heuven2011b), and collocations (e.g., Jeong & DeKeyser, Reference Jeong and DeKeyser2023; Yamashita & Jiang, Reference Yamashita and Jiang2010). A collocation is a string of words, for instance adjective-noun (e.g., deep love) or verb (preposition) noun combinations (e.g., drop [the] bombs), that co-occur more routinely than probability would predict, and could be considered as an existing entity beyond the component words (Manning & Schütze, Reference Manning and Schütze1999). Knowledge of which words are most likely to occur together, or collocate, has long been promoted as a vital component of successful language learning (e.g., Lewis, Reference Lewis1993; Pawley & Syder, Reference Pawley, Syder, Richards and Schmidt1983), whereby understanding of and familiarity with frequent collocations enables more efficient processing, as attested to by faster reading times (e.g., Jiang et al., Reference Jiang, Jiang and Siyanova-Chanturia2020; Wolter & Yamashita, Reference Wolter and Yamashita2018; Yi, Reference Yi2018), freeing cognitive resources for top-down comprehension processes (Wray, Reference Wray2002).
Despite such benefits, research has consistently demonstrated that even advanced L2 learners exhibit weaker collocational knowledge compared to L1 users (e.g., Henriksen, Reference Henriksen, Bardel, Lindqvist and Laufer2013; Henriksen & Stær, Reference Henriksen, Stæhr, Barfield and Gyllstad2009; Nesselhauf, Reference Nesselhauf2003; Siyanova & Schmitt, Reference Siyanova and Schmitt2008). Furthermore, research has also indicated the existence of a collocation congruency effect, whereby L2 learners process congruent collocations—those with a translation shared between their L1 and L2 at the lexical, semantic, syntactic, or structural level depending on the study—more rapidly and accurately than incongruent ones, with this effect diminishing as proficiency increases (e.g., Sonbul et al., Reference Sonbul, El-Dakhs and Alharbi2024; Terai et al., Reference Terai, Fukuta and Tamura2024; Wolter & Gyllstad, Reference Wolter and Gyllstad2011, Reference Wolter and Gyllstad2013; Wolter & Yamashita, Reference Wolter and Yamashita2018; Yamashita & Jiang, Reference Yamashita and Jiang2010). For instance, a Japanese learner of English with relatively low proficiency would be predicted to process strong wind [強い風: tsuyoi kaze] faster than bitter wind [苦い風: nigai kaze] because the former is congruent while the latter is not.
Considering that collocational knowledge is a prerequisite for advanced language learning, and also that the collocation congruency effect must be carefully controlled for in applied psycholinguistic experiments, identifying congruent and incongruent collocations constitutes a critical methodological procedure that should be conducted in a principled manner. To identify (in)congruent collocations, researchers have typically relied on three methods. First, in many studies collocational congruency has been judged by one, or at most a small number of expert L1 users (e.g., Gyllstad & Wolter, Reference Gyllstad and Wolter2016; Takizawa, Reference Takizawa2024; Wolter & Gyllstad, Reference Wolter and Gyllstad2013; Yamashita & Jiang, Reference Yamashita and Jiang2010). Second, researchers have occasionally employed corpus-derived frequency for congruency decisions, whereby incongruent collocations are those with zero occurrences in a target language corpus (e.g., Jeong & DeKeyser, Reference Jeong and DeKeyser2023; Öksüz et al., Reference Öksüz, Brezina and Rebuschat2021). Finally, relatively large participant groups consisting of target language L1 users have been recruited for norming studies, where rating scales are employed to assess congruency status (e.g., Ma & Hong, Reference Ma and Hong2026; Wolter & Yamashita, Reference Wolter and Yamashita2018). However, the binary judgments on congruency status elicited by these three methods are somewhat subjective because what is considered a congruent collocation in one study might be classified as incongruent in another. Furthermore, there is no research to date that systematically compares the effects of these three methods for defining collocational congruency; thus, the present study attempts to address this gap.
Literature review
Processing congruent and incongruent collocations
Collocation researchers have reported a number of robust effects relating to qualities that enhance collocation processing speed in psycholinguistic tasks, such as frequency (Gyllstad & Wolter, Reference Gyllstad and Wolter2016; Sonbul, Reference Sonbul2015), word association strength (Öksüz et al., Reference Öksüz, Brezina and Rebuschat2021; Yi, Reference Yi2018), and collocational congruency effects (e.g., Sonbul et al., Reference Sonbul, El-Dakhs and Alharbi2024; Terai et al., Reference Terai, Fukuta and Tamura2024; Wolter & Gyllstad, Reference Wolter and Gyllstad2011, Reference Wolter and Gyllstad2013; Wolter & Yamashita, Reference Wolter and Yamashita2018; Yamashita & Jiang, Reference Yamashita and Jiang2010), the latter of which constitute the focus of the present study. Research conducted with L2 English learners indicates that incongruent collocations are processed more slowly and are responded to with less accuracy in decision tasks. In a pioneering study, Yamashita and Jiang (Reference Yamashita and Jiang2010) presented L1 English speakers, advanced Japanese English as a second language (ESL) students, and intermediate Japanese English as a foreign language (EFL) students with 24 congruent and 24 incongruent collocations in a phrasal judgment task (PJT), which required participants to respond Yes or No depending on whether they believed the target collocation to be an acceptable English phrase. While the L1 English speakers displayed no significant difference between collocation type in terms of reaction times (RT) speed and errors, the ESL learners produced more errors with incongruent collocations (although the RTs were comparable) and the EFL learners RTs were both slower and more liable to errors with incongruent phrases.
Yamashita and Jiang’s initial study prompted several follow-up studies (e.g., Wolter & Gyllstad, Reference Wolter and Gyllstad2013; Wolter & Yamashita, Reference Wolter and Yamashita2015), with each subsequent study refining the methodology. For instance, while Yamashita and Jiang’s (Reference Yamashita and Jiang2010) study used two types of adjective-noun (e.g., lucky winner) and verb-noun (e.g., drink soup) collocations, subsequent studies focused on one or the other (e.g., Chen, Reference Chen2024; Jeong & DeKeyser, Reference Jeong and DeKeyser2023; Wolter & Yamashita, Reference Wolter and Yamashita2018). Regardless of the methodology, these studies all found evidence of a congruency effect. Wolter and Yamashita (Reference Wolter and Yamashita2018), for example, conducted a PJT requiring participants to respond to 24 items in each of three conditions consisting of English-only, Japanese-only, and congruent collocations, along with 72 baseline word pairs. The 24 intermediate Japanese L2 English users in the study displayed a processing advantage when responding to congruent collocations. However, unlike Yamashita and Jiang (Reference Yamashita and Jiang2010), the 23 advanced learners in Wolter and Yamashita’s study also displayed a congruency advantage. The authors suggested that the discrepancy was due to the latter study being more tightly controlled, involving no repeated target words and an even number of collocation types.
While Yamashita and Jiang (Reference Yamashita and Jiang2010) and Wolter and Yamashita (Reference Wolter and Yamashita2018) found evidence of the congruency effect with L1 Japanese L2 English users, a similar congruency advantage in terms of processing speed was also observed with participants from other linguistic backgrounds, such as Chinese (Chen, Reference Chen2024) and Swedish (Wolter & Gyllstad, Reference Wolter and Gyllstad2011). For example, Chen (Reference Chen2024) presented 40 congruent and 40 incongruent adjective-noun collocations to 33 L1 Chinese ESL learners and 33 L1 Chinese EFL learners in a judgment task. In line with Yamashita and Jiang (Reference Yamashita and Jiang2010), the EFL learners displayed a congruency effect in terms of both speed and accuracy, while the ESL learners, who possessed greater L2 exposure, displayed a congruency effect in accuracy only. These results suggest that the congruency effect weakens as proficiency increases and learners develop more robust L2 collocation representations, reducing reliance on L1 mediation. In another study, Wolter and Gyllstad (Reference Wolter and Gyllstad2013) recruited 25 L1 Swedish L2 English speakers who displayed a congruency effect in a judgment task involving adjective-noun collocations. The authors attributed the finding to age of acquisition and order of acquisition effects, whereby existing L1 concepts for the congruent collocations accelerate acquisition of the L2 equivalents resulting in faster processing. In contrast to these findings, in an eye-tracking experiment involving 31 Arabic EFL learners, El-Dakhs et al. (Reference El-Dakhs, Sonbul and Masrai2024) failed to find a significant difference between reading times for congruent and incongruent collocations. This result might be due to the collocations being presented in sentences, offering a more ecologically valid representation of online reading processes than judgment tasks.
When synthesized, the research reviewed above suggests that L2 learners are generally slower when processing incongruent collocations than congruent ones in judgment tasks, but the effect gradually deteriorates as proficiency increases. However, the congruency effect is potentially task dependent because eye-tracking, which offers a more naturalistic representation of reading than judgment tasks, resulted in a negligible difference in reading times for Arabic learners.
Approaches to defining collocational congruency
If incongruent collocations influence RTs in psycholinguistic experiments, questions arise regarding how such phrases should be identified. In L2 collocation processing research, congruency has been determined via one of three approaches: expert judgment, corpus-derived frequency, or norming study results. In numerous studies, researchers have tended to treat congruency as a binary variable based on L1 speakers’ judgment. For instance, in Wolter and Gyllstad’s studies (Reference Wolter and Gyllstad2011, Reference Wolter and Gyllstad2013), congruency was determined by one researcher, who is an L1 Swedish user and highly proficient L2 English user. In other similar studies, Gyllstad and Wolter (Reference Gyllstad and Wolter2016) consulted two experts to determine the congruency of their target collocations and interrater reliability, κ = .802, was measured, while in Yamashita and Jiang’s (Reference Yamashita and Jiang2010) study, which involved Japanese learners, the initial decision was determined by one researcher, an L1 Japanese user and highly proficient L2 English user, and then supported via analysis of translations from five L1-Japanese users with high L2-English proficiency. Five raters were also recruited by Li et al. (Reference Li, Wolter, Yang and Siyanova-Chanturia2025) to support initial expert judgments on items for use in an acceptability task. The raters comprised L1 Chinese PhD students who were asked to judge the congruency of 200 collocations and 200 baseline items constructed from the collocations’ component words. However, instead of requiring a binary decision, Li et al. presented raters with four categories: congruent, partially congruent, L1-only, or L2-only. However, a binary or even categorical approach to congruency can be considered subjective. For example, Wolter and Yamashita’s (Reference Wolter and Yamashita2018) L1-English users were not in total agreement when it came to judging their target collocations as acceptable, with 95% agreement for the congruent collocations and 92% for the incongruent. They also judged 24% of the baseline word pairs as acceptable. Furthermore, although the interrater reliability measured by Gyllstad and Wolter (Reference Gyllstad and Wolter2016) was high, κ = .802, it was not perfect, which also suggests an element of subjectivity is at play.
Although expert judgment generally involves a binary decision regarding collocational congruency, two other approaches result in continuous measures, which potentially avoid the subjectivity involved with coarse-grained categories. A corpus-derived frequency approach to congruency is based on the belief that if an English collocation such as bitter wind is directly translated (i.e., word-for-word with the same meanings) into the participants’ L1 (e.g., 苦い風 [nigai-kaze] in Japanese), and the resulting translation does not appear, or appears with extreme infrequency in a Japanese corpus, then the collocation can be deemed incongruent. This logic also dictates that a collocation such as strong wind, which when translated (i.e., 強い風 [tsuyoi-kaze]) occurs 151 times, or 1.45 times per million words (pmw) in a Japanese corpusFootnote 1 , is “more congruent” with English than a collocation such as deep love, which when translated (i.e., 深い愛 [fukai-ai]) occurs 0.90 pmw in the same Japanese corpus.
Although a corpus-derived frequency approach is relatively rare, it has precedent in the collocation processing literature for determining incongruent items to use in acceptability judgment tasks. In a study investigating L2 English collocational processing with L1 Turkish speakers, Öksüz et al. (Reference Öksüz, Brezina and Rebuschat2021) were concerned with the effects of frequency and association strength, and considered congruency a confound; thus, collocations that did not appear in a Turkish corpus were deemed incongruent and excluded from the study. Similarly, while developing a list of collocations that were incongruent between English and Korean, one of the criteria employed by Jeong and DeKeyser (Reference Jeong and DeKeyser2023) was that the Korean translations of the English collocations were nonexistent in the 36-million-word Korean language corpus.
Despite this precedent, the concept of corpus frequency as a proxy for congruency could be criticized for not reflecting the idiosyncratic experience of language users. A collocation that frequently occurs in one language user’s experience might not frequently occur in others due to factors such as personal preferences or group membership (Gablasova et al., Reference Gablasova, Brezina and McEnery2017). Furthermore, the frequency value assigned to a given word or phrase could be due to the choice of corpus, which might not be representative of the language, topics, and genres encountered by the sample. For instance, Brown (Reference Brown2017) compared the Corpus of Contemporary American English (COCA; Davies Reference Davies2008) and the British National Corpus (BNC; Kilgarriff, Reference Kilgarriff1995) and found that while the first 1,000 most frequent COCA and BNC wordlists shared 83 % of their respective lemmas, this amount dropped to less than 50% by the third 1,000 most frequent. Additionally, Wolter and Gyllstad (Reference Wolter and Gyllstad2013) found that their NNS sample displayed sensitivity to both congruent and incongruent collocations, with high-frequency incongruent collocations generally eliciting raster responses than low-frequency congruent ones. Thus, defining congruency via frequency might be oversimplistic in ignoring such an interaction. As a final point, although a collocation such as deep love appears less frequently in a corpus than strong wind, it is potentially due to less opportunities for the former to occur than to it being less congruent.
A third approach for defining congruency involves eliciting data from a norming study, resulting in another continuous congruency measure. In contrast to studies that rely on binary judgments from between one to several experts, norming studies involve piloting target items with generally larger groups and a rating scale. Although relatively uncommon, norming studies have been conducted by L2 formulaic language researchers. For instance, Ma and Hong (Reference Ma and Hong2026) obtained familiarity ratings on a 7-point scale from 19 L1 Vietnamese L2 Chinese learners for their collocation processing study. The 19 judges gave ratings for potential target items, which consisted of 109 L2 (i.e., Chinese) collocations and 188 L1-L2 collocations that were all translated into Vietnamese. The authors used the ratings to screen collocations, whereby only L1-L2 collocations with an average familiarity of 4 or above were retained for their experiment because “it indicates that these collocations are shared between Chinese and Vietnamese, that is, Chinese–Vietnamese congruent collocations” (p. 372). While Ma and Hong involved asking participants to judge familiarity, Takizawa (Reference Takizawa2024) recruited three Japanese Applied Linguistics graduate students to rate congruency via a 4-point naturalness scale. Spearman’s correlations between raters ranged from medium, ρ = .45 to large, ρ = .76, according to Plonsky and Oswald’s (Reference Plonsky and Oswald2014) thresholds, indicating that disagreement was common between the raters.
It is noteworthy that neither study reported above explicitly involved participants judging the “congruency” of the target items, presumably because congruency is a concept and/or term that their participants would be unfamiliar with. However, congruency is the quality that the researchers were investigating when asking about familiarity, naturalness, or even acceptability, which has been used in PJTs (e.g., Wolter & Gyllstad, Reference Wolter and Gyllstad2013; Yamashita & Jiang, Reference Yamashita and Jiang2010; Yi, Reference Yi2018). Accordingly, Yamashita (Reference Yamashita2014) demonstrated that asking whether collocations were “acceptable,” “commonly used,” or “natural” did not obscure the congruency effect in a PJT, concluding that congruency researchers have some latitude of freedom with terminology in norming study instructions. Although these alternative terms did not obscure the effect, it is important to realize that they were only proxies for congruency. A collocation might well be familiar and sound natural to an L2 user, but at the same time also be incongruent. It is also noteworthy that when scales have been administered to judge qualities such as familiarity and naturalness, the results put the items on a continuous scale, which poses the question of whether congruency exists on a continuous scale and not as a subjective, binary quality.
When these results are synthesized, it seems that all three methods reviewed had problems in terms of assessing collocational congruency. Congruency judgments seemingly involve an element of subjectivity and perhaps should not be determined with a binary decision made by a single researcher or even via a rating scale administered to a small group. Although this issue is fleetingly acknowledged by Wolter and Yamashita (Reference Wolter and Yamashita2015), who recognized that collocation categorization is “complicated” (p. 1194), and also by Gyllstad and Wolter (Reference Gyllstad and Wolter2016), who conceded in their footnotes that one reviewer questioned their classification of a number of their items, the issue has been largely ignored. Furthermore, although human judgments have been demonstrated as successful for rating the familiarity, semantic transparency, and frequency of formulaic language (Columbus, Reference Columbus2013), the strength of norming studies as a means of defining collocational congruency has not yet been empirically investigated. There is also the issue that the psychometrics of collocation norming studies have not been investigated, which casts doubt on such scales’ reliability. This problem requires attention when considering that rating scales are potentially inappropriate for L2 vocabulary research because the meaning of each category does not display invariance across participants (e.g., Nicklin, Reference Nicklin2021).
To summarize, collocation processing research indicates that incongruent collocations are processed with less efficiency and invoke a greater learning burden for low-proficiency learners; thus, congruency needs to be controlled for in applied psycholinguistic collocation experiments. However, the most common method of defining congruency reduces the quality to binary choice, generally based upon L1 speakers’ judgment. In some studies, corpus-derived frequency has been utilized to assess incongruency, but the appropriateness of this method has not been investigated. Finally, when norming studies have been employed the resulting scales imply that relying on instinct produces mixed results due to the subjective nature of binary congruency categorization. In the present study, the three approaches to operationalizing collocational congruency outlined above are compared and assessed in relation to one another.
The present study
In the present study, Wolter and Yamashita’s (Reference Wolter and Yamashita2018; henceforth WY2018) data were re-analyzed with four variables: two L1 frequency variables derived from Japanese corpus data and two variables resulting from a norming study. WY2018 was selected because the items were more tightly controlled and more comprehensive in terms of congruency types when compared to previous collocation processing studies. For instance, the items from WY2018, which were initially developed for Wolter and Yamashita’s (Reference Wolter and Yamashita2015) study, were checked with both the COCA and the Chunichi Newspaper database (Chunichi Shimbun Co. Ltd., 1987-), but also included a Japanese and English category that was not featured in the 2015 study. In WY2018, congruency was modeled as a four-level categorical variable, consisting of English-only, Japanese-only, and congruent collocations, along with the baseline word pairs. The four variables created for the present study were first compared with the original study’s categorical congruency variable and then with each other, before being integrated into WY2018’s linear mixed-effects model (LMM) as substitutes for the categorical congruency variable. This re-analysis aimed to evaluate whether corpus-based frequency metrics or norming study outcomes could serve as viable proxies for modeling collocational congruency as a continuous variable. Although uncommon in L2 research, data re-analysis has been successfully employed to demonstrate the benefits of bootstrapping (Larson-Hall & Herrington, Reference Larson-Hall and Herrington2010; Plonsky et al., Reference Plonsky, Egbert and LaFlair2015) and Bayesian analyses (Norouzian et al., Reference Norouzian, De Miranda and Plonsky2019), as well as to investigate outlier treatment methods (e.g., Hui & Wu, Reference Hui and Wu2024; Nicklin & Plonsky, Reference Nicklin and Plonsky2020). With this in mind, the following research questions were addressed:
RQ1: How well are expert-judged categories (i.e., baseline, English-only, Japanese-only, and both) confirmed by (a) L1 corpus frequency, and (b) norming study results?
RQ2: How strong is the relationship between two approaches to defining collocational congruency as a continuous variable, consisting of (a) L1 corpus frequency, and (b) norming study results?
RQ3: To what extent do LMM coefficients differ from WY2018 when collocational congruency is defined by (a) L1 corpus frequency, and (b) norming study results?
Definition and operationalization of collocational congruency
Given that the focus of this study revolves around defining collocational congruency, it is important that the construct is defined. The present study is a re-analysis of WY2018, in which a congruent collocation was defined as one that is felicitous in Japanese and English, and an incongruent collocation is felicitous in one or the other. Personal communication with the second author of WY2018 confirmed that congruent collocations consisted of word-for-word translations. Congruency was initially determined via expert judgment by the second author, a native speaker of Japanese with advanced proficiency in English (Wolter & Yamashita, Reference Wolter and Yamashita2015, Reference Wolter and Yamashita2018). Subsequently, the presence or absence of the collocations was verified using the COCA. The English-only and Japanese-only collocations were drawn from a prior study (Wolter & Yamashita, Reference Wolter and Yamashita2015) and were checked for occurrences in a Japanese corpus, the Chunichi Shinbun database. When proxy variables are compared against WY2018’s categorical variable, they are evaluated against this definition and operationalization.
Method
Participants
To address RQ1b and RQ2b, a collocation norming task was administered to L1 Japanese users via Crowdworks, which is a Japanese crowdsourcing website. Crowdworks was previously used by Patterson and Nicklin (Reference Patterson and Nicklin2023) to collect online self-paced reading data and resulted in better quality data than from Japanese university students. L1 Japanese users were also important because the WY2018’s materials were designed to be (in)congruent between Japanese and English. Furthermore, no information on English proficiency was collected because the job posting and all of the materials were in Japanese, and at no point was it mentioned to the participants that some of the items they were judging were (in)congruent between Japanese and English. In total, data were collected from 59 Japanese workers aged between 22 and 76, M = 43.85, SD = 11.08. The task took approximately 10 minutes, and participants received 500 yen (US$3.36). Details about the Crowdworks recruitment, ethics, and data protection are reported in Supplementary Material A, which is available along with all of the other Supplementary Material, data, and R scripts on the Open Science Framework (OSF; https://osf.io/a24nw).
RQ3 was addressed with a re-analysis of WY2018’s data; thus, the participants were the same three groups as the original study. The L1 English user (NS) group consisted of 27 students from a North American university. The lower-proficiency L2 English user (LNNS) group comprised 23 L1 Japanese undergraduate university students, and the higher-proficiency (HNNS) group contained 24 L1 Japanese graduate students. Therefore, the HNNS group were older and had studied English longer than the LNNS group. The HNNS group also achieved significantly higher vocabulary test scores on the Eurocentres Vocabulary Size Test (EVST; Meara & Jones, Reference Meara and Jones1990).
Procedure
Target items and translations
The target phrases were the 144 items from WY2018. The items consisted of English-only collocations (E-only; n = 24), Japanese-only collocations (J-only; n = 24), and collocations deemed congruent between Japanese and English (JE; n =24). In addition, there were 72 baseline word pairs, which were created by combining words from the three other conditions to create novel phrases that were unlikely to be used in either language (e.g., bitter hole).
For all 144 items, two L1 corpus-derived frequency variables and two norming study-derived variables were created. The two L1 frequency variables consisted of Balanced Corpus of Contemporary Written Japanese (BCCWJ; Maekawa et al., Reference Maekawa, Yamazaki, Ogiso, Maruyama, Ogura, Kashino, Koiso, Yamaguchi, Tanaka and Den2014) calque translation frequency values and BCCWJ phrase translation frequency values. Calque translations, which are word-for-word translations, were important because they are common in congruency research (e.g., Bahns, Reference Bahns1993; Gyllstad, Reference Gyllstad and Webb2021; Nesselhauf, Reference Nesselhauf2005; Sonbul & El-Dakhs, Reference Sonbul and El-Dakhs2020; Yamashita, Reference Yamashita2018). Phrase translations, which involved translating the entire phrase as a single unit, were also used. This decision was made because the translation process resulted in phrase translations that were markedly different from the calque translation but held the same meaning, and we were interested in how they compared with calque translations during the re-analysis. Figure 1 presents sharp rise to illustrate the distinction between calque and phrase translations. The calque translation, which involves translation sharp and rise separately, results in 鋭い上昇 [surudoijōshō] and does not occur in the BCCWJ. However, the phrase translation resulted in 急上昇 [kyūjōshō], which translates as rapid rise and appears in the BCCWJ 2.15 pmw. All translations of the single words and phrases in this study were first obtained with DeepL (n.d.), which is an AI-driven translation application, and were subsequently checked by three expert raters. The raters initially agreed on 285 (68.35%) of 417 translations, Fleiss’ (Reference Fleiss1971) κ = .39, 95% BCa CI [.31, .47] and then discussed the results with the first researcher until 100% agreement was achieved. Finally, a back-translation step was conducted on a subset of 50 (11.99%) items by a fourth bilingual rater, who agreed with all 50 translations. For reasons of space, the procedures for acquiring the Japanese translations and obtaining BCCWJ frequency data are described in greater detail in Supplementary Material B.
Japanese calque and phrase translations of sharp rise.
Note. BCCWJ = Balanced corpus of contemporary written Japanese (Maekawa et al., Reference Maekawa, Yamazaki, Ogiso, Maruyama, Ogura, Kashino, Koiso, Yamaguchi, Tanaka and Den2014).
Japanese BCCWJ frequency variables were required because Japanese was the participants’ L1, and L2 English corpus frequency was already incorporated in WY2018’s model in the form of COCA fiction frequency. Both Japanese and English frequency were modeled simultaneously as a necessity because WY2018’s congruency variable contained both Japanese- and English-only collocations. Thus, for frequency to be modeled as a proxy for congruency there had to be values for the Japanese- and English-only collocations; without BCCWJ frequency, there would theoretically be Japanese-only collocations modeled without frequency. All 144 items had a calque translation. For 61 (42.36%) of the items, the calque and phrase translations were identical, meaning that 83 (57.64%) items had two different translations: a calque translation and a phrase translation. Thus, BCCWJ frequency was obtained for all 227 (i.e., 61 + 83 + 83) unique translations of the 144 items.
The two norming study-derived variables represented naturalness judgments of calque translations and phrase translations, which were obtained with a 6-point scale. A 6-point scale was preferred because (a) we believed that it would elicit sufficient variance without overwhelming the participants with choice, and (b) the removal of a midpoint would force a decision regarding naturalness, which would lead to better data (e.g., Nemoto & Beglar, Reference Nemoto, Beglar, Sonda and Krause2014). These variables were also modeled with COCA fiction frequency because WY2018’s original model included it along with the categorical congruency variable. COCA fiction frequency for the collocations displayed a strong relationship with COCA frequency, ρ = .88 [.85, .91]; thus, register bias concerns were mitigated.
Norming instrument development
To construct the norming study instrument, the 227 phrase and calque translations of the 144 target phrases were placed into simple, non-context-biasing sentences. For instance, 厚い霧 (atsui kiri [thick fog]) was presented to participants as 厚い霧です (atsui kiri desu [It is thick fog]), with “It is…” added to signal the part of speech. All 227 Japanese sentences for the norming study were presented to the participants in random order on a Google Form, which also included the consent form and instructions and was written entirely in Japanese. The participants were instructed to read each sentence and answer the following question: [角括弧] の中のフレーズは、日本語として自然なものですか? ([Kakukakko] no naka no furēzu wa, nihongo to shite shizen’na monodesu ka; “Is the phrase in [square brackets] natural in Japanese?”). The word natural was selected because it has a precedent in previous collocation research (e.g., Gyllstad & Wolter, Reference Gyllstad and Wolter2016) and has also been shown by Yamashita (Reference Yamashita2014) to produce no significant difference in comparison with acceptable and commonly used as instructions in phrasal decision tasks assessing collocational congruency. Furthermore, the L1 Japanese assessors described in Supplementary Material B agreed that acceptable was not appropriate in this context in Japanese. The participants were instructed to select one of the following answers, which were coded on a scale of 1 to 6, where 1 = 非常に不自然 (Hijō ni fushizen; “Very unnatural”) and 6 = 非常に自然 (Hijō ni shizen; “Very natural”). Figure 2 presents a screenshot of the instrument. The norming study results provided each target calque and phrase translation with a naturalness score ranging from 0.00 to 6.00, which represented a congruency proxy.
Norming study instrument (a) in Japanese as presented to the participants, and (b) translated into English.
Note. All 227 items were randomly presented in a single form. This figure displays the first two items only.
The norming study results for the calque and phrase translations were initially analyzed with two separate Rasch rating scale models (RSM; Andrich, Reference Andrich1978) with the Test Analysis Modules (TAM; Robitzsch et al., Reference Robitzsch, Kiefer and Wu2024) R package to assess their reliability and determine the extent to which the distance between the categories on the 6-point scale was equal. Marginal maximum likelihood estimation (MMLE) was selected as the estimation method due to its focus on items, precision, and computational efficiency (see Nicklin & Vitta, Reference Nicklin and Vitta2022 for an overview of Rasch estimation methods). Because the initial Rasch analysis revealed poor category functioning, many-facet Rasch analysis (MFRA) was attempted to account for rater variance, which was a possible source of the issue. However, the data were fully crossed with 59 raters and 144 items, which made it too computationally demanding. Instead, cumulative link mixed-effect modeling (CLMM) with the ordinal package (Christensen, Reference Christensen2023) was conducted, which allowed us to control for rater severity by modeling the raters with random intercepts. The items were modeled as fixed effects, resulting in beta coefficients that are analogous to Rasch logits (Dunn, Reference Dunn2024). The results for all Rasch analyses are reported in Supplementary Material C, and the R code is available on the previously mentioned OSF page. For the remainder of the paper, raw norming study scores are reported instead of Rasch logits or CLMM beta coefficients because they are more intuitive and displayed an almost perfect relationship with the other coefficients; r > ± .93 for the calque translations and r > ± .95 for the phrases.
Analysis
RQ1 asked how well the expert-judged categories (i.e., baseline, English-only, Japanese-only, and both) from WY2018 were confirmed by (RQ1a) corpus frequency and (RQ1b) norming study calque and phrase translation results. To address this question, the corpus-derived frequency values and the naturalness ratings from the norming study were compared with descriptive statistics and boxplots. Median and interquartile range (IQR) were preferred to mean and standard deviation because of skewed distributions and outliers.
To address RQ2, which was posed to investigate the relationship between two approaches to defining collocational congruency as a continuous variable, BCCWJ calque and phrase frequency were correlated with the norming study data to assess how similar they were. The resulting correlation coefficients were interpreted according to Plonsky and Oswald’s (Reference Plonsky and Oswald2014) guidelines.
To address RQ3, which asked to what extent do LMM results differ from WY2018’s when collocational congruency is modeled via corpus frequency and norming study results, six LMMs were constructed with the lme4 (Bates et al., Reference Bates, Maechler, Bolker and Walker2015) package for R. The initial LMM (model a.) was identical to WY2018’s in the original study and represented collocational congruency defined by expert judgment (see Supplementary Material D for a comparison of the original published model and the present study’s re-analyzed model). The random effects structure consisted of by-item and by-participant random intercepts without any random slope termsFootnote 2 . The fixed effects consisted of group (NS, LNNS, HNNS), item type (JE, J-only, E-only, baseline), log-transformed adjective, noun, and phrasal COCA fiction frequency, and trial number. The final fixed effects were adjective and noun order, which represented whether a words’ appearance was the first or second and was deemed necessary because the baseline words were recombinations of the target collocations’ component words. Interactions were modeled between group and item type, group and three frequency variables, and phrase frequency and trial. Model effect sizes were compared with Nakagawa et al.’s (Reference Nakagawa, Johnson and Schielzeth2017) conditional- and marginal-R 2 , which represent percentage of variance explained by the total model and the fixed effects, respectively. Each effect size was accompanied with bootstrapped 95% CIs that were calculated with the performance package (Lüdecke et al., Reference Lüdecke, Ben-Schachar, Patil, Waggoner and Makowski2021).
The structure of the remaining five comparison LMMs were the same as the first, except the item type variable representing collocational congruency was replaced. First, a null model without any collocational congruency proxy (model b.) was modeled as a control to determine the amount of variance explained by the other fixed effects. Following this, the remaining LMMs involved either BCCWJ corpus-derived frequency (models c. and d.) or norming study data (models e. and f.) as congruency proxies. The four proxies and COCA frequency variables were centered to assist model fit.
The six LMMs were compared in three ways. First, conditional- and marginal-R 2 effect sizes for all six LMMs were compared to assess which model explained a greater amount of the variance in the data. Second, the Akaike information criteria (AIC) and Bayesian information criteria (BIC) for the four models containing the frequency and norming study data were compared to determine which of the five comparison models fit the data best. Finally, the regression coefficients were compared to determine whether the norming study-derived congruency and corpus-derived frequency variables resulted in the same pattern of results as the original study.
Results
RQ1: comparisons between expert judgment, corpus-derived frequency, and norming study results
RQ1 was posed to investigate how well (RQ1a) corpus-derived frequency values and (RQ1b) norming study responses for calque and phrase translations reflected the expert-judged categories from WY2018. For RQ1a, descriptive statistics of the two corpus-derived variables (i.e., BCCWJ calque translation frequency and BCCWJ phrase translation frequency) for the four groups (i.e., Baseline, English-only, Japanese-only, and Congruent [JE]) are displayed in Table 1. The median frequency values in Table 1 were all less than 1.00 pmw, which is relatively infrequent. However, the IQR and maximum values for English-only and congruent (JE) phrase translations show that there was considerable spread, which was confirmed by the boxplots in Figure 3. Importantly, the spread of the phrase translation corpus values for the English-only collocations and the fact that this group’s median—0.22 pmw in the BCCWJ—was higher than the Japanese-only group—0.04 pmw—did not align with expert judgment. In contrast, the calque translations did conform to the expert categories, whereby the baseline word pairs and English-only collocations had practically no BCCWJ occurrences. However, seeing as WY2018 involved calque translations, this is unsurprising.
Descriptive statistics for target phrase frequency per million words (N = 144)
Note. Although log-transformed values were employed in the final model to account for the data’s Zipfian distribution, frequency is reported here as raw occurrences per million words for ease of interpretation.
Boxplots of log frequency per million occurrences.
The baseline item outliers visible in Figure 3 offer initial evidence that the corpus checks employed in WY2018 to check the status of the word pairs worked to an extent. The baseline word pairs were constructed by rearranging the collocations’ component words and were intended to lack semantic meaningfulness (e.g., bitter hole). Although the majority of the baseline word pairs did not occur in the BCCWJ, the boxplots show that some baseline pairs occurred more frequently than the median value of the Japanese-only collocations. A list of the baseline outliers is provided in Table 2. The largest outliers from the calque translations were broad water, near feet, and young health, which occurred between 0.20 and 0.04 pmw in the BCCWJ, suggesting that despite being outliers in Figure 3 they are still highly infrequent. On the other hand, the phrase translation of hard injury occurred 3.83 pmw, which was more than the majority of the collocations, as visible in Figure 3 (log frequency pmw = 0.58). Overall, despite a small set of outliers, the distinctions made between the word pairs and collocations via expert judgment were supported by BCCWJ corpus frequency, with the exceptions being a minority of English-only phrase translations. Interestingly, although the word pairs in the original study were checked with a different Japanese corpus (the Chunichi newspaper database), the results are largely consistent.
Baseline word pair outliers
For RQ1b, descriptive statistics for the calque and phrase translation norming study data across the four groups are displayed in Table 3. The results, which are illustrated in Figure 4, indicated that the norming study responses generally supported expert judgment. For the calque translations, the ratings of the baseline word pairs and English-only collocations were significantly lower than the Japanese-only and congruent collocations, as attested to by the lack of confidence interval overlap in Table 3. However, one outlier in the baseline group, good moment, M = 5.17, SD = 1.21, was rated as more natural than the average score for the Japanese-only collocations. For the phrase translations, however, the median of the English-only collocations, 5.36, was slightly higher than the Japanese-only ones, 5.06, which was unexpected.
Descriptive statistics of norming study data
Boxplots of norming study data.
To summarize the results for RQ1, while expert judgment was broadly supported, the distributions of the collocation groups observed for corpus frequency in Figure 3 and norming study data in Figure 4, particularly for the calque translations, indicate that collocation congruency may lie on a continuum. In turn, this suggests that categorical classification could be an overly simplistic approach to defining collocational congruency. In other words, certain collocations may be perceived as more congruent than others, and categorizing them by perceived congruency risks overlooking critical nuances, akin to the limitations associated with reducing continuous variables to arbitrary categories for ANOVA (Plonsky & Oswald, Reference Plonsky and Oswald2017). To assess whether the continuum realized by the frequency and norming study data offered an alternative to expert-judged categories for modeling collocational congruency, WY2018 was re-analyzed by substituting the four-level categorical variable, Type’ (i.e., baseline, English-only, Japanese-only, and congruent [JE]), from the original study with continuous variables representing corpus-derived frequency and norming study ratings.
RQ2: correlations between corpus-derived frequency and norming study results
Before WY2018 was re-analyzed, the relationship between the corpus-derived and norming study-derived variables was investigated with correlations. The scatterplot matrix in Figure 5 revealed that the data failed to meet Pearson’s r assumptions of (a) linearity and (b) no outlying data points. However, the data met the assumptions of paired observations, continuous or ordinal variables, and monotonicity (Dellinger, Reference Dellinger and Allen2017), rendering Spearman’s ρ appropriate. Thus, Spearman’s ρ along with 95% confidence intervals (CIs) were calculated using the spearmanCI package (de Carvalho, Reference de Carvalho2024) in R (R Core Team, 2024). The coefficients were interpreted according to Plonsky and Oswald’s (Reference Plonsky and Oswald2014) thresholds.
Correlation matrix of proxy congruency variables.
Note. The “calqueNorm” and “phraseNorm” variables are the norming study data.
The correlation matrix in Figure 5 displays the collocation coefficients along with histograms of each variable’s distribution and scatterplots of the relationships between the variables. The correlation coefficients representing the relationship between BCCWJ calque and phrase frequency, ρ = .63 [.50, .76], and the calque and phrase norming study data, ρ = .72 [.62, .81], were both large, suggesting that the two approaches to translation were more similar than different. The coefficient representing the relationship between BCCWJ phrase frequency and the phrase translation norming results, ρ = .69 [.59, .78], was larger than the same, medium-sized, comparison for the calques, ρ = .57 [.45, .70]. To summarize the results for RQ2, strong relationships were observed between the calque and phrase frequency variables and norming study results, suggesting that the variables were largely measuring the same construct; the naturalness of the translations in Japanese, which was intended as a congruency proxy.
RQ3: re-analysis of WY2018 with corpus-derived frequency, and norming study results
RQ3 addressed the extent to which results from WY2018 change when collocational congruency is defined by norming study data (models c. and d.) and corpus frequency (models e. and f.). For reasons of space, only the results of interest are reported here. However, all six LMMs are fully reported in Supplementary Material E, along with the assumption checks in Supplementary Material F. The first LMM (model a.) was an attempt to replicate WY2018’s results with their data. Supplementary Material D shows that the correlations were almost perfect; thus, the structure of the models for the following re-analyses was correct. The second model was a null model that did not contain any collocational congruency variable and was modeled to determine the amount of variance explained by the other fixed effects. The remaining four models involved substituting the Type variable for the four proxies investigated above: calque and phrase translation frequency, and calque and phrase translation norming study data.
The marginal- and conditional-R 2 effect sizes and bootstrapped 95% CIs for all six models are displayed in Table 4. When comparing the original study’s results with the null model, the marginal-R 2 effect size representing the fixed effects dropped by .01 in the null model, while the conditional-R 2 representing the whole model remained the same, .44 [.43, .50], suggesting that the four-level Type variable representing collocational congruency in the original study explained approximately 1% of the variance in the data. However, the values for all four models containing congruency proxies were the same, marginal-R 2 = .19, indicating practically no difference between the amount of fixed effect variance that they explained. The variance explained by the fixed effects in the four proxy variable models, marginal-R 2 = .19, was the same as the null model, indicating that the variables added almost no explanatory power. Additionally, the amount of variance explained by three of the proxy variable models in total was .01 smaller than the WY2018 and null models, conditional-R 2 = .43, which again showed that they provided little explanatory power. Only the phrase frequency model explained the same amount of variance as the WY2018 and null models. However, the almost complete overlap of the CIs indicates the differences were negligible.
Linear mixed-effects model effect sizes and fit statistics
Table 4 also presents the AIC and BIC fit statistics for the four models incorporating frequency or norming study data, which are comparable because they contain the same number of parameters. The lower values for the calque norming study data indicate the best model fit, whereas the phrase norming study data offer the worst fit. However, the numbers are all practically identical, suggesting that there is very little difference between the variables in terms of model fit.
The full set of LMM coefficients are reported for all four models in Supplementary Material E. The coefficients revealed that the parameters most affected by the four frequency and norming data proxy variables’ inclusion in the re-analyzed LMMs were frequency and its interaction with proficiency, which for reasons of space will be the sole focus here.
Amplified frequency effect in proxy models
Table 5 shows that the only significant predictor of RTs from the four frequency and norming data proxy variables was calque frequency in model c., t(142.02) = −2.57. This result was similar in magnitude to the significant effects observed in WY2018 (model a.) for the comparison between baseline pairs and E-only collocations, t(185.76) = −3.19, and JE collocations, t(172.86) = −2.85. However, unlike in the calque frequency model (model c.), the VIFs for these latter two comparisons were greater than James et al.’s (Reference James, Witten, Hastie and Tibshirani2021) conservative threshold of 5.00, which raises multicollinearity concerns. In all four models containing proxy variables (models c. to f.), the effect of COCA frequency was more pronounced than in WY2018 (model a.), t(171.23) = −2.78. The smallest increase over this result was observed for phrase norming data in model f., t(208.84) = −9.37, and the largest for calque frequency in model c., t(225.04) = −11.07. The COCA frequency VIF in WY2018’s model, 8.27, once again implied multicollinearity issues, whereas the COCA frequency VIFs in the proxy models did not.
Coefficients of interest from six re-analyzed linear mixed-effects model
Amplified frequency interaction by group
The final coefficients of interest in Table 5 pertained to the interaction between proficiency and COCA collocation frequency. All six models displayed a significant difference between the NSs and LNNSs interaction with collocation frequency; as frequency increased, the NSs RTs increased at a faster rate than the LNNSs (see Figure 6). Interestingly, Figure 6a suggests that the LNNSs RTs became slightly slower as the collocations became more frequent, although the slope’s angle is close to zero. Similarly, the difference between the NSs and HNNSs interaction with frequency was significant, except for WY2018 (model a.) and the phrase norming data (model f.). In these two models, the NSs and HNNSs RTs increased at a similar rate as collocation frequency increased. However, Figure 6 shows that despite being statistically significant, the difference in rate between the groups was not particularly drastic in any of the other models. As with the frequency comparisons reported above, the VIFs for the proficiency and collocation frequency interactions in WY2018 (model a.) were greater than 5.00, suggesting multicollinearity. This concern was mitigated in all of the other models, including the null model, which is consistent with the interpretation that the categorical congruency variable was the source of the issue.
Interactions between log-transformed COCA Frequency and Group for (a) WY2018, (b) null model, (c) calque frequency, (d) phrase frequency, (e) calque norming data, and (f) phrase norming data.
Figure 6. Long description
Six line graphs depict interactions between log-transformed COCA Frequency and Group for different models and data. Panel A: The line graph shows RT (Log Transformed) on the vertical axis and Coca Frequency (Fiction) on the horizontal axis. The graph includes three groups: NS, HNN, and LNN, each represented by different line styles. The lines show a downward trend as Coca Frequency increases. Panel B: The line graph shows a similar setup with RT (Log Transformed) on the vertical axis and Coca Frequency (Fiction) on the horizontal axis. The lines for NS, HNN, and LNN show a slight downward trend. Panel C: The line graph shows RT (Log Transformed) on the vertical axis and Coca Frequency (Fiction) on the horizontal axis. The lines for NS, HNN, and LNN show a more pronounced downward trend. Panel D: The line graph shows RT (Log Transformed) on the vertical axis and Coca Frequency (Fiction) on the horizontal axis. The lines for NS, HNN, and LNN show a downward trend. Panel E: The line graph shows RT (Log Transformed) on the vertical axis and Coca Frequency (Fiction) on the horizontal axis. The lines for NS, HNN, and LNN show a downward trend. Panel F: The line graph shows RT (Log Transformed) on the vertical axis and Coca Frequency (Fiction) on the horizontal axis. The lines for NS, HNN, and LNN show a downward trend. The legend indicates the groups: NS with a solid line, HNN with a dashed line, and LNN with a dotted line.
To summarize the results for RQ3, calque frequency (model c.) was the only proxy variable that was a significant predictor of RTs when replacing categorical congruency. Additionally, frequency became the dominant predictor in all four proxy variable models that involved replacements for categorical congruency (models c. to f.). The main effect of frequency increased substantially in all four proxy variable models, while frequency’s interaction with group became stronger. The results for all three RQs are compiled in Table 6.
Summary of results
Discussion
Three distinct approaches for defining collocational congruency were evaluated, comprising expert judgment, corpus-derived frequency, and norming study data. The three research questions are now considered in turn.
Augmenting expert judgment with calque translations
RQ1 was formulated to determine whether L1 corpus-based frequency and norming study data confirmed expert judgment regarding collocation status from WY2018. Overall, the calque translations outperformed phrase translations in both the frequency and norming study data by aligning more closely with expert-judged categories. This likely reflects that the items in WY2018 were developed with word-for-word translations. Figure 3 shows that most baseline and English-only calque translations had zero BCCWJ occurrences, while Figure 4 indicates that these groups also received relatively low naturalness scores compared to Japanese-only and JE collocations. Conversely, English-only phrase translations appeared more frequently in the BCCWJ than their Japanese-only counterparts and yielded slightly higher naturalness scores.
A more detailed data examination reveals that a noticeable baseline phrase translation outlier, hard injury, was translated into a compound, 重傷 (jūshō), by DeepL and agreed upon by the three raters. The compound occurred 3.25 pmw in the BCCWJ, which Figure 3 shows was more frequent than most of the collocations in the other three groups, while the calque translation, 硬い傷 (katai kizu), did not occur at all. This discrepancy’s cause becomes apparent when the compound is translated back into English and equates to the adjective-noun collocation serious injury, which is an acceptable English adjective-noun collocation with 825 COCA occurrences. In essence, the phrase translations of certain baseline word pairs, such as hard injury, reflected the semantic meaning of the closest “acceptable” phrase. Such translations resulted in higher frequency counts and naturalness ratings for the phrase translations over the calque equivalents. This constitutes an issue for relying solely on AI-driven translations for collocation congruency research because machine translation systems typically optimize for semantic adequacy, resulting in a phrase translation as opposed to a calque translation whenever the former is available, as happened with the phrase translations in this study.
The discrepancy between the calque and phrase translation frequency also highlights a potential terminological issue in the collocation processing literature. While some researchers explicitly define congruency in terms of a narrow, “word-for-word” (i.e., calque) translation (e.g., Gyllstad, Reference Gyllstad and Webb2021; Li et al., Reference Li, Wolter, Yang and Siyanova-Chanturia2025; Ma & Hong, Reference Ma and Hong2026; Nesselhauf, Reference Nesselhauf2005; Sonbul & El-Dakhs, Reference Sonbul and El-Dakhs2020; Yamashita, Reference Yamashita2018), or as an absent L1 construction “expressed using a different word or words in the L1” (Jeong & DeKeyser, Reference Jeong and DeKeyser2023, p. 2), others adopt broader definitions that could allow both calque and phrase translations. For instance, even when calque translations have been employed, as in WY2018, incongruent collocations have been characterized as being felicitous in one language but not another (Li et al., Reference Li, Wolter, Yang and Siyanova-Chanturia2025; Wolter & Yamashita, Reference Wolter and Yamashita2018; Yamashita, Reference Yamashita2014), or defined in terms of similarity between languages (e.g., Takizawa, Reference Takizawa2024; Yamashita & Jiang, Reference Yamashita and Jiang2010; see Table G1 in Supplementary Material G for the complete quotations of these examples), which could mistakenly be interpreted as being a phrase translation. Given this potential confusion, collocation processing research would benefit from an unmistakable, standardized definition of congruency to facilitate cross-study comparisons. We propose adopting the term “calque translation” as the standard because (a) the present study’s results indicate that calques are less prone to semantic adaptation, as exemplified by the hard injury example, and (b) it is the appropriate term from the field of linguistics.
However, this finding does not imply that phrase translations should be disregarded. When constructing baseline word pairs, an item that has zero corpus occurrences as a calque translation might have a phrase translation that serves as a semantically close approximation, as seen with hard injury. Therefore, while we advocate for calque translations to define collocational congruency status, phrase translations might function as a complementary reference. If neither the calque nor phrase translation appears in target language corpora, a collocation is more likely to serve as an ideal baseline word pair. It should also be emphasized that corpus-derived frequency alone is not a definitive measure. Rather, L1 and L2 corpus-derived frequency should be considered necessary but insufficient criteria for determining collocational congruency and should both be employed in conjunction with initial categorization via expert judgment.
Potential issues with norming study data
RQ2 aimed to investigate the relationship between the corpus-derived frequency variables and norming study data. The correlation between the calque and phrase translations was large for both BCCWJ frequency, ρ = .63 [.50, .76], and the norming data, ρ = .72 [.62, .81]. However, the large effects were unsurprising considering that 61 (42.36%) of the 144 items had the same translation. A post hoc correlation analysis involving the 83 (57.64) items with different calque and phrase translations revealed that the relationship shrank to medium for the BCCWJ frequency, ρ = .49 [.30, .68], but remained strong for the norming data, ρ = .63 [.49, .77], despite the smaller magnitude. The medium effect size for frequency supports the claims above that phrase frequency can serve as complementary reference to calque frequency when categorizing collocational congruency because there is enough difference between the two measures for phrase frequency to offer something new.
When considering the correlations between the four proxy variables, the evidence suggests that they exhibited greater similarity than difference. Based on this similarity, an argument can also be made for corpus-derived frequency as preferable to norming data for categorizing collocational congruency. First, corpus-derived data can be collected for a fraction of the time and cost of norming study data. Corpus data for the present study were collected with the BCCWJ and COCA, which are both free and intuitive to use. Similar resources are available for numerous other languages, such as Chinese (e.g., Academia Sinica Balanced Corpus of Modern Chinese [Institute of Information Science, Academia Sinica, 2013]), Dutch (Stevin Dutch Reference Corpus [SoNaR; Oostdijk et al., Reference Oostdijk, Reynaert, Hoste, Schuurman, Spyns and Odijk2013]), and Spanish (e.g., El Corpus del Español [Davies, Reference Daviesn.d.]). Conversely, the norming study data, while relatively easy to collect via crowdsourcing, engendered financial costs for comparable results.
Second, and more critically, the norming study results initially demonstrated poor psychometric quality. While Rasch analysis revealed high reliability for responses to both the calque and phrase translations, it also highlighted that the distance between the categories were unequal. To improve the instrument, categories 1 and 2 (i.e., strongly disagree and disagree) could be collapsed into a single category, and for the calque translations, it is arguable that categories 3 to 6 could also be collapsed, which would create a binary naturalness variable. Although this modification could be investigated with the current data, a new sample would be necessary to verify its effectiveness because the original responses were collected using a different scale (i.e., 1 to 6). Furthermore, reducing the scale to a binary format in alignment with Rasch model expectations would conflict with the premise that congruency exists on a continuum, which was supported by RQ1. Although category functioning concerns were somewhat mitigated when rater variance was modeled in a CLMM, this approach was limited by an assumption of equal thresholds across items; thus, by-item threshold data were not available.
Another advantage of the Rasch analysis over CLMM was that the former approach also demonstrated that numerous items misfit the expectations of the model. To address this issue, the items’ responses would require individual examination to determine why they misfit. Problematic items would require replacing, which would necessitate data from a new sample. Collecting data from a new sample to evaluate whether collapsing the scale and replacing items improves the instrument would entail significant time and finances, without guarantee that the instrument would be improved or would not require further development. The results of the LMM re-analysis also indicated that there was little difference between the outcomes when norming study results were substituted for corpus frequency, which is freely available. Additionally, the only proxy variable across the four models that was a significant RT predictor was calque frequency. Considering these points, norming studies appear more demanding in terms of the time and resources required to meet psychometric acceptability and provide comparable results to corpus-derived frequency. Consequently, further research is warranted to determine whether the extra time and resources required to develop better scales result in significantly improved results.
No proxy for collocational congruency
RQ3 was formulated to assess whether any of the four variables could be considered a proxy for congruency in collocation processing research. Although calque norming data (model e.) fit the model slightly better according to AIC and BIC in Table 4, the re-analyzed LMM results suggested that none of the variables could be considered as suitable proxies. The effect size for the null model’s fixed effects, marginal-R 2 = 0.19 [.17, .22], was .01 lower than the WY2018 one (model a.), marginal-R 2 = 0.20 [.18, .22], indicating that the original study’s congruency variable accounted for approximately 1.00% of the explained variance. Although this is a relatively small amount, this variance was not accounted for by the proxy variables, for which the marginal-R 2 values were all .19 (see Table 4).
Crucially, all three collocation groups from WY2018’s categorical congruency variable elicited significantly faster reaction times compared to the baseline word pairs (see Table 5). Among the four proxy variables, only calque frequency (model c.) significantly predicted reaction times, t(142.02) = −2.57, p = .011. Notably, when the four proxy variables replaced the categorical congruency variable in the four re-analyzed LMMs (models c. to f.), the influence of the L2 corpus variable, COCA fiction frequency, became dominant. While previously a relatively weak but significant predictor in WY2018, t(171.23) = −2.78, p = .006, COCA fiction frequency emerged as the strongest predictor across all four re-analyzed LMMs, with the weakest effect observed with the phrase norming data in model f., t(208.84) = −9.37, p < .001. These findings suggest that WY2018’s categorical congruency variable accounted for unique variance in processing latencies between collocations and baseline word pairs, representing the congruency effect. Once this variable was replaced with one of the four proxy variables, the proxies failed to explain the same variance, which was instead captured by COCA fiction frequency.
The amplified frequency effect was further reflected in the interaction between COCA fiction frequency and proficiency group. In Figure 6a, representing WY2018, the LNNS learners’ RTs exhibit a marginal slowing as COCA frequency increases. Once the categorical variable was removed, however, a negative relationship was observed, which was comparable in strength to the HNSs and NSs (see Figures 6b∼6f). Again, this pattern reflects a redistribution of variance, where COCA frequency captured variance that was previously explained by the categorical congruency variable. Because the LNNSs were more strongly influenced by the congruency effect than the more proficient participants, frequency had a greater effect on their RTs once the congruency variable was removed.
When synthesized, the findings addressing RQ3 suggest that modeling collocational congruency as a categorical variable based upon expert judgment explained variance that the four proxy variables failed to capture. In other words, the variance captured by the categorical variable, which uniquely reflected congruency in the WY2018 model, was instead explained by frequency in the proxy models. This result could be interpreted as the categorical variable performing somewhat superior to the proxy variables for modeling congruency effects when considering that only one of the proxy variables, calque frequency (model c.), was a significant predictor. While WY2018’s categorical variable approach prevented COCA frequency from disproportionately influencing the model, it also introduced concerns regarding multicollinearity. Specifically, most congruency and frequency terms reported in Table 5 for WY2018’s LMM exhibited VIFs exceeding the conservative threshold of 5.00 for model inclusion (James et al., Reference James, Witten, Hastie and Tibshirani2021). In contrast, the largest VIF in the models without categorical congruency was 2.22, indicating that the multicollinearity concerns were mitigated.
This observation does not imply that categorical congruency should be disregarded. For instance, if James et al.’s liberal threshold of 10.00 had been employed, multicollinearity would not have been a concern. However, the high VIF values suggest that the regression coefficients may be unreliable. One strategy for countering the multicollinearity observed with the categorical congruency variable is to increase the sample size. As Fox (Reference Fox2002) noted,
$\sqrt {VIF} \;$
reflects the extent to which the standardized beta coefficient’s CI is increased in comparison with non-multicollinear data. Furthermore, Morrissey and Ruxton (Reference Morrissey and Ruxton2018) demonstrated through simulations that increasing sample sizes reduces standard errors, thereby improving the precision of CIs and lowering VIF values. Despite WY2018’s sample size of 74 being respectable in terms of L2 vocabulary research, there were just 23 or 24 members of each group. An even larger sample might enhance the regression coefficients’ precision and mitigate the multicollinearity concerns indicated by the VIFs. Although small sample sizes are endemic in SLA research (Loewen & Hui, Reference Loewen and Hui2021; Nicklin & Vitta, Reference Nicklin and Vitta2021), the increasing accessibility of online experiments and crowdsourced participant recruitment makes achieving larger sample sizes more feasible. For example, McConnell et al. (Reference McConnell, Hintz and Meyer2025) recruited over 500 participants for a battery of online psycholinguistic experiments.
Limitations and future research
Despite our best intentions, this study is not without limitations. First, the results are restricted to a specific language learning context: L1 Japanese learners of L2 English. It is possible that languages with greater similarity, such as Swedish and English, exhibit different patterns from those reported here or may be less susceptible to multicollinearity concerns.
Second, although the poor psychometric properties identified in the Rasch analysis of the norming study results are consistent with similar analyses of rating-based data that also revealed unequal distance between response levels (e.g., Nicklin, Reference Nicklin2021; T. Yamashita, Reference Yamashita2022), it might be premature to dismiss the value of norming studies for collocation research entirely. Further investigation is needed to explore methods for improving the psychometric properties of such data, whether it be via collapsing categories and modeling rater severity, or perhaps even investigating alternative methods such as graded response models (Samejima, Reference Samejima1969) or anchoring vignettes (Hopkins and King, Reference Hopkins and King2010)Footnote 3 .
Third, although we have argued that expert judgment is effective but should not be solely relied upon, we acknowledge that we relied on three experts when judging the accuracy of the DeepL translations. However, there will often be an element of subjectivity regarding translations, especially between linguistically distant languages, and our intention was to investigate the potential of modeling congruency on a continuum as opposed to with discrete categories. Further research would be warranted to determine an optimal method for ensuring translation reliability across languages.
Fourth, the use of DeepL to initially translate the words might also constitute a limitation. Although DeepL is an AI-driven translation tool, significant advances in large language models (LLMs), such as ChatGPT (OpenAI, 2025), emerged after the data for this study were collected. It is likely that contemporary LLMs offer higher-quality translations than those reviewed by the expert raters in this study. Future research should explore the potential of LLMs for generating translations. Moreover, given that LLMs have been successfully employed to estimate familiarity ratings for multi-word expressions (Brysbaert et al., Reference Brysbaert, Martínez and Reviriego2025), future studies should investigate their utility in categorizing or rating collocational congruency.
Finally, because this study was a re-analysis of WY2018’s data, all of the findings are limited to adjective-noun collocations that are (in)congruent between English and Japanese. Future research is warranted with other constructions, such as verb-noun collocations. Future work might also consider investigating non-calque phrase translations that share semantics but not form, such as with sharp rise in English and Japanese (see Figure 1). Further investigation into pattern congruency at the item level (e.g., bitter X and 苦い X [nigai X]) through cross-lingual distributional values would also prove valuable. Such values could be integrated into a weighted variable incorporating calque frequency, potentially yielding a more precise operationalization of the congruency effect; the entrenchment of comparable form-meaning mappings, not just calque translationsFootnote 4 . As a final thought, future research should also consider using L1 and L2 association strength measures, such as mutual information (MI) and log-likelihood (G 2), to quantify the relative strength of co-occurrence between target collocations, although arbitrary thresholds should be avoided (see Gries, Reference Gries2022 for details).
Conclusion
In this study, expert judgment, corpus-derived frequency, and norming study data were compared to determine which was superior for modeling collocational congruency in research involving the congruency effect. Although a continuous variable that could act as a proxy for expert judgment was not successfully identified, a set of best practice suggestions emerged from the results that have implications for future research. While we argue that discrete categories should likely remain the standard approach for modeling the congruency effect, expert-judged categories have two key concerns: (a) misidentification of word pairs or collocations, and (b) multicollinearity. The following suggestions aim to alleviate these key concerns.
First, collocation researchers should adopt calque translations as the standard for congruency research. When the definition of congruency varies across studies, the results become less comparable. Calques should be preferred to phrase translations because the latter may lead to the misidentification of semantically similar non-collocations, such as compounds and noun phrases (see Supplementary Materials B for details). For example, brief moment is an acceptable noun phrase in Japanese, 束の間 (tsukanoma), when strict adherence to calque translation is not utilized. Second, both L1 and L2 corpora should be consulted to ensure that baseline items do not appear in either and that L1-only items are absent from L2 corpora (and vice versa). Third, norming studies require further investigation because despite being resource-intensive, in the present study they yielded results comparable to corpus frequency. Additionally, norming studies have traditionally used proxy terms for congruency judgments, such as “familiar” (e.g., Li et al., Reference Li, Wolter, Yang and Siyanova-Chanturia2025; Ma & Hong, Reference Ma and Hong2026) and “natural” (e.g., Takizawa, Reference Takizawa2024). Although such proxy terms do not affect results when used in place of “congruent” in a decision task prompt (Yamashita, Reference Yamashita2014), they are still proxies and might result in mislabeling if used to identify items for a psycholinguistic task when considering that an incongruent collocation might also be “familiar” and “natural” to some extent. Finally, large sample sizes should be recruited to avoid potential multicollinearity issues with categorical congruency variables. While the present study’s data do not allow for precise recommendations regarding an ideal sample size, a substantially larger sample than the 74 participants in WY2018 might be necessary to reduce variance inflation factors (VIFs).
Replication package
The R code and data for all of the analyses is available on the Open Science Framework page for this study at https://osf.io/a24nw.
Open access
