RT data processing and accuracy

First and foremost, RT data need to be considered in conjunction with accuracy data. That is, RT data in isolation can lack substantive significance. A key reason is that, in many, if not all, cases, researchers preprocess RT data such that only correct responses are analyzed. This step is potentially meant to align the data preprocessing procedure with conventional practices in psycholinguistics where incorrect responses are considered to be an indication that the participant engaged in nontarget psycholinguistic operations, and hence are irrelevant to the phenomenon in question. However, in SLA, researchers are most interested in learning gains and, as a result, SLA studies often involve some form of progression from no knowledge to partial knowledge to mastery, as typically demonstrated by an improvement in accuracy. In other words, a high accuracy rate is not guaranteed, which is in sharp contrast with psycholinguistic research, where a relatively high accuracy is almost a prerequisite for an effect. A key implication of discarding incorrect trials is that a lower accuracy means a reduced number of observations for analysis, not only lowering statistical power but also potentially shifting the inference made by the researchers. In this light, interpretations of RT data in the same way as psycholinguists require some caution.

For example, if we were to take RT data as a speed measure at its face value, one might suggest that two groups of learners with a similar mean RT are at a comparable level of mastery, at least in terms of processing speed. However, one group may be more accurate in its responses, while the other is less accurate. When that is the case, believing that they are indistinguishable in general terms can be problematic. The seeming equivalence in performance is merely an artifact of the data preprocessing, namely removing incorrect responses from the analysis. The RT data for the low-accuracy group lose representativeness disproportionately as more data are discarded for them. On this account, using processing speed (alone) to predict language use performance requires more caution, too (e.g., Hui & Godfroid, Reference Hui and Godfroid2021; Zhang & Yang, Reference Zhang and Yang2023), because a fast responder might not be a more proficient learner, as hypothesized. They can be merely an individual who compromises accuracy for speed. Indeed, this accuracy–speed trade-off is not new to researchers using RT data, but it has not received sufficient attention, in part, because the modeling approach undertaken can almost always handle only one outcome variable. As a result, variability due to leaning toward accuracy or speed is somewhat accepted as measurement error in practice.

To further illustrate the relationship between RT and accuracy, we plot the open data shared by Hui (Reference Hui2020), who investigated the processing trajectory of intentional word learning (see Figure 1). In the study, the author trained English speakers to learn 16 Swahili–English translation pairs, after which the participants performed 10 blocks of animacy judgments (i.e., to decide whether the stimulus denotes a living thing). Overall, the accuracy gradually increased as the learners took advantage of the feedback given after each trial. RT of correct responses decreased, more obviously during the first blocks. From these patterns, suggesting that there was “faster processing by the participants over time” is not wrong (Hui, Reference Hui2020, p. 340). However, the more accurate and complete description would be that processing of the words the participants have learned and responded correctly to was faster over time. The key here is that researchers must match their interpretations of the results with their data analysis approach, including the preprocessing steps. It is because when incorrect trials are excluded, researchers artificially create systematic missingness in the data, changing the inference from what is initially intended (i.e., from processing speed for all items to that only for correctly responded items).

Figure 1. Accuracy (green line) and RT (blue line) trajectories in intentional word learning reported in Hui (Reference Hui2020).

This need to consider accuracy along with RT also applies to secondary measures, such as lexical decisions in the priming paradigm. As discussed, when priming is observed, it means that the lexical item is represented in the mental lexicon and is connected to other items in the mind, depending on the manipulation. In this regard, priming has been taken as evidence of lexical knowledge. Perhaps more importantly, priming measures have been interpreted as tapping into tacit or nondeclarative lexical knowledge, especially when the prime is masked by symbols and a very brief presentation, such that the participant is not consciously aware of its presence. It is because, first, the psycholinguistic operations that drive the facilitation are out of the participant’s awareness (e.g., Elgort, Reference Elgort2011; Hui et al., Reference Hui, Godfroid and Elgort2022). Second, the participants are almost always unaware of the masked prime (or, at least, the researcher would confirm that they are) and still show a priming effect. At the same time, it is useful to bear in mind that this interpretation applies only to the items that the participants successfully recognize in the first place. This is because, again, only correct responses are included in the analysis.

Crucially, some theoretical tensions start to emerge when considering the above. On the one hand, a lexical decision task itself is an explicit task. The participants are well aware of their judgments. Correct responses then require explicit knowledge. On the other hand, tacit knowledge is believed to be categorically different from explicit knowledge although they are intimately related (Hui et al., Reference Hui, Godfroid and Elgort2022). But the measurement of it (lexical decisions in the priming paradigm) requires the participant to first demonstrate explicit knowledge through a correct response. In a sense, it seems that researchers may operate under the assumption that the development of tacit knowledge first requires explicit knowledge. Whether this assumption holds is an ongoing debate in the literature. What we want to stress here is that researchers using the priming paradigm to measure tacit or implicit knowledge must be aware of this assumption, which they inherently accept through their data analysis procedure.

Finally, a closer look at Figure 1 suggests that the decrease in RT of correct responses was relatively more obvious during the initial stages and gradually leveled off. This was also the stage in which the accuracy was relatively low in the trajectory. In other words, the most noticeable result (i.e., development of speed) was observed during a stage where RT data were less informative due to low accuracy, creating an interpretative RT–accuracy dilemma for researchers using RT data to index learning. An important implication of the need to consider accuracy and RT in tandem is that RT measures might be more suitable for examining the refinement of partial knowledge in the context of a word learning study. If researchers are interested in earlier stages of acquisition where accuracy can be low, they should consider measures that combine speed and accuracy such that, for example, processing speed is adjusted for error rates (e.g., $\frac{{mean\,RT}}{{error\,rate}}$) (see Vandierendonck, Reference Vandierendonck2017 for an overview).

RT differences as an individual measure

Another issue to consider is the extent to which RT differences can be used as an individual difference measure, that is, a measure that can index a given learner’s ability or knowledge levels. In almost all areas of SLA, a high correlation is expected between the learner’s test score and their level of knowledge or skills. Someone achieving higher on a vocabulary test, for example, should possess more or better lexical knowledge than someone who has a lower score. In this regard, SLA researchers must maintain a high measurement bar for our measures. Measurement, by definition, is the principled assignment of numerical values to attributes, such as L2 knowledge (Stevens, Reference Stevens1946). Unfortunately, this basic principle of measurement does not necessarily apply to RT differences, at least in the ways that they are used in SLA. Researchers, for instance, cannot necessarily claim that a learner who slows down more when encountering ungrammaticality in a self-paced reading task has more knowledge than someone who slows down less. Similarly, a larger priming effect with lexical decisions (more facilitation) may not be interpreted as the learner having better lexical knowledge. This is especially true when RT measures are more sensitive to factors such as handedness, physical difficulties, and coordination.

Part of the issue is that some of the paradigms SLA researchers use to capture RT differences have been borrowed from psycholinguistics, where researchers primarily focus on group-level effects. A higher degree of unreliability is accepted because the performance of individuals is influenced by factors that the researcher cannot control, such as the familiarity of the stimuli by the participant, the order of presentation, and so on. Also, psycholinguists often adopt a counterbalanced design in which participants only see one version of an item to avoid multiple exposures. As a result, researchers understand their limitations when it comes to examining the RT difference of an individual. Some would even argue that RT differences are only meant to be analyzed at the group level, although others still use them to index individual ability in a moderation analysis to investigate differential effects of factors, such as proficiency, on priming, for example. In most cases, psycholinguists also seldom use RT differences as a predictor variable, which assumes perfect reliability.

In this light, SLA researchers must decide for ourselves how we want to use RT differences and understand the consequences of accepting RT differences as merely a group-level measure. First of all, to be able to examine aptitude-treatment interactions (e.g., whether a given treatment is particularly useful for someone with certain characteristics; see DeKeyser, Reference DeKeyser2021), researchers must have an outcome measure that can index an individual’s ability. On the contrary, when RT differences are used as a group-level outcome, the researcher’s hands are tied, because the moderation analysis can no longer reveal differential treatment effects that are based on the characteristics of the learner, such as perceptual abilities (e.g., sound discrimination skills; Shao et al., Reference Shao, Saito and Tierney2023) and cognitive abilities (e.g., working memory capacity; Suzuki & DeKeyser, Reference Suzuki and DeKeyser2017). Therefore, when the outcome does not tap into one’s ability at the individual level, the analysis itself is not meaningful. This is an immensely important limitation, especially for researchers who rely on RT differences to tap into tacit lexical knowledge (with a lexical decision task in the priming paradigm), or implicit or automatized explicit grammatical knowledge (with a word-monitoring or a self-paced reading task). What this means is that the field can face challenges when conducting individual differences research in terms of how this type of knowledge is acquired, as measured by RT differences.

The second consequence of accepting RT differences as group-level measurse is that researchers cannot examine their predictive validity. The predictive validity of measures is critical because when a measure indexing a given construct can strongly predict overall language performance, researchers have empirical evidence that this construct is germane to authentic language use, an important goal of SLA research. For example, it is useful to know how relevant implicit grammatical knowledge is to reading and listening comprehension. In analytic terms, one would use the RT difference, indexing implicit knowledge, to predict comprehension performance in a regression model. But this analysis would require the RT measure to be perfectly reliable, a psychometric property that RT differences often lack and the very reason why psycholinguists only use RT differences at the group level. Indeed, many arguments for measuring tacit, implicit, or automatized explicit knowledge, using RT differences, are exactly that these knowledge types are key for real-time language use. However, when group-level RT differences cannot be scrutinized in terms of predictive validity (or be used to predict language use in a statisical model), such arguments can only remain at the conceptual level, which is truly undesirable.

Relatedly, if RT differences only represent a group-level measure, researchers are not in a position to test their construct validity – specifically, their convergence and divergence validities – in relation to other measures because the statistical technique often used to investigate construct validity, confirmatory factor analysis, relies on covariance between variables that index one’s ability at the individual level. A group-level priming measure, for example, may not offer a starting point for examining its correlation with other tests. When that is the case, it is almost impossible to empirically differentiate these RT difference measures, supposedly tapping into tacit or implicit knowledge, from other assessments of explicit knowledge.

In summary, SLA researchers must further understand RT difference measures and the price for accepting them as merely group-level measures. Collectively, we need to make informed decisions on how to use RT differences, or whether to use them at all. The first step in this direction is to properly examine the psychometric properties of RT differences. For example, there has already been work on estimating the reliability of RT differences using a model-based approach (Hui & Wu, Reference Hui and Wu2024). If RT differences can be reliable, the next step is to more closely scrutinize their validity by, for example, correlating priming measures with existing vocabulary measures or by using RT differences to predict language performance. It is worth mentioning that the motivation of such work can be initially questioned, because those who assert that RT difference measures are designed for group-level analyses would consider this kind of investigation as a misuse of the data. At the same time, we stress that the stakes are high for SLA researchers to understand what RT differences are measuring at an individual level.

RT data involving multiple processes

Finally, reaction time can be less informative if a task involves more than one process (Jiang, Reference Jiang, Godfroid and Hopp2022). When multiple processes are involved in a task, it is difficult to disentangle the observed effects to make meaningful interpretations. One example is a self-paced reading task (see Marsden et al., Reference Marsden, Thompson and Plonsky2018 for an overview). As mentioned, self-paced reading tasks are often used to measure implicit grammatical knowledge deployed in real time. The idea is that learners with the target knowledge are expected to show sensitivity to grammatical violations, operationalized as a slowdown in reading times, compared with the baseline grammatical condition. Another example also involving sentence reading is a grammaticality judgment task, which has been used to measure explicit and implicit knowledge, depending on modality and time–pressure manipulations. In both self-paced reading and grammaticality judgment tasks, it can be difficult to pinpoint specifically what aspect of the stimuli, or what cognitive processes, lead to the delay in processing ungrammatical sentences, as sentence reading involves processes such as word recognition, parsing, and semantic integration. Therefore, it is perhaps not straightforward to take the tasks at face value and claim that these tasks afford relatively pure measures of grammatical knowledge.

A similar issue can be observed in vocabulary research. For example, the word association task is sometimes used to obtain RT data to index the strength of networks in a learner’s mental lexicon. The key here is that researchers find it difficult to specify the factors and their contribution to the observed effects. For example, when a group of learners is faster, a number of interpretations are sensible, including more efficient word recognition, stronger associations, and/or a larger number of associations available. It is also challenging to spell out the specific level(s) of association (lexical vs. conceptual) the task is targeting. It must be acknowledged that these issues also apply to accuracy-based measures. These are perhaps not problems unique to RT research. The key message here is that to ensure validity of their measures (i.e., to make accurate interpretations of their results), SLA researchers should be mindful of the psychological processes involved in completing the tasks. While no measure is a pure measure of anything, knowing what is or can be underpinning a numerical result that we interpret is of paramount importance.