1. Introduction
1.1. Data-driven learning in the era of generative AI
Data-driven learning (DDL) is an instructional approach in which learners engage directly with authentic language data – typically concordance lines extracted from corpora – to discover patterns, test hypotheses, and draw conclusions about language use (see Boulton & Vyatkina, Reference Boulton and Vyatkina2021, for a research synthesis). Grounded in inductive learning principles (Flowerdew, Reference Flowerdew2009; Römer, Reference Römer2024), DDL shifts learners’ roles from passive recipients of knowledge to active language analysts (Boulton, Reference Boulton2010; O’Keeffe, Reference O’Keeffe2021). This shift promotes deeper engagement with linguistic forms and fosters learner autonomy, critical thinking, and data literacy (Boulton, Reference Boulton2009; O’Keeffe, Reference O’Keeffe2021, Reference O’Keeffe2023). DDL is particularly valuable in language learning because it exposes learners to real-world language usage (Boulton, Reference Boulton2009), helps them notice subtle patterns and variations, and enables more personalised and exploratory learning experiences (Römer, Reference Römer2024).
With the advent of generative artificial intelligence (GenAI), the corpus and DDL community has begun actively exploring its potential integration into corpus linguistics and DDL, with emerging debates reflecting both optimism and concern (e.g. Boulton, Reference Boulton, Choubsaz, Díez-Arcón, Gimeno-Sanz, Morgana, Murphy and Seracini2025; Cheung & Crosthwaite, Reference Cheung and Crosthwaite2025; Crosthwaite & Baisa, Reference Crosthwaite and Baisa2023; Dong & Wang, Reference Dong and Wang2025; Flowerdew, Reference Flowerdew2025; Lin, Reference Lin2023; Mizumoto, Reference Mizumoto2023; Pérez-Paredes, Reference Pérez-Paredes2026; Pérez-Paredes et al., Reference Pérez-Paredes, Curry and Aguado Jiménez2025; Sun & Mizumoto, Reference Sun and Mizumoto2025a). From one perspective, integrating DDL into GenAI platforms (e.g. ChatGPT) can enhance concordancer accessibility, promoting learners’ independent use. However, issues such as the authenticity of GenAI-generated sentences for concordance lines and the risk of hallucinations have been identified as potential threats to its implementation (see Crosthwaite & Baisa, Reference Crosthwaite and Baisa2023, for a detailed discussion). Nevertheless, DDL researchers (Crosthwaite & Baisa, Reference Crosthwaite and Baisa2023; Flowerdew, Reference Flowerdew2025) have acknowledged that the field should now focus on how to integrate this technology into the DDL domain, rather than on whether it should be used, given its growing status in the language learning and teaching field. In this context, understanding how learners personally experience and evaluate the effectiveness of GenAI-based DDL beyond its technical affordances is paramount (Sun & Mizumoto, Reference Sun and Mizumoto2025b), as such perceived effectiveness would influence the uptake and sustainable use of the technology.
1.2. A learner-centred framework for evaluating DDL tool effectiveness
Despite the demonstrated effectiveness of DDL as a learning tool, the literature has identified several critical considerations, particularly regarding the nature of the language data and the tools through which learners access them. Together, these considerations provide the conceptual foundation for an integrated framework to evaluate whether a particular DDL tool is effective in practice, rather than assessing it solely on its pedagogical rationale. To operationalise this evaluation, several key criteria for the effectiveness of DDL tools have been synthesised from the literature and structured into a multidimensional framework consisting of six distinct domains. These criteria were identified based on both the propositions of DDL researchers regarding the characteristics of DDL and its operation as well as empirical studies that have examined its use and effectiveness for second language (L2) learners, each of which is discussed below.
First, comprehensibility of sentences depends on whether the language data provided to learners are at an appropriate level of difficulty. If the data are overly complex or beyond learners’ level of comprehension, they may struggle to make sense of the materials, thereby hindering the learning process (Allan, Reference Allan2009; Lee et al., Reference Lee, Warschauer and Lee2017, Reference Lee, Warschauer and Lee2019). In such cases, the cognitive effort required simply to decode sentences leaves little capacity for noticing patterns or reflecting on language use, undermining one of the central rationales for DDL. Thus, ensuring that concordance lines are broadly comprehensible is a basic precondition for effective engagement with corpus data.
Second, relevance to semantic learning refers to the extent to which the data contain information that is useful for learners’ semantic development. When the data are comprehensible but lack a sufficient proportion of relevant linguistic information, learners may understand the content but still derive little benefit in terms of language development. This dimension is consistent with work emphasising the role of DDL in supporting the meaning-focused exploration of lexis and collocation (Flowerdew, Reference Flowerdew2009; Geluso & Yamaguchi, Reference Geluso and Yamaguchi2014), during which exposure to authentic examples is expected to deepen learners’ understanding of lexical meaning, usage patterns, and phrasing.
Third, relevance to syntactic learning specifies whether the same data offer opportunities to notice and reflect on syntactic patterns, so that learners can use DDL to develop grammatical awareness and control. This aligns with research suggesting the potential of corpus-based approaches to raise awareness of form–function relationships and recurrent patterns in grammar (Boulton & Vyatkina, Reference Boulton and Vyatkina2021; Römer, Reference Römer2024). When concordance lines are rich in salient syntactic structures that match learners’ needs, they can function as a resource for inductive grammar learning rather than as isolated decontextualised examples.
Fourth, accessibility for independent use captures how easy it is for learners to operate a tool on their own. Even if language data are both accessible and relevant, the effectiveness of DDL can be compromised if the tool itself is difficult to use or requires considerable effort to operate (Boulton, Reference Boulton2010; Lin, Reference Lin2023); a tool that is not user-friendly may discourage learners from engaging with the materials, ultimately reducing the overall impact of DDL (Geluso & Yamaguchi, Reference Geluso and Yamaguchi2014). Interface complexity, opaque commands, and time-consuming search procedures can all raise the threshold for independent use and confine DDL activities to tightly controlled classroom settings rather than everyday learning (Sun & Mizumoto, Reference Sun and Mizumoto2025b).
Fifth, perceived pedagogical value reflects learners’ beliefs about how beneficial the DDL tool is for their language use and development (Aliponga, Reference Aliponga2013; Sun, Reference Sun2007). When learners regard a DDL tool as helpful and aligned with their learning goals, such positive perceptions can enhance their motivation to engage with it (Boulton, Reference Boulton2009; Flowerdew, Reference Flowerdew2009). Perceived value links the immediate experience of using a tool to broader learning outcomes; even a technically well-designed tool is unlikely to be used sustainably if learners do not see it as contributing meaningfully to their progress (Leńko-Szymańska, Reference Leńko-Szymańska, Turula, Mikołajewska and Stanulewicz2015).
Sixth, support for autonomous learning relates to whether learners perceive the tool as suitable for their individual learning needs and thus worth integrating into their everyday study routines. When a DDL tool is seen as both effective and personally relevant, learners are more likely to use it autonomously in their daily lives, similar to a dictionary or reference resource, which is consistent with perspectives of DDL as a means of fostering learner autonomy and self-directed engagement (Boulton, Reference Boulton2009; O’Keeffe, Reference O’Keeffe2021, Reference O’Keeffe2023). In this sense, a tool’s contribution to autonomy depends not only on its technical affordances but also on whether learners recognise it as a viable part of their own learning repertoire.
1.3. Evaluation of DDL from learners’ perspective
In DDL literature, while many empirical studies measure quantifiable L2 gains and skills, an equally critical inquiry focuses on learners’ perception of DDL tools. The present study situates itself in the latter, emphasizing that perceived usefulness is a primary determinant of sustained tool adoption, as proposed in the technology acceptance model (TAM; Abdullah et al., Reference Abdullah, Ward and Ahmed2016; Almaiah et al., Reference Almaiah, Jalil and Man2016). However, despite interest in the field, little is known about how learners judge DDL and associated tools with respect to multidimensional criteria. Previous studies have primarily reported learners’ general attitudes towards DDL activities (Geluso & Yamaguchi, Reference Geluso and Yamaguchi2014; Sultana, Reference Sultana2024; Vyatkina, Reference Vyatkina2016), the perceived usefulness of concordancers (Aliponga, Reference Aliponga2013; Sun, Reference Sun2007), and overall satisfaction with corpus-based tasks (Benavides, Reference Benavides2015; O’Sullivan & Chambers, Reference O’Sullivan and Chambers2006). While these investigations provide valuable insights into user engagement, very few have examined learners’ evaluations of the DDL and associated tools themselves using clearly specified, systematic criteria.
Moreover, assuming that a DDL tool is effective simply because it satisfies all criteria to a similar degree is problematic, as this view ignores the relative importance of each individual criterion. In practice, it is rarely possible for a single tool to optimise all dimensions simultaneously; design constraints (Forti, Reference Forti, Bédi, Choubsaz, Friðriksdóttir, Gimeno-Sanz, Björg Vilhjálmsdóttir and Zahova2023), technological limitations (Boulton, Reference Boulton2010; Dong & Wang, Reference Dong and Wang2025; Sultana, Reference Sultana2024), and pedagogical priorities (Boulton, Reference Boulton2012) often require trade-offs. From the learner’s perspective, the relative importance of these criteria varies by individual and context (Leńko-Szymańska, Reference Leńko-Szymańska, Turula, Mikołajewska and Stanulewicz2015): some learners may perceive greater value in semantic relevance (Yılmaz, Reference Yılmaz2017), whereas others may appreciate syntactic support (Aliponga, Reference Aliponga2013; O’Sullivan & Chambers, Reference O’Sullivan and Chambers2006), ease of use (Geluso & Yamaguchi, Reference Geluso and Yamaguchi2014), or compatibility with autonomous study (Mizumoto et al., Reference Mizumoto, Chujo and Yokota2016).
Therefore, what is needed is a list of relevant criteria and an empirical account of how learners prioritize them and how such priority structures inform context-sensitive choices among available DDL tools. This requirement is particularly salient in the current landscape, where traditional corpus-based concordancers and GenAI-based tools coexist as alternative implementations of DDL (e.g. Crosthwaite & Baisa, Reference Crosthwaite and Baisa2023; Dong & Wang, Reference Dong and Wang2025; Flowerdew, Reference Flowerdew2025). Although GenAI-based tools, such as GPT-powered concordancers, have recently been proposed (Flowerdew, Reference Flowerdew2025; Lin, Reference Lin2023), systematic multi-criteria comparisons remain limited. Recent exceptions, such as Satake (Reference Satake, Wang and Cárdenas Claros2024) and Sun and Mizumoto (Reference Sun and Mizumoto2025a), have begun to address this by comparing AI-assisted and traditional DDL from the perspectives of learning preference and task performance. Satake (Reference Satake, Wang and Cárdenas Claros2024) found that while ChatGPT facilitates idea structuring, its output and suggestions necessitate critical verification for accuracy. Meanwhile, corpus-based research helps writers acquire specialized vocabulary through authentic data, although it often lacks the contextual relevance needed for specific writing tasks. Consequently, Satake suggests a complementary integration of both tools to foster a more comprehensive and contextually rich L2 writing process. Similarly, Sun and Mizumoto (Reference Sun and Mizumoto2025a) observed that while learning outcomes were comparable, GenAI was perceived more positively due to its ease of use. However, even these contemporary studies focus on general preferences following specific activities, rather than modelling the underlying priority structures learners apply when evaluating tool effectiveness. Overall, there remains a significant gap in examining how learners’ own priority criteria shape their evaluations of traditional and GenAI-based DDL. This highlights the need for an analytical framework that can model these priorities and incorporate them into structured comparisons of different DDL tools.
1.4. Analytic hierarchy process for multi-criteria evaluation in language learning
One methodological framework that directly addresses the aforementioned multi-criteria judgements is the analytic hierarchy process (AHP). Initially developed by T. L. Saaty (Reference Saaty1979) and R. W. Saaty (Reference Saaty1987), the AHP is a structured method for analysing complex judgements through pairwise comparisons of multiple criteria. Although AHP was originally created to support decision-making in management and engineering, its core strength, namely systematically capturing how people assign relative importance to different factors, makes it well suited to domains in which subjective evaluations and contextual variability play a central role (e.g. Lu et al., Reference Lu, Lian and Lien2015).
In language education, which requires consideration of diverse learner needs, instructional contexts, and pedagogical goals, AHP offers a powerful yet underutilised framework for examining how different elements of language learning are perceived and valued. Unlike traditional survey analyses, which often assume equal importance of variables or treat responses in isolation, AHP makes it possible to model how different groups or individuals prioritize multiple aspects of language-learning tools and experiences by assigning explicit weights to each criterion. This capacity to account for differentiated weighting is particularly valuable in fields where effectiveness is rarely universal and frequently depends on specific learner profiles, instructional settings, and educational contexts.
Importantly, the applicability of AHP is not limited to the evaluation of DDL tools. This approach can be extended to a wide range of research questions related to language learning and teaching. Whether used to understand teacher priorities in curriculum design, student preferences in learning strategies, or institutional goals in program development, AHP provides a transparent and replicable method for integrating and interpreting diverse judgements within a coherent, multi-criteria framework. Accordingly, AHP has considerable potential to support more context-sensitive, learner-informed, and data-rich investigations of what matters most in language learning.
2. Current study
Building on this background, the present study investigates how language learners evaluate DDL tools when multiple effectiveness criteria must be considered simultaneously. Although previous research has discussed various conditions under which DDL can be effective, it has rarely examined how learners prioritise these conditions or how such priority structures can be incorporated into a systematic comparison of different DDL approaches (or tools). This issue is particularly salient in the current landscape, where traditional corpus-based concordancers and GenAI-based tools coexist as alternative implementations of DDL (Crosthwaite & Baisa, Reference Crosthwaite and Baisa2023; Dong & Wang, Reference Dong and Wang2025; Flowerdew, Reference Flowerdew2025; Lin, Reference Lin2023).
To address this gap, this study adopts six effectiveness criteria derived from the literature – comprehensibility of sentences, relevance to semantic learning, relevance to syntactic learning, perceived pedagogical value, accessibility for independent use, and support for autonomous learning – and uses them as a common framework for evaluating DDL tools from the learner’s perspective. Within this framework, the study compares a traditional corpus-based concordancer and a GenAI-based concordancer. The AHP is employed to model learners’ multi-criteria judgements by eliciting their relative priorities among the six criteria and using these learner-derived weights to obtain priority-sensitive evaluations of them.
This study seeks to provide both substantive and methodological contributions. Substantively, it clarifies which aspects of DDL effectiveness learners regard as more or less important and how these priority structures are reflected in their assessments of traditional and GenAI-based concordancers. Methodologically, it illustrates how AHP can be used to move beyond simple mean ratings and derive evaluations of competing DDL tools that explicitly incorporate learner-valued criteria. The study addresses the following research questions (RQs):
-
1. How do learners prioritize the identified criteria for DDL tools when judging their effectiveness?
-
2. How do a traditional corpus-based concordancer (BNC) and a GenAI-based concordancer (GPT) compare in terms of overall perceived effectiveness when learner-derived criterion weights are applied?
-
3. How do learners’ evaluations of the BNC and GPT differ across the identified criteria, and what distinct strengths and weaknesses emerge for each tool?
3. Method
This study is part of a research project investigating the potential of GenAI-mediated DDL, specifically focusing on measurable learning gains and learner perceptions of the approach.
3.1. Participants and DDL tools
A total of 35 undergraduate students majoring in English education at a private university in Seoul participated. Although small, this sample size (N = 35) is justifiable; AHP is not an inferential statistical method requiring large samples for statistical power but rather focuses on eliciting consistent priorities from a targeted group. The participants (11 men, 24 women) had an average age of 21.7 years. Their English proficiency was classified as B2 or C1 according to the Common European Framework of Reference for Languages (CEFR; Council of Europe, 2018), based on departmental assessments. Participants were recruited from a linguistics course taught by one of the authors. They were briefed on the study’s objectives and the availability of extra credit for their participation. Finally, students were informed of their right to withdraw from the study at any time without disadvantage and were offered alternative assignments to earn equivalent extra credit. After signing an informed consent form, they had the opportunity to engage in DDL twice: first with a traditional online concordancer accessing the British National Corpus (BNC), and then with a GPT-based concordancer developed specifically for this research. The participants were asked to study 10 target words (bestow, furnish, begrudge, ascribe, perjure, instill, bequeath, render, entrust, and avail) selected from a vocabulary test in the course. This vocabulary test comprised 17 target verbs characterised by complex argument structures; from these, the 10 words with the lowest mean scores across participants were selected for the study. To ensure a balanced comparison, five words were randomly assigned to each condition for each participant; thus, participants encountered different word sets for each tool. The participants engaged in DDL activities with each concordancer for 20 minutes per session, conducted once a week over two consecutive weeks.
For the traditional concordancer, participants were instructed to use Tom Cobb’s website (https://www.lextutor.ca/conc/) to engage in DDL. They were taught to access the BNC (Brown + BNC Written), enter a target word, and generate concordance lines containing that word. Participants were then required to study the five target verbs by analysing their meanings and syntactic structures through the generated concordance lines. To guide the DDL process, participants were provided with a learning sheet and asked to engage in a four-step pedagogical cycle for each target word: (1) formulating an initial hypothesis about the verb’s meaning and argument structure, (2) cross-checking this hypothesis against multiple concordance lines, (3) confirming their final understanding, and (4) producing an original sentence using the target verb.
For the GenAI-based DDL experience, a customised concordancer was developed using the GPT-4o “My GPTs” feature. This tool was designed to generate 20 concordance lines when a target word is entered. To ensure the output adhered to a standard keyword-in-context (KWIC) format (Boulton, Reference Boulton2010) consistent with traditional online concordancers, this concordancer was developed based on Lin’s (Reference Lin2023) prompting framework. To ensure the pedagogical aim of the DDL approach, the prompt prevented the tool from providing definitions for the entered verbs. Consistent with the procedure for the traditional concordancer, participants were asked to analyse the five target verbs, focusing on their semantic properties and syntactic structures through the generated concordance lines, and to engage in the four-step cycle.
3.2. Instrument: Questionnaire
To examine learner perceptions and preferences in relation to BNC and GPT, a questionnaire grounded in AHP was administered via Google Forms. The initial version of the questionnaire was piloted with three students from the same department who were not involved in the main study. This pilot focused on the clarity and comprehensibility of the items and the nine-level preference scale, particularly the nuanced distinctions between levels. Based on their feedback, the items were refined to enhance respondent understanding (see Appendix A in the supplementary material for the finalised version). The questionnaire was administered in Korean (i.e. the participants’ L1), to ensure accurate comprehension of the subtle nuances of the items.
In the main study, to ensure the validity of the ratio-scale judgements, participants completed a 10-minute training session before the assessment. This session provided a detailed explanation of the AHP pairwise comparison logic, explicitly addressing the nuances of the nine-level preference scale. To further familiarise participants with the current scale, a practice session using illustrative examples (e.g. comparing preferences for different transportation modes) was also included.
The questionnaire comprised two sections, as described below.
3.2.1. Section 1: Prioritization of criteria
Participants were asked to make a series of pairwise comparisons between the six evaluation criteria to indicate their relative importance. The six criteria and their definitions are listed in Table 1. In accordance with the AHP assumption of criteria independence, the following criteria were defined as analytically distinct categories. While concepts such as “comprehensibility” and “accessibility” may overlap in general educational contexts, they are treated here as separate practical facets to more precisely evaluate the effectiveness of DDL tools (encompassing both traditional and GPT-based DDL tools).
Six criteria and their definitions
Table 1. Long description
A table with six rows and two columns. The first column lists six criteria for evaluating the BNC/GPT tool in language learning: Comprehensibility of sentences, Relevance to semantic learning, Relevance to syntactic learning, Perceived pedagogical value, Accessibility for independent use, and Support for autonomous learning. The second column provides definitions for each criterion. Row 1: Criterion 1, Comprehensibility of sentences, Definition: Does the BNC/GPT tool provide example sentences that are understandable and accessible enough to support language learning? Row 2: Criterion 2, Relevance to semantic learning, Definition: Do the example sentences provided by the BNC/GPT tool help you accurately understand the core meaning of the target word in context? Row 3: Criterion 3, Relevance to syntactic learning, Definition: Do the example sentences help you recognize how the target word functions grammatically in a sentence? Row 4: Criterion 4, Perceived pedagogical value, Definition: To what extent do you feel the BNC/GPT tool is educationally valuable in supporting your English learning? Row 5: Criterion 5, Accessibility for independent use, Definition: Is the BNC/GPT tool easy to access and operate on your own, without requiring extensive instructions or technical support? Row 6: Criterion 6, Support for autonomous learning, Definition: Do you think the BNC/GPT tool is suitable for ongoing, self-directed learning beyond the classroom?
Because AHP requires participants to make all unique pairwise combinations among the criteria, participants responded to 15 pairwise comparison questions in this section (calculated as all unique pairs of the six criteria). An example comparison was “Which is more important to you when using a DDL tool: ‘comprehensibility of sentences’ or ‘accessibility for independent use’?” This 9-point scale was adopted based on R. W. Saaty’s (Reference Saaty1987) standard AHP practice, as it optimally reflects the cognitive limits of human ability to discriminate the intensity of relationships between elements. The values of 5, 4, 3, and 2 indicated an increasing preference for Criterion A over Criterion B. For example, a value of 5 indicated that Criterion A was considered five times more important than Criterion B. A value of 1 represented an equal preference for A and B, while values of 1/2, 1/3, 1/4, and 1/5 indicated an increasing preference for Criterion B over A. In detail, the scales were as follows:
-
• 5: I strongly prefer A over B.
-
• 4: I clearly prefer A over B.
-
• 3: I moderately prefer A over B.
-
• 2: I slightly prefer A over B.
-
• 1: I have equal preference for A and B.
-
• 1/2: I slightly prefer B over A.
-
• 1/3: I moderately prefer B over A.
-
• 1/4: I clearly prefer B over A.
-
• 1/5: I strongly prefer B over A.
3.2.2 Section 2: Prioritization of alternatives
In the second section, the participants compared the BNC and GPT with respect to each of the six criteria. For each criterion, the participants were asked which tool they preferred, resulting in six pairwise comparisons. The same 9-point scale was used to express the strength of their preferences.
Notably, the six criteria used in the questionnaire were developed by the authors based on a comprehensive review of the DDL literature to ensure content validity (see Section 1.2). Unlike traditional surveys that measure latent constructs using multi-item Likert scales, AHP relies on pairwise ratio-scale judgements; thus, logical consistency (measured via the consistency ratio [CR]) is used in place of traditional reliability metrics such as Cronbach’s alpha (details are included in the following section).
3.3. Data analysis
This section outlines the data analysis procedure using the AHP, which was conducted entirely in Microsoft Excel without the use of specific statistical software. The analysis consisted of two main phases: first, determining the priority weights of the six evaluation criteria; second, calculating the global preference scores for the two concordancers (BNC and GPT) based on these weighted criteria. Across both phases, individual participants’ responses were aggregated using geometric means to form group consensus matrices. Subsequently, the logical reliability of their pairwise judgements was validated using CR, ensuring all responses met the acceptable threshold (CR ≤ 0.10). Because the complete AHP procedure involves extensive matrix calculations and normalizations, the detailed step-by-step procedures are provided in Appendix B in the supplementary material due to space limitations.
4. Results
4.1. RQ1. How do learners prioritize the identified criteria for DDL tools when judging their effectiveness?
The first research question concerned learners’ prioritization of the six decision criteria for effective DDL tools. Based on the aggregated pairwise comparisons, the priority weights and rankings of the six criteria were derived, as presented in Table 2.
Priority weights and rankings of the six effectiveness criteria
Table 2. Long description
The table presents the priority weights and rankings of six effectiveness criteria for digital learning tools. It consists of three columns: Rank, Criterion, and Priority Weight. The criteria are ranked from 1 to 6 based on their importance. The first row indicates that Criterion 2, Relevance to semantic learning, has the highest priority weight of 0.230. The second row shows Criterion 6, Support for autonomous learning, with a priority weight of 0.186. The third row lists Criterion 4, Perceived pedagogical value, with a priority weight of 0.156. The fourth row includes Criterion 3, Relevance to syntactic learning, with a priority weight of 0.149. The fifth row shows Criterion 1, Comprehensibility of sentences, with a priority weight of 0.141. The sixth row lists Criterion 5, Accessibility for independent use, with a priority weight of 0.137. The total priority weight sums up to 1.000.
As shown in Table 2, the resulting weight vectors were 0.141, 0.230, 0.149, 0.156, 0.137, and 0.186 for Criteria 1 to 6, respectively. These values indicate that learners prioritized relevance to semantic learning (Criterion 2) most highly (0.230), followed by support for autonomous learning (Criterion 6; 0.186), perceived pedagogical value (Criterion 4; 0.156), relevance to syntactic learning (Criterion 3; 0.149), and sentence comprehensibility (Criterion 1; 0.141), whereas accessibility for independent use (Criterion 5; 0.137) received the lowest weight.
The weight ratios further enabled interpretable comparisons between criteria; for example, learners considered semantic relevance to be approximately 1.544 times more important than syntactic relevance (0.230/0.149), suggesting that they believed DDL tools should place greater emphasis on semantic learning than on syntactic learning. Further comparisons across the criteria reveal other notable priority differences. For instance, semantic relevance (Criterion 2), the highest-ranked criterion, was deemed 1.679 times more important than accessibility for independent use (Criterion 5; 0.230/0.137), which ranked lowest. Similarly, support for autonomous learning (Criterion 6), the second-highest ranked criterion, was valued 1.319 times higher than sentence comprehensibility (Criterion 1; 0.186/0.141) and 1.358 times higher than accessibility for independent use (Criterion 5; 0.186/0.137). These ratio-based comparisons provide a mathematically grounded basis for interpreting the relative importance learners attach to different aspects of DDL tools.
A CR of 0.009 was obtained, far below the conventional 0.10 threshold, confirming that the aggregated group-level judgements were logically consistent.
4.2. RQ2. How do a traditional corpus-based concordancer (BNC) and a GenAI-based concordancer (GPT) compare in terms of overall perceived effectiveness when learner-derived criterion weights are applied?
The aim of RQ2 was to evaluate learner preferences between the BNC and GPT across the six criteria. By applying the global criterion weights derived in the first phase, the overall preference scores for each DDL tool were calculated to allow for a comprehensive, criterion-sensitive comparison, as presented in Table 3.
Comparison of BNC and GPT across the six effectiveness criteria (local and global scores)
Table 3. Long description
A table with three columns compares the BNC and GPT scores across six effectiveness criteria. The columns are labeled Criterion, BNC score, and GPT score. The criteria include comprehensibility of sentences, relevance to semantic learning, relevance to syntactic learning, perceived pedagogical value, accessibility for independent use, and support for autonomous learning. The BNC scores are lower across all criteria compared to the GPT scores. Notable findings include both tools scoring higheset in relevance to semantic learning, and large gaps in accessibility for independent use and support for autonomous learning. The total global score for BNC is 0.274, while for GPT it is 0.726.
As shown in Table 3, participants demonstrated a strong overall preference for GPT, which achieved a global score of 0.726 compared to the BNC’s 0.274. GPT consistently outperformed the BNC across all six criteria, with relative differences ranging from 2.17 to 3.76 times.
The gap was most pronounced in Criterion 5 (accessibility for independent use), where the GPT’s score (0.109) was 3.76 times higher than the BNC’s (0.029). Significant leads were also observed in Criterion 2 (relevance to semantic learning), the highest-weighted factor, with scores of 0.162 versus 0.068, and in Criterion 6 (support for autonomous learning) at 0.146 versus 0.041. These results show that while the GenAI tool was perceived as superior in all respects, its advantage was particularly pronounced in accessibility and autonomous learning facilitation.
4.3. RQ3. How do learners’ evaluations of the BNC and GPT differ across the identified criteria, and what distinct strengths and weaknesses emerge for each tool?
The AHP analysis revealed notable differences between BNC and GPT for each alternative across the six criteria. When examining within-group variations (e.g. differences across criteria for BNC or differences across criteria for GPT), the scores for each alternative showed notable differences across criteria. For BNC, Criterion 2 (relevance to semantic learning) had the highest score (0.068), whereas Criterion 5 (accessibility for independent use) had the lowest score (0.029). For GPT, the highest score of 0.162 was for Criterion 2 (relevance to semantic learning), and the lowest of 0.101 was for Criterion 1 (comprehensibility of sentences).
These variations suggest that, while GPT was consistently preferred overall, both alternatives exhibited relative strengths and weaknesses across specific criteria. Interestingly, both BNC and GPT performed best on Criterion 2 (relevance to semantic learning), whereas their relative weaknesses diverged—with BNC scoring lowest on Criterion 5 (accessibility for independent use) and GPT on Criterion 1 (comprehensibility of sentences). These within-group differences suggest that, although GPT was generally preferred (with even its lowest score exceeding BNC’s highest), each tool presents a unique profile depending on the specific learning objective, reinforcing the importance of context-sensitive tool evaluation in educational settings.
5. Discussion
This study examined how EFL learners prioritize multiple criteria for evaluating DDL tools, and how these learner-derived priorities inform comparisons between a traditional corpus-based concordancer and a GenAI-based concordancer. The AHP revealed that learners attached great importance to semantic relevance, autonomous learning support, and perceived pedagogical value and that the GenAI-based concordancer was evaluated as substantially more effective than the traditional BNC-based concordancer when these priorities were considered. In this section, we discuss the three main implications of these findings: (1) how the effectiveness of DDL tools can be reconceptualised through learner-derived priority structures; (2) the methodological implications of using AHP for criterion development and multi-criteria evaluation in applied linguistics; and (3) the pedagogical implications for classroom decision-making and learner-centred use of traditional and GenAI-based DDL tools.
5.1. Reconceptualising the effectiveness of DDL tools through learners’ priority structures
This study was prompted by the need to bridge theoretical discussions of the effectiveness of DDL tools with empirical evidence of learners’ experiences (i.e. emic dimensions). Although previous research has extensively examined the criteria for effective DDL tools, empirical investigations of how learners perceive them in specific pedagogical applications remain relatively scarce. Furthermore, the rapid emergence of GenAI presents a timely opportunity to re-examine these perceptions within an evolving technological landscape (Crosthwaite & Baisa, Reference Crosthwaite and Baisa2023; Flowerdew, Reference Flowerdew2025; Lin, Reference Lin2023; Mizumoto, Reference Mizumoto2023; Sun & Mizumoto, Reference Sun and Mizumoto2025a), as the integration of GenAI into the DDL approach necessitates a critical evaluation of established effectiveness criteria. Using the AHP framework, which is optimised for modelling multi-criteria, perception-sensitive evaluations of a target approach or intervention, our findings revealed that language learners prioritized semantic relevance (i.e. how useful the DDL tool is for understanding the core meaning of the target word in context) as the most important aspect of the effectiveness of DDL tools. This result aligns with DDL researchers’ proposition that DDL supports meaning-focused exploration of vocabulary (Boulton, Reference Boulton2009, Reference Boulton2010; Flowerdew, Reference Flowerdew2009; O’Keeffe, Reference O’Keeffe2021), as well as meta-analytic findings on the effects of DDL (Boulton & Cobb, Reference Boulton and Cobb2017; Lee et al., Reference Lee, Warschauer and Lee2019). The second-highest rated criterion was support for autonomous learning, which has been widely regarded as another defining feature of the DDL approach (Boulton, Reference Boulton2009; O’Keeffe, Reference O’Keeffe2021, Reference O’Keeffe2023). Consistent with previous studies of learners’ perceptions of DDL (Mizumoto et al., Reference Mizumoto, Chujo and Yokota2016), this suggests that autonomous learning support constitutes a core component of learners’ overall evaluation of the effectiveness of DDL tools.
The results further revealed that the GPT-based concordancer was rated more positively than its traditional counterpart for all six criteria, although the magnitude of preference varied across criteria. Although the present study cannot fully determine the extent to which this finding is attributable to a novelty effect, given that participants had no prior exposure to such an application at the time of the study, it nevertheless supports the call of DDL researchers to investigate the potential of GenAI-based tools for DDL (Crosthwaite & Baisa, Reference Crosthwaite and Baisa2023; Dong & Wang, Reference Dong and Wang2025; Flowerdew, Reference Flowerdew2025; Mizumoto, Reference Mizumoto2023; Sun & Mizumoto, Reference Sun and Mizumoto2025a). Among the six criteria, the largest gap was found in accessibility for independent use, suggesting that learners perceived the GPT-based tool as substantially more convenient than the traditional concordancer. This result is unsurprising, considering that the inconvenience associated with traditional concordancers has been identified as a major barrier to the implementation of DDL in L2 classrooms (Boulton, Reference Boulton2010; Crosthwaite & Baisa, Reference Crosthwaite and Baisa2023; Lin, Reference Lin2023; Sun & Mizumoto, Reference Sun and Mizumoto2025b). Taken together, these findings imply that usability-related affordances may play an important role in shaping learners’ acceptance and sustained use of DDL tools.
Regarding within-group variations, the results revealed that each tool possessed relative merits and areas of lower priority as perceived by participants. For the BNC concordancer (i.e. the traditional tool), the highest score was observed for relevance to semantic learning and the lowest for accessibility for independent use, a pattern that is broadly consistent with DDL researchers’ propositions (Boulton, Reference Boulton2010; Flowerdew, Reference Flowerdew2009). Interestingly, for the GPT-based concordancer, the highest score was also recorded for relevance to semantic learning, suggesting that GenAI can complement traditional DDL approaches as an effective concordancer for their primary purpose: to facilitate learners’ semantic learning through discovery learning. Furthermore, its scores across other areas, notably support for autonomous learning, remained robustly high and vastly superior to the traditional tool, demonstrating that GenAI can simultaneously provide the accessible autonomous learning experiences widely advocated in recent DDL literature (Crosthwaite & Baisa, Reference Crosthwaite and Baisa2023; Mizumoto, Reference Mizumoto2023; Sun & Mizumoto, Reference Sun and Mizumoto2025a). Such developments may help address long-standing barriers to independent use and extend the potential of DDL beyond formal classroom settings.
5.2. Methodological contributions of AHP to research and criterion development
While existing evaluation models gauge general technology acceptance (e.g. TAM), predict usability issues via expert heuristics (Hendry & Sheepy, Reference Hendry, Sheepy, Jablonkai and Csomay2022), or assess specific interface dimensions like accessibility and manageability (Lee et al., Reference Lee, Lee and Sert2015), AHP provides a complementary lens. Building on these approaches, AHP elicits comparative judgements, capturing how learners prioritize among multiple criteria. This perception-driven approach can serve as an alternative or supplementary methodological option, particularly in fields in which subjective evaluations play a central role. Our results revealed that although various aspects of DDL (e.g. comprehensibility of language data and convenience) have been widely discussed in the research community (Boulton & Vyatkina, Reference Boulton and Vyatkina2021; Crosthwaite & Baisa, Reference Crosthwaite and Baisa2023; Lee et al., Reference Lee, Warschauer and Lee2017; O’Keeffe, Reference O’Keeffe2021; Römer, Reference Römer2024), L2 learners may prioritize certain aspects over others. These priorities are likely to vary across different learner populations and contexts and should be considered when developing tools for DDL. Overall, incorporating AHP into existing evaluation frameworks can expand a researcher’s methodological toolkit and deepen the understanding of complex, perception-sensitive constructs. DDL is a particularly well-fitting case, given that its actual and perceived effectiveness are shaped by a wide array of interrelated factors (Boulton & Cobb, Reference Boulton and Cobb2017; Lee et al., Reference Lee, Warschauer and Lee2019; Sun & Mizumoto, Reference Sun and Mizumoto2025b).
AHP can also be effectively utilised to determine and prioritize a list of criteria through expert consultation, serving as a foundational step that enables teachers and educators to apply these criteria to research or instructional settings. Specifically, beyond merely identifying relevant constructs through a literature review, experts in the field can contribute their evaluative judgements to refine or validate these criteria. Examples include extensive lists of criteria for language program evaluation (Brown, Reference Brown1995; Norris, Reference Norris2016) and an array of variables that influence the effectiveness of task-based language teaching (Bryfonski & McKay, Reference Bryfonski and McKay2019), all of which are often collectively considered without explicit prioritization. AHP offers a structured means to address this gap, ensuring that the selected variables or criteria reflect not only theoretical relevance but also practical significance as perceived by experienced professionals. In applied linguistics, such expert-informed weighting can enhance the alignment between research design and real-world educational needs, particularly in areas such as curriculum and textbook development, language assessment, and teacher training.
5.3. Applying AHP to classroom decision-making and learner-centred instruction
In this section, we provide pedagogical implications for the use of AHP. First, when considering the implementation of a new tool or approach in the classroom, instructors may gather students’ (or those involved in students’ education) opinions about their perceived priority of criteria (e.g. convenience and effectiveness of learning the target language), as well as the prioritization of the new tool (or approach) compared with more traditional methods. Some examples of the application of this technique in L2 pedagogical contexts include (1) examining learners’ preference for teacher or peer feedback on L2 writing (Paulus, Reference Paulus1999; Yu & Lee, Reference Yu and Lee2016), (2) determining whether to maintain L2-only instruction or implement translanguaging (i.e. using multilingual and multimodal resources to enhance students’ learning; Fang et al., Reference Fang, Zhang and Sah2022), and (3) teaching academic content through the target language (i.e. content and language integrated learning; Coyle et al., Reference Coyle, Hood and Marsh2010). Furthermore, with GenAI being increasingly adopted in L2 teaching and learning, the decision to employ this emerging technology represents another area in which AHP can be applied effectively. Our data on the GPT-based concordancer serve as a well-fitting example in this regard, addressing the call from DDL proponents to consider whether new technologies (i.e. ChatGPT) should be utilised when developing tools for a pedagogical approach that has persisted for over 30 years (Crosthwaite & Baisa, Reference Crosthwaite and Baisa2023; Flowerdew, Reference Flowerdew2025; Lin, Reference Lin2023). Given that our results using AHP are exclusively based on learners’ perceptions, it seems necessary to complement these perception-based findings with data collected from experimental designs, allowing for comparison of learning outcomes under two different conditions (i.e. GPT-based and more traditional methods; see Sun & Mizumoto, Reference Sun and Mizumoto2025a, for a recent example).
Meanwhile, although our example demonstrated that one particular tool (i.e. GPT) was largely favoured over the other across the selected criteria, it is possible that students may differ in their preferences for specific criteria when evaluating any given tool or approach (see Lee & Lo, Reference Lee and Lo2017, for students’ preference for English-only instruction and code-switching as teachers’ instruction). Such variations in individual preferences could complicate the decision-making process for instructors aiming to adopt a single tool or approach, as the chosen option may not align with the diverse needs and learning styles of all students. In such situations, the application of AHP can be particularly valuable, as it can reveal the reasons underlying each student’s preference for one tool (or approach) over another. By analysing these insights, instructors, for example, can gain a deeper understanding of student priorities of the diverse criteria (e.g. comprehensibility of the target materials, relevance to learning, pedagogical value, to use our own example). Therefore, such a data-driven method enables instructors to consider differentiated or individualised instruction that caters to the varying needs of students.
5.4. Limitations of the study
Despite its contributions, this study had several limitations. First, although AHP is a non-inferential statistical technique less constrained by statistical power requirements, our small sample size (N = 35) limits the stability and generalizability of the findings. The primary aim of the present study was to examine the feasibility of adopting AHP to address research questions in DDL that have been difficult to investigate owing to the lack of an appropriate methodological framework, rather than to make strong population-level claims. In this sense, a modest sample size does not undermine the methodological contribution of demonstrating how learner-derived priorities can be modelled and incorporated into the evaluation of DDL tools. However, future research with larger and more diverse samples is required to replicate and extend the present findings regarding the effectiveness of DDL tools from a learner’s perspective.
Second, because AHP is not an inferential procedure, it cannot address questions about statistical predictions or group differences on its own. Follow-up inferential analyses, such as regression models using AHP-derived weights as predictors, could complement the present results by addressing additional questions – for example, whether learners who prioritize semantic relevance as the most important criterion tend to achieve greater language learning gains or whether learners who place higher value on autonomy support benefit more when assigned to a GenAI-based DDL tool. Owing to space constraints and the limited sample size, we did not pursue such analyses in the current study. Therefore, we encourage future studies to combine the AHP with inferential statistical methods to explore these issues in greater detail.
Third, while the geometric mean was employed to aggregate individual pairwise judgements, a standard procedure in AHP to ensure the reciprocal property of the group consensus matrix, this approach inevitably obscures the variability and dispersion of individual preferences. Consequently, the degree of consensus among learners cannot be fully assessed from aggregated results alone.
Fourth, no sensitivity analysis was conducted in the current study. Because small changes in weights can occasionally affect the final rankings in AHP, these results should be interpreted cautiously.
Finally, we chose the BNC as the traditional DDL resource. Although it is a foundational resource in corpus linguistics, more recently developed, learner-friendly interfaces (e.g. BNClab, SkELL) might have yielded different results regarding learner perceptions of accessibility for independent use. Therefore, the study findings, particularly concerning the perceived difficulty of traditional DDL, should be interpreted with the understanding that the specific user interface may have influenced the participants’ responses.
6. Conclusion
This study used the AHP framework to investigate how EFL learners prioritize the criteria for evaluating DDL tools and how such priority structures relate to tool preference. The findings highlight that learners attach differentiated importance to various aspects of DDL effectiveness, and that these priority patterns are closely associated with their judgements of tool effectiveness. The study contributes by being one of the first to examine EFL learners’ evaluations using literature-based criteria, showing how these inform learners’ decision-making regarding DDL tool effectiveness. Based on these findings, we suggest that developers and educators consider the theoretical rationales discussed in the DDL literature and learner-valued criteria when selecting or designing DDL tools. This study further contributes to the growing body of research on GenAI-mediated DDL by highlighting its potential from learners’ perspectives and underscoring the need for its integration into the domain.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/S0958344026100512
Data availability statement
Data will be made available upon reasonable request.
Acknowledgements
We thank the editor and reviewers for their valuable comments on this manuscript.
Authorship contribution statement
Hansol Lee: Methodology, Conceptualization, Formal analysis, Writing – original draft; Jang Ho Lee: Conceptualization, Writing – original draft, Writing – review & editing.
Funding disclosure statement
This work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF-2025S1A5A2A01004947).
Competing interests statement
The authors declare no competing interests.
Ethical statement
Participants received a detailed explanation of the aim and procedures of the study, were clearly informed of their right to withdraw at any time without disadvantage, and provided written informed consent before taking part in the study.
GenAI use disclosure statement
ChatGPT 5.1 was used solely for checking the English grammar of some sentences. No text generation or data analysis was performed, and the authors take full responsibility for the final manuscript.
About the authors
Hansol Lee, a professor at the Korea Military Academy, earned his PhD from UC Irvine and served as a visiting professor at the University of Oxford. Integrating applied linguistics and educational psychology, he investigates learning mechanisms via advanced quantitative frameworks. His work is published in premier journals across education, linguistics, and psychology.
Jang Ho Lee is a professor of English education at Chung-Ang University. His research focuses on GenAI-mediated language instruction and EMI. He has published more than 80 journal articles and has served as a visiting professor at the Hong Kong Polytechnic University and at King Mongkut’s University of Technology Thonburi.
Open access
