Scholars are in agreement that replication studies are essential in the field of computer-assisted language learning (CALL), as well as in its cognate disciplines, if our aim is to establish a more robust evidence base and to strengthen the theoretical background of CALL. In order to justify the widespread adoption of technology in as well as out of the foreign language classroom, replication studies are essential, not just to enhance the reliability of empirical findings but also to ensure that at least a significant proportion of learners can benefit from the introduction of CALL in the curriculum. But despite this often-invoked need for more replications, the number of published replication studies remains relatively low (e.g. Marsden, Morgan-Short, Thompson & Abugaber, Reference Marsden, Morgan-Short, Thompson and Abugaber2018; McManus, Reference McManus2022). Why are these calls for more replications not heeded more widely? We easily understand that using a standardized language proficiency test is the better choice when grouping language learners into levels compared to using one that was created by the researcher for one study only; the results will be reliable and comparable beyond the small group tested. Replication studies simply take this principle a step further and apply it across most of the methodology of a study. Notwithstanding several calls for more replications specifically in CALL within the last decade (e.g. Handley, Reference Handley2014; Plonsky, Reference Plonsky2015; Smith & Schulze, Reference Smith and Schulze2013) as well as in other areas of second language acquisition (SLA), perhaps as few as 1 in 400 studies are ever replicated (Marsden et al., Reference Marsden, Morgan-Short, Thompson and Abugaber2018). Even proponents of more replications recognize that there is still a certain lack of prestige (McManus, Reference McManus2022) that may well contribute to the fact that, as a field, CALL does not have the number of replication studies it should have. Perhaps too many students are still taught that finding the “gap in the research” is essential to their own first research project, and that this gap cannot be filled by a replication study if they want to make their mark as aspiring researchers in their field. McManus’s (2022) survey shows that academic respondents agreed about the important contribution replications make to their field, but views on originality, innovation and the perceived contribution of replication studies to the author’s academic career progression were less clear. However, as long as journal editors accept submissions of replication studies – and, clearly, those who publish special issues on this topic do – such studies are an excellent tool to further one’s academic credentials. Marsden et al. (Reference Marsden, Morgan-Short, Thompson and Abugaber2018) even suggest that replication studies have an advantage over what they call “initial studies” where citations are concerned.
In fact, the conditions for replication studies, both for conducting such studies and for publishing them, have never been better. There are excellent and easily accessible databases such as IRIS (https://www.iris-database.org/), detailed instructions on how to conduct replications (Porte & McManus, Reference Porte and McManus2019), and a small but constantly growing list of published replication studies (e.g. in the journal Language Teaching) that can be consulted to see successful replication publications. The various calls for more replication studies and the increased perception of replication as necessary for the discipline to make progress in its theoretical basis and empirical findings all contribute to pull replication out of its overlooked existence and into the spotlight.
Nonetheless, there remain a number of challenges for any researcher who is considering conducting a replication study specifically in CALL. In contrast to traditional SLA studies, where software plays no central role in the methodology, the defining characteristic of CALL is the use of software and/or hardware in the service of language learning. With the mean length of time between a study and its replication at over six years (Marsden et al., Reference Marsden, Morgan-Short, Thompson and Abugaber2018), it is almost inevitable that anyone planning a replication study in CALL needs to carefully consider how to address the often fast-paced technological development in this time gap. Even gaps of one or two years can see popular apps go out of fashion or being replaced by newer versions with updated features. CALL applications and platforms most likely will have evolved, become obsolete, or look hopelessly dated to the young adults who typically serve as participants in CALL studies. As well as the software, the typical human participants in replication studies will have changed as well, if only in terms of their digital habits and their day-to-day exposure to various apps. They are used to fresh apps appearing on their mobile phones at frequent intervals and ever more functions in the more established programmes. The changes in the availability and preferences of different technologies such as search engines or social media apps influence the way learners interact with these tools and therefore also influence their views of tools used for language learning. If learners play computer games in their spare time, they are accustomed to seeing high-quality graphics and enjoying the products of a multibillion-dollar entertainment industry that no CALL programme can ever match. Replication studies in CALL are therefore more demanding compared to studies in non-CALL contexts. Both the software and the habits of humans vis-à-vis software change quite rapidly; the tools change more quickly than the scholarly publishing schedule can turn around studies that might merit replication, and the potential study participants might struggle to remember the apps they used so frequently just a few years earlier.
Given this twofold extra challenge for replications in CALL, we can say that exact or very close replications (in the sense used in Porte & McManus, Reference Porte and McManus2019) are not realistic in CALL, as pointed out by Chun (Reference Chun2012) some time ago. What is feasible, as well as sensible and necessary, are approximate and conceptual replications. On the continuum of replications between close at one end and conceptual at the other, there is still a huge amount of variety possible, and the studies assembled in this special issue of ReCALL exemplify a modest range of this variety. The authors of the studies in this special issue have accepted the twofold challenge of undertaking a replication study in the field of CALL and have produced a range of replications, with varying time gaps, frequently expanding the geographical reach of the findings of the initial study, often expanding on the methodology by adding a further research instrument, or improving the methodology in a different way, and thus contributing to more transparency and increased robustness of findings in CALL.
The study by Joanna Kic-Drgas, Gölge Seferoğlu, Ferit Kılıçkaya and Ricardo Pereira is a multinational replication of the survey on open educational resources (OERs) by Pérez-Paredes, Ordoñana Guillamón and Aguado Jiménez (Reference Pérez-Paredes, Ordoñana Guillamón and Aguado Jiménez2018), enriching the results of this initial study (in two countries) to three more countries and adding a similar number of respondents. Despite the gap of a few years since the original publication and the concomitant growing prevalence and use of mobile phones during those years, the authors decided to use the same questionnaire. This publicly available instrument probes the knowledge and use of OERs by in-service teachers. The results broadly confirm the findings of the earlier study in terms of the tools that are best known and most often used, while also showing some differences between countries. The best-known OERs are still spellcheckers and online dictionaries, even if these are now used outside of computer labs; thus, the need to make OERs better known and more easily accessible to pre- and in-service teachers remains.
The two studies that follow both explore student satisfaction (or dissatisfaction) in an online study environment. First, Hye Won Shin and Sarah Sok explore the model for the interaction between student engagement, satisfaction and perceived learning originally proposed by Gray and DiLoreto (Reference Gray and DiLoreto2016). Gray and DiLoreto had surveyed students in the United States and found that the structure of a course had a significant influence on student satisfaction. In the replication, Korean students answered two open-ended questions in addition to those in the original study. In the intervening years, the online teaching environment shifted and included the arrival of Zoom, something reflected in the replication study. The findings here do not fully support the original model, with interaction playing a more important role in the case of the Korean learners. In addition, this replication contributes to the data that shows a clear positive relationship between course structure and instructor presence on the one hand and student satisfaction on the other. But even if the course structure is perfect and the instructor ever-present, online and blended learning courses have higher dropout rates than in-person courses; and such a group of dropouts is the focus of the next paper. Elke Stracke, Giang Hong Nguyen and Vinh Nguyen’s study represents the type of replications that have a partial overlap of authors. According to Marsden et al. (Reference Marsden, Morgan-Short, Thompson and Abugaber2018), about a third of replications involve (some of) the author(s) of the initial study, a starting point that obviously facilitates full access to the study instruments. In this case, the initial study, Stracke (Reference Stracke2007), focused on three students who had dropped out of blended learning classes at a German university, long before the pandemic and the subsequent massive shift to online and blended learning. In the recent replication, the authors explore the same questions in a contemporary Vietnamese context. This longitudinal perspective shows that the shift away from paper-based materials towards work and study online is not a problem for today’s learners anymore. However, now as then, students cite the lack of complementarity between the different parts of their blended course as one of their main reasons for dropping out of blended or fully online courses. Integration of the various parts of a language course and careful planning of feedback therefore remain as key elements of a successful blended language course, as evidenced by both of these qualitative studies.
The fourth study also expands earlier qualitative findings to two more countries. Cristina A. Huertas-Abril and Barbara Muszyńska replicated Lee’s (Reference Lee2019) study on the use of a digital game, Her Story, to foster creativity and increase the motivation of Korean EFL students. The authors of the current study used the same game with learners in Spain and Poland. Replicating CALL studies on gaming is particularly challenging, as games can look dated even more quickly than other software; hence, a short time gap between the initial study and the replication is helpful for avoiding a retro look and feel of the game for the learners. In order to increase the validity of the findings, the authors here opted to replace the original questionnaire with externally validated instruments to test the students’ creativity experiences. Their results show that learners benefited from playing the game, independently of their sometimes negative views of their own creativity, when prompted later to write a creative piece based on the game they had played earlier. The study confirms findings from Lee (Reference Lee2019) and other studies that show that digital game-based learning increases learners’ motivation, at least as long as the game is carefully chosen.
The next study, by Yanxia Yang, Xiangqing Wei, Ping Li and Xuesong Zhai, also looked at writing, but this time at the role of machine translation in the CALL classroom and the influence it has on the revisions to students’ writing. It is an approximate replication of Lee’s (Reference Lee2020) mixed-methods study, which showed a marked decrease of the number of errors in that study’s Korean participants’ writing after they had revised their texts with the help of machine-translated texts. But this improvement in terms of errors is not accompanied by an increase in lexical complexity or density. Participants of the replication study were Chinese EFL learners of similar proficiency to those in Lee’s (Reference Lee2020) study. The authors used screen recordings to replace the reflection paper from the initial study and also refined the quantitative analysis by employing Coh-Metrix instead of Lee’s simpler analysis. The findings confirmed the clear effect of machine translation use reducing the number of errors, and further found statistically significant, positive effects on the quality of the students’ writing on a number of indicators. Together with the findings from the qualitative part of the study, this leads the authors to call for better training on how best to integrate machine translation tools in language learning and teaching contexts.
The final paper, by Dennis Foung and Lucas Kohnke, uses the data from the original paper by Łodzikowski (Reference Łodzikowski2021), who investigated the association between allophonic transcription and phonological awareness in Polish learners of English using a regression model. The authors of the present study reanalysed the data after removing two of the correlated variables, thus reducing multicollinearity and simplifying and improving the model. The resulting model will be easier to test with a new set of students using the same transcription tool to improve learners’ pronunciation. A further replication could thus contribute to a firmer evidence base for the benefits of such teaching tools, even given the relatively modest number of participants in this study. More data would be needed for a solid regression analysis, and proposals are made here how this could be achieved, even if the number of learners who use such a tool is likely to be limited, as is often the case in classroom studies.
The studies assembled in the special issue of ReCALL demonstrate the opportunities for further research and the richness of replication studies in the area of CALL, with a large variety in terms of research methods, research instruments used, group sizes, distance to the time of the original study and number of authors involved. The authors often explicitly worked toward an improvement of the methodology employed in the replicated study and as a result contribute to advancing the field of CALL by revisiting findings from the published literature and exploring their robustness in new contexts. Let’s hope for more replication studies!
