1. Introduction
Historically, teaching pronunciation has come in and gone out of fashion, at times marginalised and at other times the centre of language teaching (Pennington & Rogerson-Revell, Reference Pennington and Rogerson-Revell2019). While English as a second language (ESL) learners consider pronunciation extremely valuable (Derwing & Rossiter, Reference Derwing and Rossiter2002), instructors often lack the training, resources and confidence to teach it (Bai & Yuan, Reference Bai and Yuan2019; Pennington & Rogerson-Revell, Reference Pennington and Rogerson-Revell2019), and it tends to play an insignificant role in the curriculum (Gilbert, Reference Gilbert2012). ESL learners often have a restricted conception of pronunciation (i.e. striving not for intelligibility, but to pronounce English like a native speaker; Levis, Reference Levis2005; Scales, Wennerstrom, Richard & Wu, Reference Scales, Wennerstrom, Richard and Wu2006; Thomson & Derwing, Reference Thomson and Derwing2015). This is a difficult, if not impossible, task because their pronunciation is impacted by the phonological and phonetic features of their first languages (Saito, Reference Saito2021). In addition, ESL learners often communicate among themselves, so non-native pronunciation is “normal” (Pennycook, Reference Pennycook2017). Arguably, therefore, pronunciation instruction should emphasise more realistic goals, such as communicative success (e.g. Derwing, Munro, Foote, Waugh & Fleming, Reference Derwing, Munro, Foote, Waugh and Fleming2014; Isaacs, Trofimovich & Foote, Reference Isaacs, Trofimovich and Foote2018).
Some scholars have proposed that ESL learners should receive explicit pronunciation instruction (Saito & Plonsky, Reference Saito and Plonsky2019), such as articulatory and/or auditory information about L2 segments (“minimal units of sound defined in phonetic terms”; Pennington & Richards, Reference Pennington and Richards1986: 208) and suprasegmental features (“speech sounds larger than phonemes”; Parker & Graham, Reference Parker and Graham2009: 218), as this can improve their pronunciation (e.g. Derwing, et al., Reference Derwing, Munro, Foote, Waugh and Fleming2014; Gao & Weinberger, Reference Gao and Weinberger2018; Kissling, Reference Kissling2013; Munro & Derwing, Reference Munro and Derwing1995; Rogerson-Revell, Reference Rogerson-Revell2012, Reference Rogerson-Revell, van den Doel and Rupp2014; Trofimovich, Kennedy & Blanchet, Reference Trofimovich, Kennedy and Blanchet2017). Studies have found that segmental phonemes may be easier to learn and teach (Levis, Reference Levis2005; Saito, Reference Saito2014), although suprasegmental speech attributes play a significant role in comprehensibility and intelligibility (e.g. Hahn, Reference Hahn2004; Isaacs & Trofimovich, Reference Isaacs and Trofimovich2012; Kang, Rubin & Pickering, Reference Kang, Rubin and Pickering2010; Saito & Saito, Reference Saito and Saito2017). Lee, Jang and Plonsky’s (Reference Lee, Jang and Plonsky2015) meta-analysis of 86 studies found that explicit pronunciation instruction had a medium-sized effect. Similarly, Sakai and Moorman’s (Reference Sakai and Moorman2018) meta-analysis demonstrated that it led to moderate improvement in the perception and production abilities of learners. However, review articles on the subject are far from conclusive; they often contradict each other and focus on different issues (Zhang & Yuan, Reference Zhang and Yuan2020).
Generally, phonological awareness is related to the ability to pronounce words in the L2 accurately and is believed to be affected by one’s understanding of the phonologies of the L1 and the L2 (Flege & Bohn, Reference Flege, Bohn and Wayland2021) and learner-related factors (Derwing, Reference Derwing, Garrett and Cots2017). The use of the International Phonetic Alphabet (IPA) can help ESL learners understand the English phonological system better and increase their awareness of spelling-to-pronunciation correspondence (Mompean, Reference Mompean2015). Recently, using IPA to teach phonological awareness and sound structure (Wagner & Torgesen, Reference Wagner and Torgesen1987) has gained popularity (Łodzikowski, Reference Łodzikowski2021). This phonemic transcription system represents speech sounds using unique symbols. It helps beginning learners reduce segmental errors that impact intelligibility (e.g. substituting consonantal phonemes; Gao & Weinberger, Reference Gao and Weinberger2018). A more systematic and narrow form of transcription – allophonic transcription – helps advanced learners improve their pronunciation of salient features, such as fortis plosives or word-initial epenthesis.
The phoneme /t/, which has two allophones, [t] and [tʰ], illustrates the difference; these two allophonic variants are features of English. This phoneme is more complex than others because its sound changes depending on its position in the word. The acquisition of allophonic features is essential for L2 learners. Reinisch, Juhl and Llompart (Reference Reinisch, Juhl and Llompart2020) demonstrated that it affects L2 learners’ ability to identify regional speakers and varieties. Shea and Curtin (Reference Shea and Curtin2010) found that L2 learners can acquire allophones based on contextual cues. Evidence from these studies supports the idea that learners need to acquire pronunciation at both the phonetic and allophonic levels.
However, the teaching of allophonic transcriptions continues to be rare in the ESL classroom as (1) teachers lack the time and resources (e.g. transcription tools; Bai & Yuan, Reference Bai and Yuan2019; Pennington & Rogerson-Revell, Reference Pennington and Rogerson-Revell2019) and (2) learners find that transcription exercises without detailed feedback do not aid in their language acquisition process (Lecumberri & Gallardo, Reference Lecumberri, Gallardo, García Mayo and García Lecumberri2003). Several automated transcription tools for teachers are available, such as IPAtranscriptor (Braun, 2020) and Web Transcription Tool (Lecumberri & Gallardo, Reference Lecumberri, Gallardo, García Mayo and García Lecumberri2003), which can be used for phonemic transcription activities and selected speech processes (Lecumberri & Gallardo, Reference Lecumberri, Gallardo, García Mayo and García Lecumberri2003). Although these tools provide some feedback, the instructor must provide a reference transcription for each activity. Currently, few comprehensive allophonic transcription tools are available for instructors: some have minimal features, while others only cover a few allophonic processes.
The present study aimed to replicate a study by Łodzikowski (Reference Łodzikowski2021). This study relied on a custom transcription tool designed for Polish learners of English. Previous studies on Polish learners have adopted a general approach. For example, Rojczyk (Reference Rojczyk2019) examined how Polish learners use allophonic cues to identify word boundaries, finding that they did so with 59% accuracy, comparable to results achieved by Spanish and French learners of English. Other scholars have focused on how Polish learners acquire allophones, especially stops (Rojczyk, Porzuczek & Bergier, Reference Rojczyk, Porzuczek and Bergier2013; Schwartz, Balas & Rojczyk, Reference Schwartz, Balas and Rojczyk2014). They consider stops to be particularly important for Polish learners because stop release is mandatory in Polish. Rojczyk et al. (Reference Rojczyk, Porzuczek and Bergier2013) found that Polish learners can imitate stops properly, but their ability to do so is affected by distraction. (In their study, the distraction was a flashing digit that appeared on a screen after the participants heard the model pronunciation and before they imitated the stop.) Schwartz et al. (Reference Schwartz, Balas and Rojczyk2014) found that learners can be affected by their first language in the production of stops, but the mechanism for these patterns needs further research. A more recent study developed machine learning algorithms to assist Polish learners with using stop-related allophonic features and aspiration (Piotrowska et al., Reference Piotrowska, Czyżewski, Ciszewski, Korvel, Kurowski and Kostek2021). These studies show that certain allophones are particularly important for Polish learners of English, which motivated us to replicate Łodzikowski’s (Reference Łodzikowski2021) study.
Moreover, as pronunciation is an under-researched area in computer-assisted language learning (CALL), we hope that this replication study will provide insights into the value of allophonic transcription tools and their efficacy in improving the phonological awareness of ESL students. By applying the results of Łodzikowski’s (Reference Łodzikowski2021) study to CALL, we will provide additional insight and give greater validity to the findings. ESL learners need to understand allophonic features to develop their pronunciation. This study will address how this can be achieved by developing specific, tailor-made tools, which can be applied in different contexts by future researchers.
1.1 Replication studies: A review
Replication studies in applied linguistics can be divided into two groups: internal replication and external replication (Porte & McManus, Reference Porte, McManus, Porte and McManus2019). External replication adopted the methodology in the original study to produce new data with “new contexts” and “other participants”. Internal replication, which is also called “exact replication”, is to “re-examine the methods used and conclusions drawn”. For example, Nicklin and Plonsky (Reference Nicklin and Plonsky2020), in an L2 and applied linguistics study, obtained the data set from 104 studies and reanalysed the data with a better outliner detection strategy, and this is one example of internal replication. Luef (Reference Luef2022) obtained a new data set to replicate the research of Siew and Vitevitch (Reference Siew and Vitevitch2020) on phonological networks, and that is an example of external replication. Although both external and internal replication studies have achieved some success, conducting an exact replication as an internal replication by another researcher is rarely possible because most research is conducted in a highly controlled setting (Crandall & Sherman, Reference Crandall and Sherman2016; Porte & McManus, Reference Porte, McManus, Porte and McManus2019). Past studies may only validate a previously published model with new data (Zeigler, Muzy & Kofman, Reference Zeigler, Muzy and Kofman2018); however, this process rests on the assumption that the model is a “reliable measure” (Cipriano, Barnes, Kolev, Rivers & Brackett, Reference Cipriano, Barnes, Kolev, Rivers and Brackett2019: 10). Therefore, before validating a model, one must verify it; one must reproduce it as an exact replication using the original data set to confirm that it is reliable. The present exact replication of Łodzikowski’s (Reference Łodzikowski2021) study is designed to achieve this purpose: to verify the models presented in the original study.
1.2 The original study and suggested approaches to replication
Łodzikowski’s (Reference Łodzikowski2021) study included two research questions: first, how advanced ESL learners used a supplementary allophonic transcription tool and, second, how their usage of the tool was associated with their declarative phonological awareness. The participants in the original study were 55 Polish English-major undergraduates who used an English transcription tool to supplement a two-semester course on English phonetics and phonology.
The tool allows learners to enter phonetic transcriptions in IPA and see allophonic features. For example, if learners enter /ben/, the tool will show [b˚e˜n]. The tool shows 13 allophonic features by default. These features were based on Cruttenden’s (Reference Cruttenden2014) study, which was also used to develop the course taken by the students in the study. Students accessed the tool through the university learning management system. It has since been posted on a publicly accessible repository (https://bit.ly/phontrans_webapp; Łodzikowski & Aperliński, Reference Łodzikowski and Aperliński2016), and the public can utilise the tool with minimal technical support.
The tool was introduced as a supplementary resource for the two-semester course and the students were not required to use it. However, they were required to complete various transcription activities and assessments, including post-class quizzes, midterm tests and final assessments, all of which required an understanding of allophonic features. The activities and assessments may have incentivised the students to use the tool. A free analytics application, Piwik, was installed to track students’ use of the tool (Matomo, 2022). The original study provided sufficient details on how the tracking logs were able to record learners’ use of the tool with reasonable accuracy.
Data visualisation was employed to examine the usage data (entries across time and among learners) to answer the study’s first research question regarding how the tool was used. Łodzikowski (Reference Łodzikowski2021) reported that 91% of the 55 learners used the tool at least once. A total of 3,119 entries were made using the tool over 312 visits. Over the two-semester course, there was a substantial increase in the use of the tool near quizzes and midterm tests and a long period of low use because of the holidays. It is important to note that the raw data for the time-based analysis was not made available, so the current study cannot conduct time-based analysis as the original study did. Besides time-based and learner-based analyses, Łodzikowski conducted brief entry-based analyses. Łodzikowski reported a total of 3,119 entries, with 68 being non-words (i.e. IPA transcriptions entered by learners that were not English words). Among the English word entries, there were 1,105 distinct words and 3,051 total words (a word could be entered multiple times). The 15 most frequently entered words were words on the class transcription worksheets.
To answer the second research question, the original study established three regression models to examine the association between the use of the tool and phonological awareness. Phonological awareness was the target variable for all the regression analyses. This was operationalised using the results of the midterm test (Models 1 and 2) and phonotactics quiz scores (Model 3). A full list of variables and results for three models can be found in Appendices 1a–1c (see supplementary material). The first model included 11 predictors related to learner use and learner background and used the midterm test score as the target variable. It also included the grouping of the students as a random variable. Along with other variables, this mixed-effects model explained 51% of the variance (R 2 = 0.51) in phonological awareness, with a student’s secondary school writing exam score being a statistically significant predictor (b = 0.93, p < 0.05). The second model used the same dependent and 11 independent variables but removed grouping as a random variable, making Model 2 a fixed-effects model. Model 2 explained 56% of the variance in phonological awareness, and a student’s secondary school writing exam score was once again a statistically significant predictor (b = 0.93, p < 0.05).
Model 3 was established with the phonotactics quiz score as the target variable and the same list of predictors as Model 2, with the addition of a binary variable (whether learners had entered three or more non-words). Model 3 explained 30% of the variance in phonological awareness, with a student’s secondary school oral English score (b = 3.01, p < 0.05) and mean duration of visit (to the tool) (b = 0.02, p < 0.05) being statistically significant predictors. Although detailed data processing and cleansing procedures were reported, no details of the assumption-testing procedures were provided in the original study (i.e. the main text), despite indications in the code posted on a separate site. The author concluded that using the tool, as well as visiting it for an additional day, could boost a student’s midterm test score. The study also found that entries of non-words could help develop learners’ English phonotactics.
2. Methodology
2.1 Research design: Exact replication study
The current study positions itself as an exact replication study that adopted the original data set from Łodzikowski (Reference Łodzikowski2021) and followed all of the protocols as closely as possible to recreate the results. It is important to note that although the purpose of an exact replication study is to confirm the results of an original study (Morrison, Reference Morrison2022), we do not argue that the findings of the original study were invalid or unfounded. The purpose of the current study was to extend the original study by discussing some of the limitations that were not emphasised in it and examining some of the findings that could have been further discussed.
Like the original study, the present study examined learner usage data and then developed models to examine the association between the use of the tool and phonological awareness. However, the current study differed from the original study in three ways: (1) we did not recreate the time-based results, (2) we examined the allophonic features of entries, and (3) we revised the models to include fewer variables. See Table 1 for a summary.
Similarities and differences of original and current study
2.2 Data retrieval, processing and cleaning
The original study provided two data sets: (1) data on the usage of the transcription tool, organised by the learners; and (2) data about the entries (i.e. transcriptions presented as words) produced by the learners. The data sets were available in the folder “Data” on the website of the tool (https://bit.ly/phontrans_analysis). The previous section has provided more details about the procedures and participants in the original study.
The first data set, recording learners’ use of the transcription tool, did not require any processing. The original study reported that detailed screening and cleansing procedures were used on the data concerning the use of the tool by the 55 learners, so no further processing or cleansing was performed. The data set included variables for each learner that were relevant to the current study, including gender, prior achievements (e.g. secondary school writing and oral exam scores), visiting the tool at least once (binary), raw number of visits (i.e. visiting twice on Monday counted as two), number of visits on distinct days (i.e. visiting twice on Monday counted as one), raw number of inputs (i.e. entering the same word twice counted as two), number of distinct inputs (i.e. entering the same word twice counted as one), mean duration of visit (in seconds), number of visits within seven days before assessment, number of visits within one day before assessment, entering three or more non-words (binary), midterm test score, and phonotactic quiz score.
The second data set had to be processed to examine the allophonic features. We removed the entries with non-words (as no details were provided for them) and those that were only entered once. We then converted the remaining 541 words into IPA using the online version of the Cambridge English Dictionary (https://dictionary.cambridge.org/) to create a transcription (with the UK pronunciation). Next, we entered the transcription into the tool used in the original study to obtain an allophonic transcription. Finally, we labelled each allophonic transcription with the relevant allophonic features. For example, the word “bead” was first transcribed as /bid/. Then, the tool converted that to [b˚id˚] with one feature: devoicing. To ensure that we did not misunderstand the diacritic, we examined the original Java file transcriptor.js as part of the tool. As the second data set was subject to exploratory analysis (i.e. data visualisation), no further processing or cleansing was needed.
2.3 Data analysis
As shown in Table 1, three types of data analysis were conducted in the current study to replicate the results of the original study. To initially examine the usage data by learner, simple descriptive statistics were produced using MS Excel; the transcription entries were processed and analysed. To examine the association between phonological awareness and the use of the transcription tool, regression analyses were conducted.
To ensure the validity of the regression analysis, we first checked whether there were enough samples, according to the criterion suggested by Tabachnick and Fidell (Reference Tabachnick and Fidell2013). The required sample size was 50 + 8 * the number of predictors. With only 55 learners, the original study did not have a large enough sample. An insufficient sample size results in the overfitting of the data (i.e. inability to generalise the results to the population). Although we could have decided to end the study here, we believed it was still meaningful to examine the regression models using an exploratory approach.
Next, we conducted three rounds of analysis with the given target variable and predictors (plus the random variable in Model 1). See Appendices 1a–1c for details about these models. As in the original study, we adopted the “enter” approach, including all of the predictors (i.e. not excluding any for non-significance). To further confirm the validity of the analyses, multicollinearity was examined by constructing a correlation matrix with all of the quantitative variables. We checked the sample size and confirmed the presence of multicollinearity while running the analysis to give us a better understanding of the results of the original study. We adopted the criterion suggested by Nimon (Reference Nimon and Frey2018) – r > 0.5 – when examining the correlation matrix. Unfortunately, two of the variables (raw number of visits and number of visits on distinct days) were highly correlated with other variables. Therefore, we decided to remove those two variables and rerun the three models with the same parameters. Details about these decisions can be found in the Results section. We examined the normality of the residuals and found no irregularities in any of the models. To evaluate the models, we examined the R 2, the overall model adequacy, for Models 2 and 3, followed by the significance of the individual predictors.
All of the initial regression analyses were conducted in R (Version 4.0.3; The R Foundation, n.d.). The original study also used R for data analysis and its codes were posted online (https://bit.ly/phontrans_analysis). We believed that the same purposes and results could be achieved with the built-in lm() (for fixed-effects model) command and the lme() command (for mixed-effects model) from the nlme library (Pinheiro et al., Reference Pinheiro, Bates, DebRoy, Sarkar, Heisterkamp, Van Willigen and Ranke2022). The analyses were conducted using those commands instead of those used in the original study. Also, the glance() command from the broom library (“Introduction to broom”, 2022) was used to reproduce the R 2.
3. Results
3.1 Transcription study patterns
As this was an exact replication study, its first aim was to examine how advanced ESL learners used the supplementary allophonic transcription tool. We examined some learner-based usage data and then entry-based data using the data sets provided by the original author. As reported in the original study, 90.90% (n = 50) of the learners used the tool at least once. We found that the average number of transcriptions entered per student was 51.78, with the highest being 240 and the lowest 1 (among those who did use the tool). There were 270 visits, with a median time per visit of 16.02 minutes. These results seemed to differ from those of the original study. A summary is presented in Table 2. A further examination of the usage data showed that three learners (in addition to the five who did not use the tool) spent a total of 0 seconds on the tool in their one or two visits. The original paper reported that 16 visits lasted 0 minutes, but we found only four visits that lasted 0 minutes. Generally, these results align with those of the original study despite minor discrepancies, which did not come as a surprise to the authors.
Summary of allophonic features of entries
To better understand the usage of the tool, the current study attempted to extend the past study by further analysing the transcriptions that the students entered. This data set was posted, along with the usage data; it contained 3,051 entries comprising 1,100 words and 68 entries of non-words. To extend the original study, we entered the transcriptions into the tool again, excluding non-word entries and words entered only once (n = 559), and analysed the allophonic features identified by the tool. The remaining 541 words (2,451 entries) had an average of 1.99 allophonic features each; 7.76% had no allophonic features (e.g. cheap [ʧip] and get [gɛt]). Some had more, such as consequence [ˈk̠ʰɑ̃nsə̥̆kʷwə̃̆ns], with seven features, and transcriptor [ˈt̠ɹ̥æ̃nˌskɹĭp̚təɹ], with six.
Among the 13 default features (Table 2), the one found in the highest proportion of words was devoicing (45.29%), followed by pre-fortis clipping (41.03%). No entries had GOAT allophony (from [əʊl] to [ɒʊɫ]) and fewer than 2% had nasal plosion (1.66%) or lateral plosion (1.48%). A summary is provided in Table 2. These findings provide preliminary indications of how learners use the tool and deserve further discussion.
3.2 Association between tool use and phonological awareness level
The second aim of the study was to examine the association between the use of the tool and phonological awareness. As reported in the Data Analysis section, the sample size seemed too small to produce reliable results, but we chose to reproduce the results using the data available. In both the original and present studies, three models were tested; they contained predictors related to tool usage and midterm test scores as the dependent variable (see Appendices 1a–1c for details). The first was a mixed-effects model that included 11 predictors and the grouping of the students as a random variable. The second was a fixed-effects model that used the same 11 predictors as the first model. The third model included one additional predictor, whether the learners had entered three or more non-words, for a total of 12 predictors. A summary of the three models and the strength of their predictors can be found in Appendices 1a–1c.
The first mixed-effects model showed an adequate R 2 that explained 51% of the variance. The second fixed-effects model produced similar results, with an R 2 of 56%. In both models, a student’s secondary school writing exam (B = 0.93, p < 0.05), the number of visits on distinct days (B = 2.62, p < 0.05) and the number of distinct inputs (B = 0.16, p < 0.05) were statistically significant predictors of the midterm test score. In the third model, which included non-word entries, a student’s secondary school oral exam score (B = 3.01, p < 0.05), the number of visits on distinct days (B = 0.47, p < 0.05), number of distinct inputs (B = 0.16, p < 0.05) and the mean visit duration (B = −0.52, p < 0.05) were statistically significant predictors. ANOVA was used to examine the overall adequacy of the second and third models. The second model was adequate (F = 5.03, p < 0.05) but the third was not (F = 1.05, p > 0.05). It was encouraging to note that our research managed to reproduce the results of the original study. Forty-five of the 46 parameters computed in both studies were identical (the one exception being BIC (Bayesian Information Criterion) for the first model).
Multicollinearity must be examined after establishing the models. As discussed in the Data Analysis section, when multicollinearity is present, statistically significant predictors can appear non-significant due to large standard errors (Tabachnick & Fidell, Reference Tabachnick and Fidell2013). In this study, a pairwise correlation of 0.8 or above was considered a concern (Nimon, Reference Nimon and Frey2018). The correlation matrix of four related variables can be found in Table 3. These four variables were chosen because they were highly relevant to each other in theory. Among the four variables, two pairs of predictors had a correlation of 0.8 or above (the number of inputs and the number of distinct inputs; the number of visits and the number of visits on distinct days; see Methodology for examples of how these variables differ). This result was expected as the variables are the same in nature. The presence of multicollinearity suggested that the original models might have had limitations.
Correlation matrix of related variables
* p < 0.05.
There are three common ways to address multicollinearity problems: using a larger sample, removing variables or transforming variables. In acknowledgement of the sample-size limitations, we decided to re-establish the regression equations after removing the two highly correlated predictors. As they measured similar constructs (i.e. distinct entries vs. all entries; distinct visits vs. all visits) and the author of the original study acknowledged the importance of counting the distinct entries, we decided to remove the “all visits” and “all entries” predictors. See Appendices 1a–1c for the three recreated regression models.
The re-established regression models were comparable to the original ones in terms of overall adequacy, R 2 and statistically significant predictors. However, the strength of the predictors for input and visit dropped by approximately 30% to 50%, except for the visit count in Model 3, which increased by 19%. Thus, two observations made in the original study need to be revised: After controlling for all other factors, visiting the tool once resulted in a 5.44% increase in midterm test score (originally, 5.03%) and each distinct day of visiting the tool resulted in a 1.73% score increase. As in the original study, most of the predictors were not statistically significant. Only the secondary school exit exams that tested written and spoken English were significant predictors of the midterm test score. The secondary school exit exam of spoken English and the average visit duration were statistically significant predictors of the phonotactics quiz score. Although the results in the recreated regression analyses seemed comparable, the sample-size limitations and the significance of predictors for CALL studies deserve further discussion.
4. Discussion
4.1 The value of exact replication studies
One of the aims of the current study was to conduct a specific type of replication for Łodzikowski’s (Reference Łodzikowski2021) study, an exact replication study. The results suggested that the reproduced results were highly comparable to those obtained in the original study, with only minor differences. However, through the replication process, we identified certain limitations in the original study – sample size and collinearity – that could have been discussed further. It is important to emphasise once again that we, by no means, are claiming that the arguments in the original study were unfounded. Instead, we would like to extend the original study by discussing some potential limitations. Although the regression models in the current study worked with the data set and the sample size was representative of the population, the sample-size requirements could have been further discussed in the limitations section. In the same vein, collinearity was to be expected for the given variables. In other words, while it is technically possible to recreate the results of the original study using the same data analysis methods, we argue that results in the original study should be read cautiously and as those of an exploratory study. While past replication studies have measured the success of replication by the extent to which the new results were similar to or different from the original results, this study was different: the results were reproducible but not successful. This finding offers a unique perspective on replication studies.
The current study, as a specific type of replication study, an exact replication study, demonstrates the value of conducting exact replication studies. We used the process of identifying and discussing limitations to identify “undiscovered bugs” in the original study, as described by Miłkowski, Hensel and Hohol (Reference Miłkowski, Hensel and Hohol2018). Models such as the three in the current study can be “reliable estimate[s]” only if they can be verified (Cipriano et al., Reference Cipriano, Barnes, Kolev, Rivers and Brackett2019: 10). Conducting a direct and conceptual replication that attempts to follow the procedures of the original study and recreate its results (Porte & McManus, Reference Porte, McManus, Porte and McManus2019) can contribute to the body of knowledge by showing how the results differ from or are similar to the original findings. If a conceptual replication were to be conducted for the current study, more could be discovered about the association between transcription tool use and phonological awareness. In the current study, we contributed to the body of knowledge in a different but equally meaningful way by verifying the findings of the original study and providing support for the evidence that it contributed to the literature. From a broader perspective, the results of this study echo the need for more exact replication studies in education research (Morrison, Reference Morrison2022). To achieve this goal, more data should be made available for educational studies (Hiver & Al-Hoorie, Reference Hiver and Al-Hoorie2020).
4.2 Phonological awareness: Extending the original study
We extended the original study and provided additional insights into the acquisition of allophones and phonological awareness using the existing data that was shared with us. The original study presented only the 15 most frequently entered words and reported that they came from the in-class transcription worksheets. As mentioned by the author, some entries may have been overrepresented, but the list of words still provided insights into the ways in which Polish learners used a transcription tool to acquire allophones. The learners in this study had some awareness of allophonic features from their classroom instruction, so the transcription entries offered insights into the words or phonemes whose corresponding allophonic features they wanted to know. The students were more interested in some allophonic features than others. It is not surprising to see that plosive release features, including devoicing, aspiration and inaudible release, were quite common in the words learners entered (accounted for 64.88% of entries analysed). Although this could be an artefact of the course materials, another logical explanation is that Polish students want to learn more about the allophonic features of stops. As release is mandatory in Polish (Schwartz et al., Reference Schwartz, Balas and Rojczyk2014), we argue that learners may want to find out more about the contrasting allophonic features of English. The Polish students, as learners of English, may be aware of the differences. Alternatively, their teachers may have drawn their attention to the phonological differences between English and Polish, motivating them to search for more information using the tool. Another interesting observation was that some allophonic features – for example, GOAT allophony – received little attention from learners. Perhaps they had not seen examples of such allophones and thus could not enter valid transcriptions to explore them. For instance, there are few examples of GOAT allophony (e.g. [əʊl]) compared to the number of examples of aspirated plosives (e.g. pat [pʰæ̆t]) and unaspirated plosives (e.g. spark [spɑɹk]). To try out the allophonic features, the learners may have typed [əʊl] to see its change to [ɒʊɫ]; however, this entry would have been classified as a non-word in the data set.
To further help learners increase their phonological awareness, especially that of allophonic features, more features could be introduced to similar transcription tools. First, to further address the need to investigate L2-specific allophonic features (e.g. plosive-related features in this study), a transcription tool could incorporate audio so students could hear words pronounced correctly. Rojczyk et al. (Reference Rojczyk, Porzuczek and Bergier2013) found that imitation without distraction (e.g. by a tool in front of the computer) helped Polish learners of English acquire allophonic features; providing audio is the logical first step of imitation. Also, to address the problem that there were not enough examples of the less common allophonic features, transcription tools could be designed to allow users to type words instead of IPA symbols. For example, typing the word “swollen” could generate the phonetic transcript, /swəʊlən/, and the corresponding transcription with allophonic features, [sʷwɒʊlə̃n]. These suggestions, which are based on the findings of the current study, could help learners increase their phonological awareness and learn the allophonic features of languages.
4.3 Implications for CALL research
The data analysis of the original study appears to have had certain limitations because of the limited sample size. Although the sample was representative of the learners (it included 78.57% of the students in the course that was studied), the number of data points for the inferential statistical analysis was limiting. An examination of the usage data showed that there were only 270 visits to the tool (versus 320 reported in the original study) from December to mid-June, which means 53 visits per month for 55 learners. As in past studies of CALL tools, some learners may have been more active and used the tool more frequently than other learners (authors). Therefore, the original study examined the behaviours of a very limited number of learners (despite being representative in the context of a blended learning course at a Polish university). As a result, its generalisability to other contexts is limited. From a statistical perspective, the use of regression requires an adequate sample size regardless of how representative a given sample is, so the use of multiple linear regression did not build a strong argument for the association between the use of the CALL tool and phonological awareness. Many CALL studies using multiple linear regression have larger sample sizes. For example, a study by Goh, Sun and Yang (Reference Goh, Sun and Yang2020) considered six variables with 268 essays and one by Hegelheimer and Tower (Reference Hegelheimer and Tower2004) considered four variables with 94 students. Therefore, it is not impossible for CALL studies to acquire sufficiently larger samples.
Despite these limitations, we see opportunities for future CALL research. We recommend increasing the number of data points for each learner so that tool usage can be assessed from a range of perspectives. One possible way to do this would be to expand the data points acquired from the tool. The original study reported that learners accessed the tool via a Moodle link and that their usage was tracked with Piwik. Therefore, it would be possible to obtain data on more learner-specific behaviours for analysis. For example, researchers could examine what transcriptions the learners entered, when they entered them, and how they performed on their midterm test on questions that assessed the corresponding allophonic features. This could tell us, for example, if a learner who examines nasalisation frequently before a midterm test (e.g. with multiple distinct inputs or visits) will have better phonological awareness of nasalisation and perform better on related questions on the midterm. A range of exploratory analysis strategies, such as data visualisation, could be used to gather evidence that will be useful for CALL researchers and practitioners.
Another way to collect more data would be to adopt a proactive approach. Given the six-month duration of the course, learners could be invited to complete instruments on phonological awareness before the use of the tool, two months after it was made accessible, before the midterm test and after the phonotactic quiz. With multiple self-reported data points from learners, it would be possible to track changes in learners’ phonological awareness over time and look for associations between these changes and their performance on the midterm test. Although such proactive data collection approaches may require complex ethical clearance procedures, they could generate better evidence for CALL researchers and practitioners.
5. Conclusion
This paper reported a unique exact replication study within CALL that attempted to recreate the results of Łodzikowski’s (Reference Łodzikowski2021) study of a transcription tool. The original study examined the use of an allophonic transcription tool and its association with phonological awareness. In reproducing this study, we extended it by examining its limitations. To a certain extent, we recreated the results of the original study, but we concluded that its models would have been more accurate if there had been fewer variables due to multicollinearity. Also, we believe that the results of the original study should be considered exploratory due to the small sample size. The extension of the study from this perspective offers other CALL researchers the opportunity to consider how they could acquire better data sets using online tools. It also provides additional information on learners’ interest in the allophonic process and insights into teaching allophones. Finally, the present study provides new insights into the role of exact replication studies in contrast to other replication studies. Journals and publishers should encourage the sharing of data sets so that more exact replication studies are possible.
Supplementary material
All appendices are available on Figshare: https://doi.org/10.6084/m9.figshare.22109027
To also view supplementary material referred to in this article, please visit https://doi.org/10.1017/S0958344023000071
Data
The data used in this study is made available by the original author. See Methodology for details.
Ethical statement and competing interests
This paper adopted an existing data set being made available online. The authors declare no competing interests.
About the authors
Dennis Foung is a writing teacher at the School of Journalism, Writing and Media at the University of British Columbia. He has a keen interest in computer-assisted language learning and learning analytics.
Lucas Kohnke is a senior lecturer at the Department of English Language Education at the Education University of Hong Kong. His research interests include technology-supported teaching and learning, professional development using information communication technologies, and second language learning/acquisition.
Author ORCIDs
Dennis Foung, https://orcid.org/0000-0002-6769-0582
Lucas Kohnke, https://orcid.org/0000-0001-6717-5719
Open access
