Skip to main content

Transparency of reporting in CALL meta-analyses between 2003 and 2015

  • Huifen Lin (a1), Tsuiping Chen (a2) and Hsien-Chin Liou (a3)

Since its introduction by Glass in the 1970s, meta-analysis has become a widely accepted and the most preferred approach to conducting research synthesis. Overcoming the weaknesses commonly associated with traditional narrative review and vote counting, meta-analysis is a statistical method of systematically aggregating and analyzing empirical studies by following well-established procedures. The findings of a meta-analysis, when appropriately conducted, are able to inform important policy decisions and provide practical pedagogical suggestions. With the growing number of publications employing meta-analysis across a wide variety of disciplines, it has received criticism due to its inconsistent findings derived from multiple meta-analyses in the same research domain. These inconsistencies have arisen partly due to the alternatives available to meta-analysts in each major meta-analytic procedure. Researchers have therefore recommended transparent reporting on the decision-making for every essential judgment call so that the results across multiple meta-analyses become replicable, consistent, and interpretable. This research explored the degree to which meta-analyses in the computer-assisted language learning (CALL) discipline transparently reported their decisions in every critical step. To achieve this aim, we retrieved 15 eligible meta-analyses in CALL published between 2003 and 2015. Features of these meta-analyses were extracted based on a codebook modified from Cooper (2003) and Aytug, Rothstein, Zhou and Kern (2012). A transparency score of reporting was then calculated to examine the degree to which these meta-analyses are compliant with the norms of reporting as recommended in the literature. We then discuss the strengths and weaknesses of the methodologies and provide suggestions for conducting quality meta-analyses in this domain.

Hide All
*Abraham, L. B. (2008) Computer-mediated glosses in second language reading comprehension and vocabulary learning: A meta-analysis. Computer Assisted Language Learning, 21(3): 199226.
Ahn, S., Ames, A. J. and Myers, N. D. (2012) A review of meta-analyses in education: Methodological strengths and weaknesses. Review of Educational Research, 82(4): 436476.
Aytug, Z. G., Rothstein, H. R., Zhou, W. and Kern, M. C. (2012) Revealed or concealed? Transparency of procedures, decisions, and judgment calls in meta-analyses. Organizational Research Methods, 15(1): 103133.
Berkeljon, A. and Baldwin, S. A. (2009) An introduction to meta-analysis for psychotherapy outcome research. Psychotherapy Research, 19(4–5): 511518.
Borenstein, M., Hedges, L. V., Higgins, J. P. T. and Rothstein, H. R. (2009) Introduction to meta-analysis. Chichester, UK: Wiley.
*Chang, M.-M. and Lin, M.-C. (2013) Strategy-oriented web-based English instruction: A meta-analysis. Australasian Journal of Educational Technology, 29(2): 203216.
*Chiu, Y.-H. (2013) Computer-assisted second language vocabulary instruction: A meta-analysis. British Journal of Educational Technology, 44(2): E52E56.
*Chiu, Y.-H., Kao, C.-W. and Reynolds, B. L. (2012) The relative effectiveness of digital game-based learning types in English as a foreign language setting: A meta-analysis. British Journal of Educational Technology, 43(4): E104E107.
*Cobb, T. and Boulton, A. (2015) Classroom applications of corpus analysis. In In Biber, D. and Reppen, R. (eds.), The Cambridge handbook of English corpus linguistics. Cambridge: Cambridge University Press, 478497.
Cooper, H. (1998) Synthesizing research: A guide for literature reviews (3rd ed.). Thousand Oaks, CA: Sage.
Cooper, H. (2003) Editorial. Psychological Bulletin, 129(1): 39.
Cooper, H. (2007) Evaluating and interpreting research syntheses in adult learning and literacy (NCSALL occasional paper). Boston, MA: National Center for the Study of Adult Learning and Literacy.
Copas, J. and Shi, J. Q. (2000) Meta-analysis, funnel plots and sensitivity analysis. Biostatistics, 1(3): 247262.
Durlak, J. A., Weissberg, R. P. and Pachan, M. (2010) A meta-analysis of after-school programs that seek to promote personal and social skills in children and adolescents. American Journal of Community Psychology, 45(3–4): 294309.
Egger, M., Smith, G. D. and Phillips, A. N. (1997) Meta-analysis: Principles and procedures. British Medical Journal, 315: 15331537.
Elvik, R. (2005) Can we trust the results of meta-analyses? A systematic approach to sensitivity analysis in meta-analyses. Transportation Research Record: Journal of the Transportation Research Board, 1908: 221229.
Garg, A. X., Hackam, D. and Tonelli, M. (2008) Systematic review and meta-analysis: When one study is just not enough. Clinical Journal of the American Society of Nephrology, 3(1): 253260.
Glass, G. V. (1976) Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10): 38.
Glass, G. V., McGaw, B. and Smith, M. L. (1981) Meta-analysis in social research. Beverly Hills, CA: Sage.
Goo, J., Granena, G., Yilmaz, Y. and Novella, M. (2015) Implicit and explicit instruction in L2 learning. In Rebuschat, P. (ed.), Implicit and explicit learning of languages (Vol. 48). Amsterdam: John Benjamins, 443482.
*Grgurović, M., Chapelle, C. A. and Shelley, M. C. (2013) A meta-analysis of effectiveness studies on computer technology-supported language learning. ReCALL, 25(2): 165198.
Higgins, J. P. T. and Green, S. (2011) Cochrane handbook for systematic reviews of interventions (Vol. 4). Chichester, UK: Wiley.
In’nami, Y. and Koizumi, R. (2009) A meta-analysis of test format effects on reading and listening test performance: Focus on multiple-choice and open-ended formats. Language Testing, 26(2): 219244.
In’nami, Y. and Koizumi, R. (2010) Database selection guidelines for meta-analysis in applied linguistics. TESOL Quarterly, 44(1): 169184.
Levy, M. (1997) Computer-assisted language learning: Context and conceptualization. Oxford: Oxford University Press.
Li, S., Shintani, R. and Ellis, R. (2012) Doing meta-analysis in SLA: Practice, choices, and standards. Contemporary Foreign Language Studies, 384(12): 117.
Liao, Y.-K. C. and Hao, Y. (2008) Large-scale studies and quantitative methods. In Voogt, J. and Knezek, G. (eds.), International handbook of information technology in primary and secondary education (Vol. 20). New York: Springer Science & Business Media, 10191035.
*Lin, H. (2014) Establishing an empirical link between computer-mediated communication (CMC) and SLA: A meta-analysis of the research. Language Learning & Technology, 18(3): 120147.
*Lin, H. (2015a) Computer-mediated communication (CMC) in L2 oral proficiency development: A meta-analysis. ReCALL, 27(3): 261287.
*Lin, H. (2015b) A meta-synthesis of empirical research on the effectiveness of computer-mediated communication (CMC) on SLA. Language Learning & Technology, 19(2): 85117.
*Lin, W.-C., Huang, H.-T. and Liou, H.-C. (2013) The effects of text-based SCMC on SLA: A meta analysis. Language Learning & Technology, 17(2): 123142.
Liou, H.-C. and Lin, H.-F. (2017) CALL meta-analyses and transparency analysis. In Chapelle, C. A. and Sauro, S. (eds.), The handbook of technology and second language teaching and learning. Hoboken, NJ: Wiley, 409427.
Lipsey, M. W. and Wilson, D. B. (2001) Practical meta-analysis. Thousand Oaks, CA: Sage.
Littell, J. H., Corcoran, J. and Pillai, V. (2008) Systematic reviews and meta-analysis. Oxford: Oxford University Press.
Norris, J. M. and Ortega, L. (2000) Effectiveness of L2 instruction: A research synthesis and quantitative meta-analysis. Language Learning, 50(3): 417528.
Norris, J. M. and Ortega, L. (2006) The value and practice of research synthesis for language learning and teaching. In Norris, J. M. and Ortega, L. (eds.), Synthesizing research on language learning and teaching. Amsterdam: John Benjamins, 350.
Plonsky, L. (2012) Replication, meta-analysis, and generalizability. In Porte, G. (ed.), Replication research in applied linguistics. Cambridge: Cambridge University Press, 116132.
Plonsky, L. (2013) Study quality in SLA: An assessment of designs, analyses, and reporting practices in quantitative L2 research. Studies in Second Language Acquisition, 35(4): 655687.
Plonsky, L. and Ziegler, N. (2016) The CALL–SLA interface: Insights from a second-order synthesis. Language Learning & Technology, 20(2): 1737.
Rosenthal, R. and DiMatteo, M. R. (2001) Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52: 5982.
Rothstein, H. R. and McDaniel, M. A. (1989) Guidelines for conducting and reporting meta-analyses. Psychological Reports, 65(3): 759770.
Rothstein, H. R., Sutton, A. J. and Borenstein, M. (eds.) (2006) Publication bias in meta-analysis: Prevention, assessment and adjustments . Hoboken, NJ: Wiley.
Scammacca, N., Roberts, G. and Stuebing, K. K. (2014) Meta-analysis with complex research designs: Dealing with dependence from multiple measures and multiple group comparisons. Review of Educational Research, 84(3): 328364.
Schmidt, F. L. and Oh, I.-S. (2013) Methods for second order meta-analysis and illustrative applications. Organizational Behavior and Human Decision Processes, 121(2): 204218.
Shadish, W. R. and Sweeney, R. B. (1991) Mediators and moderators in meta-analysis: There’s a reason we don’t let dodo birds tell us which psychotherapies should have prizes. Journal of Consulting and Clinical Psychology, 59(6): 883893.
Shea, B. J., Hamel, C., Wells, G. A., Bouter, L. M., Kristjansson, E., Grimshaw, J., Henry, D. A. and Boers, M. (2009) AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology, 62(10): 10131020.
*Taylor, A. (2006) The effects of CALL versus traditional L1 glosses on L2 reading comprehension. CALICO Journal, 23(2): 309318.
*Taylor, A. M. (2010) CALL-based versus paper-based glosses: Is there a difference in reading comprehension? CALICO Journal, 27(1): 147160.
*Taylor, A. M. (2013) CALL versus paper: In which context are L1 glosses more effective? CALICO Journal, 30(1): 6381.
Turner, L., Boutron, I., Hróbjartsson, A., Altman, D. G. and Moher, D. (2013) The evolution of assessing bias in Cochrane systematic reviews of interventions: Celebrating methodological contributions of the Cochrane Collaboration. Systematic Reviews, 2: 28.
Valentine, J. C., Cooper, H., Patall, E. A., Tyson, D. and Robinson, J. C. (2010) A method for evaluating research syntheses: The quality, conclusions, and consensus of 12 syntheses of the effects of after-school programs. Research Synthesis Methods, 1(1): 2038.
Wanous, J. P., Sullivan, S. E. and Malinak, J. (1989) The role of judgment calls in meta-analysis. Journal of Applied Psychology, 74(2): 259264.
Willis, B. H. and Quigley, M. (2011) The assessment of the quality of reporting of meta-analyses in diagnostic research: A systematic review. BMC Medical Research Methodology, 11: 111.
*Yun, J. (2011) The effects of hypertext glosses on L2 vocabulary acquisition: A meta-analysis. Computer Assisted Language Learning, 24(1): 3958.
*Zhao, Y. (2003) Recent developments in technology and language learning: A literature review and meta-analysis. CALICO Journal, 21(1): 727.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

  • ISSN: 0958-3440
  • EISSN: 1474-0109
  • URL: /core/journals/recall
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Type Description Title
Supplementary materials

Lin et al supplementary material
Lin et al supplementary material 1

 Unknown (72 KB)
72 KB


Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed