Hostname: page-component-76fb5796d-vvkck Total loading time: 0 Render date: 2024-04-25T15:57:42.033Z Has data issue: false hasContentIssue false
  • English
  • Français

Meta-analysis in Second Language Research: Choices and Challenges

Published online by Cambridge University Press:  27 July 2010

Frederick L. Oswald  and
Luke Plonsky
Get access Rights & Permissions [Opens in a new window]

Abstract

Applied linguists are increasingly conducting meta-analysis in their substantive domains, because as a quantitative approach for averaging effect sizes across studies, it is more systematic and replicable than traditional, qualitative literature reviews. Additional strengths, such as increased statistical power, moderator analyses, and model testing, have also contributed to its appeal. The current review describes typical stages of a meta-analysis in second language acquisition (SLA) research: (a) defining the research domain, (b) developing a reliable coding scheme, (c) analyzing data, and (d) interpreting results. Each stage has a host of equally reasonable decisions that can be made; each decision will influence the conduct of the meta-analysis, the nature of the results, and the substantive implications of findings for SLA. We highlight a number of benefits and challenges that inform these decisions. In general, when a meta-analysis in applied linguistics is well planned, employs sound statistical methods, and is based on a thorough understanding of relevant theory, it can provide critical information that informs theory as well as future research, practice, and policy.

Type
Research Article
Information
Annual Review of Applied Linguistics , Volume 30 , March 2010 , pp. 85 - 110
Copyright
Copyright © Cambridge University Press 2010

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

REFERENCES

Abraham, L. B. (2008). Computer-mediated glosses in second language reading comprehension and vocabulary learning: A meta-analysis. Computer Assisted Language Learning, 21, 199–226.CrossRefGoogle Scholar
Adesope, O. O., Lavin, T., Thompson, T., & Ungerleider, C. (2009, April). Systematic review and meta-analysis on the cognitive benefits of bilingualism. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.Google Scholar
Balluerka, N., Gómez, J., & Hidalgo, D. (2005). The controversy over null hypothesis significance testing revisited. Methodology, 1, 55–70.CrossRefGoogle Scholar
Bangert-Drowns, R. L. (1995). Misunderstanding meta-analysis. Evaluation & the Health Professions, 18, 304–314.CrossRefGoogle ScholarPubMed
Becker, B. J. (2005). Failsafe N or file-drawer number. In Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 111–126). Hoboken, NJ: Wiley.CrossRefGoogle Scholar
Becker, B. J. (2009). Model-based meta-analysis. In Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 377–395). New York: Sage.Google Scholar
Berlin, J. A., & Ghersi, D. G. (2005). Preventing publication bias: Registries and prospective meta-analysis. In Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 35–48). Hoboken, NJ: Wiley.CrossRefGoogle Scholar
Borenstein, M. (2005). Software for publication bias. In Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 193–220). Hoboken, NJ: Wiley.CrossRefGoogle Scholar
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2005). Comprehensive meta-analysis (Version 2). Englewood, NJ: Biostat.Google Scholar
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Hoboken, NJ: Wiley.CrossRefGoogle Scholar
Burnham, J. C. (1990). The evolution of editorial peer review. Journal of the American Medical Association, 263, 1323–1329.CrossRefGoogle ScholarPubMed
Chandler, J. (2004). A response to Truscott. Journal of Second Language Writing, 13, 345–348.CrossRefGoogle Scholar
Cheung, S. F., & Chan, D. K.-S. (2004). Dependent effect sizes in meta-analysis: Incorporating the degree of interdependence. Journal of Applied Psychology, 89, 780–791.CrossRefGoogle ScholarPubMed
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.Google Scholar
Cooper, H. (1998). Synthesizing research: A guide for literature reviews. Thousand Oaks, CA: Sage.Google Scholar
Cooper, H. (2003). Editorial. Psychological Bulletin, 129, 3–9.CrossRefGoogle Scholar
Cooper, H. M., & Rosenthal, R. (1980). Statistical versus traditional procedures for summarizing research findings. Psychological Bulletin, 87, 442–449.CrossRefGoogle ScholarPubMed
Crookes, G. (1991). Power, effect size, and second language research: Another researcher comments. TESOL Quarterly, 25, 762–765.CrossRefGoogle Scholar
Cumming, G., & Finch, S. (2005). Inference by eye: Confidence intervals and how to read pictures of data. American Psychologist, 60, 170–180.CrossRefGoogle ScholarPubMed
Dinsmore, T. H. (2006). Principles, parameters, and SLA: A retrospective meta-analytic investigation into adult L2 learners’ access to Universal Grammar. In Norris, J. M. & Ortega, L. (Eds.), Synthesizing research on language learning and teaching (pp. 53–90). Philadelphia: John Benjamins.CrossRefGoogle Scholar
Ellis, N. C. (2006). Meta-analysis, human cognition, and language learning. In Norris, J. M. & Ortega, L. (Eds.), Synthesizing research on language learning and teaching (pp. 301–322). Philadelphia: John Benjamins.CrossRefGoogle Scholar
Eysenck, H. J. (1984). Meta-analysis: An abuse of research integration. Journal of Special Education, 18, 41–59.CrossRefGoogle Scholar
Fern, E. F., & Monroe, K. B. (1996). Effect-size estimates: Issues and problems in interpretation. Journal of Consumer Research, 23, 89–105.CrossRefGoogle Scholar
Ferris, D. (1999). The case for grammar correction in L2 writing classes: A response to Truscott (1996). Journal of Second Language Writing, 8, 1–11.CrossRefGoogle Scholar
Ferris, D. (2004). The “grammar correction” debate in L2 writing: Where are we, and where do we go from here? (and what do we do in the meantime?). Journal of Second Language Writing, 13, 49–62.CrossRefGoogle Scholar
Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3–8.CrossRefGoogle Scholar
Gleser, L. J., & Olkin, I. (2009). Stochastically dependent effect sizes. In Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.), The handbook of research synthesis and meta-analysis 2nd ed., (pp. 357–376). New York: Sage.Google Scholar
Goldschneider, J. M., & DeKeyser, R. M. (2001). Explaining the natural order of L2 morpheme acquisition: A meta-analysis of multiple determinants. Language Learning, 51, 1–50.CrossRefGoogle Scholar
Goo, J., Granena, G., Novella, M., & Yilmaz, Y. (2009, October). Implicit and explicit instruction in L2 learning: Norris and Ortega (2000) revisited and updated. Paper presented at the Second Language Research Forum, East Lansing, MI.Google Scholar
Grgurović, M. (2007, October). Research on CALL comparison studies: Can a meta-analysis inform instructed SLA? Paper presented at the Second Language Research Forum, Urbana-Champaign, IL.Google Scholar
Hall, J. A., & Rosenthal, R. (1995). Interpreting and evaluating meta-analysis. Evaluation & the Health Professions, 18, 393–407.CrossRefGoogle ScholarPubMed
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.Google Scholar
Hedges, L. V., & Pigott, T. D. (2001). The power of statistical tests in meta-analysis. Psychological Methods, 6, 203–217.CrossRefGoogle ScholarPubMed
Henning, G. (1986). Quantitative methods in language acquisition research. TESOL Quarterly, 20, 701–708.CrossRefGoogle Scholar
Henson, R. K. (2006). Effect-size measures and meta-analytic thinking in counseling psychology research. The Counseling Psychologist, 34, 601–629.CrossRefGoogle Scholar
Higgins, J. P. T., White, I. R., & Wood, A. M. (2008). Imputation methods for missing outcome data in meta-analysis of clinical trials. Clinical Trials, 5, 225–239.CrossRefGoogle ScholarPubMed
Holmbeck, G. N. (1997). Toward terminological, conceptual, and statistical clarity in the study of mediators and moderators: Examples from the child-clinical and pediatric psychology literatures. Journal of Counseling and Clinical Psychology, 65, 599–610.CrossRefGoogle Scholar
Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage.CrossRefGoogle Scholar
In'nami, Y., & Koizumi, R. (2009). A meta-analysis of test format effects on reading and listening test performance: Focus on multiple-choice and open-ended formats. Language Testing, 26, 219–244.CrossRefGoogle Scholar
Jeon, E. H., & Kaya, T. (2006). Effects of L2 instruction on interlanguage pragmatic development: A meta-analysis. In Norris, J. M. & Ortega, L. (Eds.), Synthesizing research on language learning and teaching (pp. 165–211). Philadelphia: John Benjamins.CrossRefGoogle Scholar
Kavale, K. A. (1995). Meta-analysis at 20: Retrospect and prospect. Evaluation & the Health Professions, 18, 349–369.CrossRefGoogle ScholarPubMed
Keck, C. M., Iberri-Shea, G., Tracy-Ventura, N., & Wa-Mbaleka, S. (2006). Investigating the empirical link between task-based interaction and acquisition: A meta-analysis. In Norris, J. M. & Ortega, L. (Eds.), Synthesizing research on language learning and teaching (pp. 91–131). Philadelphia: John Benjamins.CrossRefGoogle Scholar
Kirk, R. E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746–759.CrossRefGoogle Scholar
Kline, R. B. (2004). Beyond significance testing: Reforming data analysis methods in behavioral research. Washington, DC: American Psychological Association.CrossRefGoogle Scholar
Kline, R. B. (2009). Becoming a behavioral science researcher: A guide to producing research that matters. New York: Guilford Press.Google Scholar
Language Teaching Review Panel (2008). Replication studies in language learning and teaching: Questions and answers. Language Teaching, 41, 1–14.CrossRefGoogle Scholar
Larson-Hall, J. (2010). A guide to doing statistics in second language research using SPSS. New York: Routledge.Google Scholar
Lazaraton, A. (1991). Power, effect size, and second language research: A researcher comments. TESOL Quarterly, 25, 759–762.CrossRefGoogle Scholar
Lazaraton, A. (2000). Current trends in research methodology and statistics in applied linguistics. TESOL Quarterly, 34, 175–181.CrossRefGoogle Scholar
Lee, S.-K., & Huang, H.-T. (2008). Visual input enhancement and grammar learning: A meta-analytic review. Studies in Second Language Acquisition, 30, 307–331.CrossRefGoogle Scholar
Li, S. (2010). The effectiveness of corrective feedback in SLA: A meta-analysis. Language Learning, 60, 309–365.CrossRefGoogle Scholar
Lipsey, M. W., & Wilson, D. B. (1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48, 1181–1209.CrossRefGoogle ScholarPubMed
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.Google ScholarPubMed
Littell, J. H., Corcoran, J. C., & Pillai, V. (2008). Systematic reviews and meta-analysis. New York: Oxford University Press.CrossRefGoogle Scholar
Lykken, D. (1968). Statistical significance in psychological research. Psychological Bulletin, 70, 151–159.CrossRefGoogle ScholarPubMed
Lyster, R., & Saito, K. (2010). Oral feedback in classroom SLA: A meta-analysis. Studies in Second Language Acquisition, 32, 265–302.CrossRefGoogle Scholar
Mackey, A., & Goo, J. (2007). Interaction research in SLA: A meta-analysis and research synthesis. In Mackey, A. (Ed.), Conversational interaction in second language acquisition: A collection of empirical studies (pp. 407–451). New York: Oxford University Press.Google Scholar
Masgoret, A.-M., & Gardner, R. C. (2003). Attitudes, motivation, and second language learning: A meta-analysis of studies conducted by Gardner and associates. Language Learning, 53, 123–163.CrossRefGoogle Scholar
Miller, P. C., & Pan, W. (2009, March). Recasts in the L2 classroom: A meta-analytic review. Paper presented at the meeting of the American Association for Applied Linguistics, Denver, CO.Google Scholar
Mitchell, G. A., & Tetlock, P. E. (2009). A renewed appeal for adversarial collaboration. Research in Organizational Behavior, 29, 71–72.Google Scholar
Moncrieff, J. (1998). Research synthesis: Systematic reviews and meta-analysis. International Review of Psychiatry, 10, 304–311.CrossRefGoogle Scholar
Morris, S. B. (2008). Estimating effect sizes from pretest-posttest-control group designs. Organizational Research Methods, 11, 364–386.CrossRefGoogle Scholar
Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods, 7, 105–125.CrossRefGoogle ScholarPubMed
Nekrasova, T., & Becker, T. (2009). Effectiveness of practice: A research synthesis and quantitative meta-analysis. Manuscript in preparation.Google Scholar
Norris, J. M., & Ortega, L. (2000). Effectiveness of L2 instruction: A research synthesis and quantitative meta-analysis. Language Learning, 50, 417–528.CrossRefGoogle Scholar
Norris, J. M., & Ortega, L. (2003). Defining and measuring SLA. In Doughty, C. & Long, M. (Eds.), The handbook of second language acquisition (pp. 717–761). Malden, MA: Blackwell.Google Scholar
Norris, J. M., & Ortega, L. (2006). The value and practice of research synthesis for language learning and teaching. In Norris, J. M. & Ortega, L. (Eds.), Synthesizing research on language learning and teaching (pp. 3–50). Philadelphia: John Benjamins.CrossRefGoogle Scholar
Norris, J. M., & Ortega, L. (2007). The future of research synthesis in applied linguistics: Beyond art or science. TESOL Quarterly, 41, 805–815.CrossRefGoogle Scholar
Ortega, L. (2010). Research syntheses. In Paltridge, B. & Phakiti, A. (Eds.), Continuum companion to research methods in applied linguistics (pp. 111–126). London: Continuum.Google Scholar
Orwin, R. G. (1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics, 8, 157–159.Google Scholar
Orwin, R. G., & Vevea, J. L. (2009). Evaluating coding decisions. In Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.), The handbook of research synthesis (2nd ed., pp. 177–203). New York: Russell Sage Foundation.Google Scholar
Oswald, F. L., & Johnson, J. W. (1998). On the robustness, bias, and stability of results from meta-analysis of correlation coefficients: Some initial Monte Carlo findings. Journal of Applied Psychology, 83, 164–178.CrossRefGoogle Scholar
Oswald, F. L., & McCloy, R. A. (2003). Meta-analysis and the art of the average. In Murphy, K. R. (Ed.), Validity generalization: A critical review (pp. 311–338). Mahwah, NJ: Erlbaum.Google Scholar
Plonsky, L. (2009, October). “Nix the null”: Why statistical significance is overrated. Paper presented at the Second Language Research Forum, East Lansing, MI.Google Scholar
Plonsky, L. (in press). The effectiveness of second language strategy instruction: A meta-analysis.Google Scholar
Polio, C., & Gass, S. (1997). Replication and reporting: A commentary. Studies in Second Language Acquisition, 19, 499–508.CrossRefGoogle Scholar
Poltavtchenko, E., & Johnson, M. D. (2009, March). Feedback and second language writing: A meta-analysis. Poster session presented at the annual meeting of TESOL, Denver, CO.Google Scholar
Porte, G. (2009, March). Encouraging replication research in the field of applied linguistics and second language acquisition. Invited colloquium presented at the meeting of the American Association for Applied Linguistics, Denver, CO.Google Scholar
Preiss, R. W., & Allen, M. (1995). Understanding and using meta-analysis. Evaluation & the Health Professions, 18, 315–335.CrossRefGoogle ScholarPubMed
Prentice, D. A., & Miller, D. T. (1992). When small effects are impressive. Psychological Bulletin, 112, 160–164.CrossRefGoogle Scholar
Rosenthal, R. (1978). Combining results of independent studies. Psychological Bulletin, 85, 185–193.CrossRefGoogle Scholar
Rosenthal, R., & DiMatteo, M. R. (2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 51, 59–82.CrossRefGoogle Scholar
Ross, S. (1998). Self-assessment in second language testing: A meta-analysis and analysis of experiential factors. Language Testing, 15, 1–20.CrossRefGoogle Scholar
Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.). (2005). Publication bias in meta-analysis: Prevention, assessment and adjustments. Hoboken, NJ: Wiley.CrossRefGoogle Scholar
Russell, J., & Spada, N. (2006). The effectiveness of corrective feedback for the acquisition of L2 grammar: A meta-analysis of the research. In Norris, J. M. & Ortega, L. (Eds.), Synthesizing research on language learning and teaching (pp. 133–164). Philadelphia: John Benjamins.Google Scholar
Schmidt, F. L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1, 115–129.CrossRefGoogle Scholar
Schmidt, F. L., & Hunter, J. E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62, 529–540.CrossRefGoogle Scholar
Schmidt, F. L., Le, H., & Oh, I.-S. (2009). Correcting for the distorting effects of study artifacts in meta-analysis. In Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 317–333). New York: Sage.Google Scholar
Schmidt, F. L., Oh, I-S., & Hayes, T. (2009). Fixed versus random effects models in meta-analysis: Model properties and an empirical comparison of differences in results. British Journal of Mathematical and Statistical Psychology, 62, 97–128.CrossRefGoogle Scholar
Schmidt, R. (2001). Attention. In Robinson, P. (Ed.), Cognition and second language instruction (pp. 3–32). New York: Cambridge University Press.CrossRefGoogle Scholar
Slavin, R. E. (1986). Best-evidence synthesis: An alternative to meta-analytic and traditional reviews. Educational Researcher, 15, 5–11.CrossRefGoogle Scholar
Smith, M. L., Glass, G. V., & Miller, T. I. (1980). The benefits of psychotherapy. Baltimore, MD: Johns Hopkins.Google Scholar
Spada, N., & Tomita, Y. (2010). Interactions between type of instruction and type of language feature: A meta-analysis. Language Learning, 60, 263–308.CrossRefGoogle Scholar
Sterne, J. A. C., Becker, B. K., & Egger, M. (2005). The funnel plot. In Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 75–98). Hoboken, NJ: Wiley.Google Scholar
Sutton, A. J., & Higgins, J. P. T. (2008). Recent development in meta-analysis. Statistics in Medicine, 27, 625–650.CrossRefGoogle ScholarPubMed
Taylor, A. (2006). The effects of CALL versus traditional L1 glosses on L2 reading comprehension. CALICO Journal, 23, 309–318.CrossRefGoogle Scholar
Taylor, A., Stevens, J. R., & Asher, J. W. (2006). The effects of explicit reading strategy training on L2 reading comprehension: A meta-analysis. In Norris, J. M. & Ortega, L. (Eds.), Synthesizing research on language learning and teaching (pp. 213–244). Philadelphia: John Benjamins.CrossRefGoogle Scholar
Teleni, V., & Baldauf, R. B. (1989). Statistical techniques used in three applied linguistics journals, Language Learning, Applied Linguistics and TESOL Quarterly 1980–1986: Implications for readers and researchers. Unpublished research report. (ERIC Document Reproduction Service No. ED312905)Google Scholar
Torgerson, C. J. (2006). Publication bias: The Achilles’ heel of systematic reviews? British Journal of Educational Studies, 54, 89–102.CrossRefGoogle Scholar
Truscott, J. (1996). The case against grammar correction in L2 writing classes. Language Learning, 46, 327–369.CrossRefGoogle Scholar
Truscott, J. (1998). Noticing in second language acquisition: A critical review. Second Language Research, 24, 103–135.CrossRefGoogle Scholar
Truscott, J. (1999). The case for “The case against grammar correction in L2 writing classes”: A response to Ferris. Journal of Second Language Writing, 8, 111–122.CrossRefGoogle Scholar
Truscott, J. (2004). Evidence and conjecture on the effects of correction: A response to Chandler. Journal of Second Language Writing, 13, 337–343.CrossRefGoogle Scholar
Truscott, J. (2007). The effect of error correction on learners’ ability to write accurately. Journal of Second Language Writing, 16, 255–272.CrossRefGoogle Scholar
Tukey, J. W. (1991). The philosophy of multiple comparisons. Statistical Science, 6, 100–116.CrossRefGoogle Scholar
Tversky, A., & Kahneman, D. (1971). Belief in the law of small numbers. Psychological Bulletin, 76, 105–110.CrossRefGoogle Scholar
Valdman, A. (1993). Replication study. Studies in Second Language Acquisition, 15, 505.Google Scholar
Valentine, J. C. (2009). Judging the quality of primary research. In Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.), The handbook of research synthesis (2nd ed., pp. 129–146). New York: Russell Sage Foundation.Google Scholar
Valentine, J. C., & Cooper, H. (2003). Effect Size Substantive Interpretation Guidelines: Issues in the Interpretation of Effect Sizes. Washington, DC: What Works Clearinghouse.Google Scholar
Vevea, J. L., & Woods, C. M. (2005). Publication bias in research synthesis: Sensitivity analysis using a priori weight functions. Psychological Methods, 10, 428–443.CrossRefGoogle ScholarPubMed
Volker, M. A. (2006). Reporting effect size estimates in school psychology research. Psychology in the Schools, 43, 653–672.CrossRefGoogle Scholar
Wa-Mbaleka, S. (2006). A meta-analysis investigating the effects of reading on second language vocabulary learning. Unpublished doctoral dissertation, Northern Arizona University, Flagstaff, AZ.Google Scholar
Wanous, J. P., Sullivan, S. E., & Malinak, J. (1989). The role of judgment calls in meta-analysis. Journal of Applied Psychology, 74, 259–264.CrossRefGoogle Scholar
White, H. D. (2009). Scientific communication and literature retrieval. In Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.), The handbook of research synthesis (2nd ed., pp. 51–71). New York: Russell Sage Foundation.Google Scholar
Wilkinson, L., & Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594–604.CrossRefGoogle Scholar
Wolf, F. M. (1986). Meta-analysis: Quantitative methods for research synthesis. Beverly Hills, CA: Sage.CrossRefGoogle Scholar
Won, M. (2008). The effects of vocabulary instruction on English language learners: A meta-analysis. Unpublished doctoral dissertation, Texas Tech University, Lubbock, TX.Google Scholar
Zhao, Y. (2003). Recent developments in technology and language learning: A literature review and meta-analysis. CALICO Journal, 21, 7–27.CrossRefGoogle Scholar