Skip to main content
×
Home
    • Aa
    • Aa

STUDY QUALITY IN SLA: An Assessment of Designs, Analyses, and Reporting Practices in Quantitative L2 Research

  • Luke Plonsky (a1)
Abstract

This study assesses research and reporting practices in quantitative second language (L2) research. A sample of 606 primary studies, published from 1990 to 2010 in Language Learning and Studies in Second Language Acquisition, was collected and coded for designs, statistical analyses, reporting practices, and outcomes (i.e., effect sizes). The results point to several systematic strengths as well as many flaws, such as a lack of control in experimental designs, incomplete and inconsistent reporting practices, and low statistical power. I discuss these trends, strengths, and weaknesses in comparison with methodological reviews of L2 research (e.g., Plonsky & Gass, 2011) as well as reviews from other fields (e.g., education, Skidmore & Thompson, 2010). On the basis of the findings, I also make a number of suggestions for methodological reforms in applied linguistics.

Copyright
Corresponding author
*Correspondence concerning this article should be addressed to Luke Plonsky, PO Box 6032, Flagstaff, AZ 86011. E-mail: luke.plonsky@nau.edu
Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

H. Aguinis , C. A. Pierce , F. A. Bosco , & I. S Muslin . (2009). First decade of organizational research methods trends in design, measurement, and data-analysis topics. Organizational Research Methods, 12, 69112.

A. W. Bangert , & J. P Baumberger . (2005). Research and statistical techniques used in the Journal of Counseling & Development. Journal of Counseling & Development ,83, 480487.

S. Brutus , H. Gill , & K Duniewicz . (2010). State-of-science in industrial and organizational psychology: A review of self-reported limitations. Personnel Psychology, 63, 907936.

L. H. Cashen , & S. W Geiger . (2004). Statistical power and the testing of null hypotheses: A review of contemporary management research and recommendations for future studies. Organizational Research Methods, 7, 151167.

A.-W. Chan , A. Hróbjartsson , M. T. Haahr , P. C. Gøtzsche , & D. G Altman . (2004). Empirical evidence for selective reporting of outcomes in randomized trials. Journal of the American Medical Association, 291, 24572465.

C Chaudron . (1986). The interaction of quantitative and qualitative approaches to research: A view of the second language classroom. TESOL Quarterly, 20, 709717.

C Chaudron . (2001). Progress in language classroom research: Evidence from The Modern Language Journal, 1916–2000. Modern Language Journal, 85, 5776.

J Cohen . (1968). Multiple regression as a general data-analytic system. Psychological Bulletin, 70, 426443.

J Cohen . (1994). The earth is round (p < .05). American Psychologist, 49, 971003.

G Crookes . (1991). Power, effect size, and second language research: Another researcher comments. TESOL Quarterly, 25, 762765.

R. DeKeyser , & R Schoonen . (2007). Editors’ announcement. Language Learning, 57, ixx.

T. A DeVaney . (2001). Statistical significance, effect size, and replication: What do the journals say? The Journal of Experimental Education, 69, 310320.

T. H Dinsmore . (2006). Principles, parameters, and SLA: A retrospective meta-analytic investigation into adult L2 learners’ access to Universal Grammar. In J. M. Norris & L. Ortega (Eds.), Synthesizing research on language learning and teaching (pp. 5390). Amsterdam: Benjamins.

S. H. Downs , & N Black . (1998). The feasibility of creating a checklist for the assessment of the methodological quality both of randomized and nonrandomized studies of health care interventions. Journal of Epidemiology & Community Health, 52, 377384.

J Egbert . (2007). Quality analysis of journals in TESOL and applied linguistics. TESOL Quarterly, 41, 157171.

J. L Fleiss . (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76, 378382.

S. Gass , C. Fleck , N. Leder , & I Svetics . (1998). Ahistoricity revisited: Does SLA have a history? Studies in Second Language Acquisition, 20, 407421.

S. M Gass . (1993). Second language acquisition: Cross-disciplinary perspectives. Second Language Research, 9, 9598.

A. Gelman , J. Hill , & M Yajima . (2012). Why we (usually) don’t have to worry about multiple comparisons. Journal of Research on Educational Effectiveness, 5, 189211.

A. Gelman , & D Weakliem . (2009). Of beauty, sex and power: Too little attention has been paid to the statistical challenges in estimating small effects. American Scientist, 97, 310316.

L. D. Goodwin , & W. L Goodwin . (1985). An analysis of statistical techniques used in the Journal of Educational Psychology, 1979–1983. Educational Psychologist, 20, 1321.

E Hatch . (1978). Apply with caution. Studies in Second Language Acquisition, 2, 123143.

G Henning . (1986). Quantitative methods in language acquisition research. TESOL Quarterly, 20, 701708.

L. G Humphreys . (1978). Doing research the hard way: Substituting analysis of variance for a problem in correlational analysis. Journal of Educational Psychology, 70, 873876.

Journal Article Reporting Standards Working Group. (2008). Reporting standards for research in psychology: Why do we need them? What might they be? American Psychologist, 63, 839851.

H. J. Keselman , C. J. Huberty , L. M. Lix , S. Olejnik , R. A. Cribbie , B. Donahue , . . . J. R Levin . (1998). Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA, and ANCOVA analyses. Review of Educational Research, 68, 350386.

K. M. Kieffer , R. J. Reese , & B Thompson . (2001). Statistical techniques employed in AERJ and JCP articles from 1988 to 1997: A methodological review. The Journal of Experimental Education, 69, 280309.

M Kubanyiova . (2008). Rethinking research ethics in contemporary applied linguistics: The tension between macroethical and microethical perspectives in situated research. Modern Language Journal, 92, 503518.

J. R. Landis , & G. G Koch . (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159174.

J. Larson-Hall , & R Herrington . (2010). Improving data analysis in second language acquisition by utilizing modern developments in applied statistics. Applied Linguistics, 31, 368–190.

A Lazaraton . (1991). Power, effect size, and second language research: A researcher comments. TESOL Quarterly, 25, 759762.

A Lazaraton . (2000). Current trends in research methodology and statistics in applied linguistics. TESOL Quarterly, 34, 175181.

A. Lazaraton , H. Riggenbach , & A Ediger . (1987). Forming a discipline: Applied linguists’ literacy in research methodology and statistics. TESOL Quarterly, 21, 263277.

S Li . (2010). The effectiveness of corrective feedback in SLA: A meta-analysis. Language Learning, 60, 309365.

P. M Lightbown . (2000). Anniversary article: Classroom second language research and second language teaching. Applied Linguistics, 21, 431462.

S Loewen . (2005). Incidental focus on form and second language learning. Studies in Second Language Acquisition, 27, 361386.

D. E Lykken . (1968). Statistical significance in psychological research. Psychological Bulletin, 70, 151159.

R. Lyster , & J Izquierdo . (2009). Prompts versus recasts in dyadic interaction. Language Learning, 59, 453498.

A. Mackey , & S. M Gass . (Eds.). (2012). Research methods in second language acquisition: A practical guide. Oxford: Wiley-Blackwell.

S. S Magnan . (1994). From the editor: The MLJ tradition and the challenges ahead. Modern Language Journal, 78, 79.

M. S. Matthews , M. Gentry , D. B. McCoach , F. C. Worrell , D. Matthews , & F Dixon . (2008). Evaluating the state of a field: Effect size reporting in gifted education. The Journal of Experimental Education, 77, 5565.

S. T. Meier , & S. R Davis . (1990). Trends in reporting psychometric properties of scales used in counseling psychology research. Journal of Counseling Psychology, 37, 113115.

M. A. Mone , G. C. Mueller , & W Mauland . (1996). The perceptions and usage of statistical power in applied psychology and management research. Personnel Psychology, 49, 103120.

E. Nicoladis , & A Krott . (2007). Word family size and French-speaking children’s segmentation of existing compounds. Language Learning, 57, 201228.

J. M. Norris , & L Ortega . (2000). Effectiveness of L2 instruction: A research synthesis and quantitative meta-analysis. Language Learning, 50, 417528.

L Ortega . (2005). Methodology, epistemology, and ethics in instructed SLA research: An introduction. Modern Language Journal, 89, 317327.

T Pica . (1997). Second language teaching and research relationships: A North American view. Language Teaching Research, 1, 4872.

L Plonsky . (2011a). The effectiveness of second language strategy instruction: A meta-analysis. Language Learning, 61, 9931038.

L. Plonsky , & S Gass . (2011). Quantitative research methods, study quality, and outcomes: The case of interaction research. Language Learning, 61, 325366.

L. Plonsky , & F. L Oswald . (2012). How to do a meta-analysis. In A. Mackey & S. Gass (Eds.), Research methods in second language acquisition: A practical guide (pp. 275295). Oxford: Wiley-Blackwell.

C Polio . (1997). Measures of linguistic accuracy in second language writing research. Language Learning, 47, 101143.

G Porte . (2010) Appraising research in second language learning: A practical approach to critical analysis of quantitative research (2nd ed.). Amsterdam: Benjamins.

D Pulido . (2004). The relationship between text comprehension and second language incidental vocabulary acquisition: A matter of topic familiarity? Language Learning, 54, 469523.

F. L Schmidt . (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training researchers. Psychological Methods, 1, 115129.

P. Sedlmeier , & G Gigerenzer . (1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309316.

L. Selinker , & U Lakshmanan . (2001). How do we know what we know? Why do we believe what we believe? Second Language Research, 17, 323325.

S. T. Skidmore , & B Thompson . (2010). Statistical techniques used in published articles: A historical review of reviews. Educational and Psychological Measurement, 70, 777795.

B. Smith , & B. A Lafford . (2009). The evaluation of scholarly activity in computer-assisted language learning. Modern Language Journal, 93, 868883.

S. Sun , W. Pan , & L. L Wang . (2010). A comprehensive review of effect size reporting and interpreting practices in academic journals in education and psychology. Journal of Educational Psychology, 102, 9891004.

B Thompson . (2001). Significance, effect sizes, stepwise methods, and other issues: Strong arguments move the field. The Journal of Experimental Education, 70, 8093.

B. Thompson , & P. A Snyder . (1998). Statistical significance and reliability analyses in recent JCD research articles. Journal of Counseling and Development, 76, 436441.

T. Vacha-Haase , C. Ness , J. Nilsson , & D Reetz . (1999). Practices regarding reporting of reliability coefficients: A review of three journals. The Journal of Experimental Education, 67, 335341.

T. Vacha-Haase , & B Thompson . (2004). How to estimate and interpret various effect sizes. Journal of Counseling Psychology, 51, 473481.

A Valdman . (1998). A note from the editor: 20th anniversary of SSLA. Studies in Second Language Acquisition, 20, 463470.

J. C. Valentine , & H Cooper . (2008). A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: The study design and implementation assessment device (Study DIAD). Psychological Methods, 13, 130149.

H. Z Waring . (2009). Moving out of IRF (initiation-response-feedback): A single case analysis. Language Learning, 59, 796824.

C. S. Wells , & J. M Hintze . (2007). Dealing with assumptions underlying statistical tests. Psychology in the Schools, 44, 495502.

K. Wells , & J. H Littell . (2009). Study quality assessment in systematic reviews of research on intervention effects. Research on Social Work Practice, 19, 5262.

L. Wilkinson , & Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594604.

V. L Willson . (1980). Research techniques in AERJ articles: 1969 to 1978. Educational Researcher, 9, 510.

Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Studies in Second Language Acquisition
  • ISSN: 0272-2631
  • EISSN: 1470-1545
  • URL: /core/journals/studies-in-second-language-acquisition
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×