Skip to main content
×
Home
    • Aa
    • Aa
  • Get access
    Check if you have access via personal or institutional login
  • Cited by 26
  • Cited by
    This article has been cited by the following publications. This list is generated based on data provided by CrossRef.

    Bergström, Kirstin Klatte, Maria Steinbrink, Claudia and Lachmann, Thomas 2016. First and Second Language Acquisition in German Children Attending a Kindergarten Immersion Program: A Combined Longitudinal and Cross-Sectional Study. Language Learning, Vol. 66, Issue. 2, p. 386.


    Bernardo, Aránzazu Amérigo, María and García, Juan A. 2016. American and Chinese students’ profiles based on Spanish-learning strategies: a transcultural comparison. Journal of Multilingual and Multicultural Development, p. 1.


    Derrick, Deirdre J. 2016. Instrument Reporting Practices in Second Language Research. TESOL Quarterly, Vol. 50, Issue. 1, p. 132.


    King, Kendall A. and Mackey, Alison 2016. Research Methodology in Second Language Studies: Trends, Concerns, and New Directions. The Modern Language Journal, Vol. 100, Issue. S1, p. 209.


    Mizumoto, Atsushi and Plonsky, Luke 2016. R as a Lingua Franca: Advantages of Using R for Quantitative Research in Applied Linguistics. Applied Linguistics, Vol. 37, Issue. 2, p. 284.


    MOSKOVSKY, CHRISTO RACHEVA, SILVIA ASSULAIMANI, TURKI and HARKINS, JEAN 2016. The L2 Motivational Self System and L2 Achievement: A Study of Saudi EFL Learners. The Modern Language Journal,


    PLONSKY, LUKE and DERRICK, DEIRDRE J. 2016. A Meta-Analysis of Reliability Coefficients in Second Language Research. The Modern Language Journal, Vol. 100, Issue. 2, p. 538.


    Torres, Julio 2016. Advances in Spanish as a Heritage Language.


    Asención-Delaney, Yuly Collentine, Joseph G. Collentine, Karina Colmenares, Jersus and Plonsky, Luke 2015. El potencial de la enseñanza del vocabulario basada en corpus: optimismo con precaución. Journal of Spanish Language Teaching, Vol. 2, Issue. 2, p. 140.


    Egbert, Jesse and Plonsky, Luke 2015. Success in the abstract: exploring linguistic and stylistic predictors of conference abstract ratings. Corpora, Vol. 10, Issue. 3, p. 291.


    Godfroid, Aline and Spino, Le Anne 2015. Reconceptualizing Reactivity of Think-Alouds and Eye Tracking: Absence of Evidence Is Not Evidence of Absence. Language Learning, Vol. 65, Issue. 4, p. 896.


    Hudson, Thom and Llosa, Lorena 2015. Design Issues and Inference in Experimental L2 Research. Language Learning, Vol. 65, Issue. S1, p. 76.


    Larson-Hall, Jenifer and Plonsky, Luke 2015. Reporting and Interpreting Quantitative Research Findings: What Gets Reported and Recommendations for the Field. Language Learning, Vol. 65, Issue. S1, p. 127.


    Lee, J. Jang, J. and Plonsky, L. 2015. The Effectiveness of Second Language Pronunciation Instruction: A Meta-Analysis. Applied Linguistics, Vol. 36, Issue. 3, p. 345.


    Liu, Qiandi and Brown, Dan 2015. Methodological synthesis of research on the effectiveness of corrective feedback in L2 writing. Journal of Second Language Writing, Vol. 30, p. 66.


    Norris, John M. 2015. Statistical Significance Testing in Second Language Research: Basic Problems and Suggestions for Reform. Language Learning, Vol. 65, Issue. S1, p. 97.


    Plonsky, Luke and Gonulal, Talip 2015. Methodological Synthesis in Quantitative L2 Research: A Review of Reviews and a Case Study of Exploratory Factor Analysis. Language Learning, Vol. 65, Issue. S1, p. 9.


    Purpura, James E. Brown, James Dean and Schoonen, Rob 2015. Improving the Validity of Quantitative Measures in Applied Linguistics Research1. Language Learning, Vol. 65, Issue. S1, p. 37.


    Wulff, Stefanie and Gries, Stefan Th. 2015. Prenominal adjective order preferences in Chinese and German L2 English: A multifactorial corpus study. Linguistic Approaches to Bilingualism, Vol. 5, Issue. 1, p. 122.


    2015. Practice and progression in Second Language Research methods. AILA Review, Vol. 27, p. 80.


    ×
  • Studies in Second Language Acquisition, Volume 35, Issue 4
  • December 2013, pp. 655-687

STUDY QUALITY IN SLA: An Assessment of Designs, Analyses, and Reporting Practices in Quantitative L2 Research

  • Luke Plonsky (a1)
  • DOI: http://dx.doi.org/10.1017/S0272263113000399
  • Published online: 01 August 2013
Abstract

This study assesses research and reporting practices in quantitative second language (L2) research. A sample of 606 primary studies, published from 1990 to 2010 in Language Learning and Studies in Second Language Acquisition, was collected and coded for designs, statistical analyses, reporting practices, and outcomes (i.e., effect sizes). The results point to several systematic strengths as well as many flaws, such as a lack of control in experimental designs, incomplete and inconsistent reporting practices, and low statistical power. I discuss these trends, strengths, and weaknesses in comparison with methodological reviews of L2 research (e.g., Plonsky & Gass, 2011) as well as reviews from other fields (e.g., education, Skidmore & Thompson, 2010). On the basis of the findings, I also make a number of suggestions for methodological reforms in applied linguistics.

Copyright
Corresponding author
*Correspondence concerning this article should be addressed to Luke Plonsky, PO Box 6032, Flagstaff, AZ 86011. E-mail: luke.plonsky@nau.edu
Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

H. Aguinis , C. A. Pierce , F. A. Bosco , & I. S Muslin . (2009). First decade of organizational research methods trends in design, measurement, and data-analysis topics. Organizational Research Methods, 12, 69112.

A. W. Bangert , & J. P Baumberger . (2005). Research and statistical techniques used in the Journal of Counseling & Development. Journal of Counseling & Development ,83, 480487.

S. Brutus , H. Gill , & K Duniewicz . (2010). State-of-science in industrial and organizational psychology: A review of self-reported limitations. Personnel Psychology, 63, 907936.

L. H. Cashen , & S. W Geiger . (2004). Statistical power and the testing of null hypotheses: A review of contemporary management research and recommendations for future studies. Organizational Research Methods, 7, 151167.

A.-W. Chan , A. Hróbjartsson , M. T. Haahr , P. C. Gøtzsche , & D. G Altman . (2004). Empirical evidence for selective reporting of outcomes in randomized trials. Journal of the American Medical Association, 291, 24572465.

C Chaudron . (1986). The interaction of quantitative and qualitative approaches to research: A view of the second language classroom. TESOL Quarterly, 20, 709717.

C Chaudron . (2001). Progress in language classroom research: Evidence from The Modern Language Journal, 1916–2000. Modern Language Journal, 85, 5776.

J Cohen . (1968). Multiple regression as a general data-analytic system. Psychological Bulletin, 70, 426443.

J Cohen . (1994). The earth is round (p < .05). American Psychologist, 49, 971003.

G Crookes . (1991). Power, effect size, and second language research: Another researcher comments. TESOL Quarterly, 25, 762765.

R. DeKeyser , & R Schoonen . (2007). Editors’ announcement. Language Learning, 57, ixx.

T. A DeVaney . (2001). Statistical significance, effect size, and replication: What do the journals say? The Journal of Experimental Education, 69, 310320.

T. H Dinsmore . (2006). Principles, parameters, and SLA: A retrospective meta-analytic investigation into adult L2 learners’ access to Universal Grammar. In J. M. Norris & L. Ortega (Eds.), Synthesizing research on language learning and teaching (pp. 5390). Amsterdam: Benjamins.

S. H. Downs , & N Black . (1998). The feasibility of creating a checklist for the assessment of the methodological quality both of randomized and nonrandomized studies of health care interventions. Journal of Epidemiology & Community Health, 52, 377384.

J Egbert . (2007). Quality analysis of journals in TESOL and applied linguistics. TESOL Quarterly, 41, 157171.

J. L Fleiss . (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76, 378382.

S. Gass , C. Fleck , N. Leder , & I Svetics . (1998). Ahistoricity revisited: Does SLA have a history? Studies in Second Language Acquisition, 20, 407421.

S. M Gass . (1993). Second language acquisition: Cross-disciplinary perspectives. Second Language Research, 9, 9598.

A. Gelman , J. Hill , & M Yajima . (2012). Why we (usually) don’t have to worry about multiple comparisons. Journal of Research on Educational Effectiveness, 5, 189211.

A. Gelman , & D Weakliem . (2009). Of beauty, sex and power: Too little attention has been paid to the statistical challenges in estimating small effects. American Scientist, 97, 310316.

L. D. Goodwin , & W. L Goodwin . (1985). An analysis of statistical techniques used in the Journal of Educational Psychology, 1979–1983. Educational Psychologist, 20, 1321.

E Hatch . (1978). Apply with caution. Studies in Second Language Acquisition, 2, 123143.

G Henning . (1986). Quantitative methods in language acquisition research. TESOL Quarterly, 20, 701708.

L. G Humphreys . (1978). Doing research the hard way: Substituting analysis of variance for a problem in correlational analysis. Journal of Educational Psychology, 70, 873876.

Journal Article Reporting Standards Working Group. (2008). Reporting standards for research in psychology: Why do we need them? What might they be? American Psychologist, 63, 839851.

H. J. Keselman , C. J. Huberty , L. M. Lix , S. Olejnik , R. A. Cribbie , B. Donahue , . . . J. R Levin . (1998). Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA, and ANCOVA analyses. Review of Educational Research, 68, 350386.

K. M. Kieffer , R. J. Reese , & B Thompson . (2001). Statistical techniques employed in AERJ and JCP articles from 1988 to 1997: A methodological review. The Journal of Experimental Education, 69, 280309.

M Kubanyiova . (2008). Rethinking research ethics in contemporary applied linguistics: The tension between macroethical and microethical perspectives in situated research. Modern Language Journal, 92, 503518.

J. R. Landis , & G. G Koch . (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159174.

J. Larson-Hall , & R Herrington . (2010). Improving data analysis in second language acquisition by utilizing modern developments in applied statistics. Applied Linguistics, 31, 368–190.

A Lazaraton . (1991). Power, effect size, and second language research: A researcher comments. TESOL Quarterly, 25, 759762.

A Lazaraton . (2000). Current trends in research methodology and statistics in applied linguistics. TESOL Quarterly, 34, 175181.

A. Lazaraton , H. Riggenbach , & A Ediger . (1987). Forming a discipline: Applied linguists’ literacy in research methodology and statistics. TESOL Quarterly, 21, 263277.

S Li . (2010). The effectiveness of corrective feedback in SLA: A meta-analysis. Language Learning, 60, 309365.

P. M Lightbown . (2000). Anniversary article: Classroom second language research and second language teaching. Applied Linguistics, 21, 431462.

S Loewen . (2005). Incidental focus on form and second language learning. Studies in Second Language Acquisition, 27, 361386.

D. E Lykken . (1968). Statistical significance in psychological research. Psychological Bulletin, 70, 151159.

R. Lyster , & J Izquierdo . (2009). Prompts versus recasts in dyadic interaction. Language Learning, 59, 453498.

A. Mackey , & S. M Gass . (Eds.). (2012). Research methods in second language acquisition: A practical guide. Oxford: Wiley-Blackwell.

S. S Magnan . (1994). From the editor: The MLJ tradition and the challenges ahead. Modern Language Journal, 78, 79.

M. S. Matthews , M. Gentry , D. B. McCoach , F. C. Worrell , D. Matthews , & F Dixon . (2008). Evaluating the state of a field: Effect size reporting in gifted education. The Journal of Experimental Education, 77, 5565.

S. T. Meier , & S. R Davis . (1990). Trends in reporting psychometric properties of scales used in counseling psychology research. Journal of Counseling Psychology, 37, 113115.

M. A. Mone , G. C. Mueller , & W Mauland . (1996). The perceptions and usage of statistical power in applied psychology and management research. Personnel Psychology, 49, 103120.

E. Nicoladis , & A Krott . (2007). Word family size and French-speaking children’s segmentation of existing compounds. Language Learning, 57, 201228.

J. M. Norris , & L Ortega . (2000). Effectiveness of L2 instruction: A research synthesis and quantitative meta-analysis. Language Learning, 50, 417528.

L Ortega . (2005). Methodology, epistemology, and ethics in instructed SLA research: An introduction. Modern Language Journal, 89, 317327.

T Pica . (1997). Second language teaching and research relationships: A North American view. Language Teaching Research, 1, 4872.

L Plonsky . (2011a). The effectiveness of second language strategy instruction: A meta-analysis. Language Learning, 61, 9931038.

L. Plonsky , & S Gass . (2011). Quantitative research methods, study quality, and outcomes: The case of interaction research. Language Learning, 61, 325366.

L. Plonsky , & F. L Oswald . (2012). How to do a meta-analysis. In A. Mackey & S. Gass (Eds.), Research methods in second language acquisition: A practical guide (pp. 275295). Oxford: Wiley-Blackwell.

C Polio . (1997). Measures of linguistic accuracy in second language writing research. Language Learning, 47, 101143.

G Porte . (2010) Appraising research in second language learning: A practical approach to critical analysis of quantitative research (2nd ed.). Amsterdam: Benjamins.

D Pulido . (2004). The relationship between text comprehension and second language incidental vocabulary acquisition: A matter of topic familiarity? Language Learning, 54, 469523.

F. L Schmidt . (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training researchers. Psychological Methods, 1, 115129.

P. Sedlmeier , & G Gigerenzer . (1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309316.

L. Selinker , & U Lakshmanan . (2001). How do we know what we know? Why do we believe what we believe? Second Language Research, 17, 323325.

S. T. Skidmore , & B Thompson . (2010). Statistical techniques used in published articles: A historical review of reviews. Educational and Psychological Measurement, 70, 777795.

B. Smith , & B. A Lafford . (2009). The evaluation of scholarly activity in computer-assisted language learning. Modern Language Journal, 93, 868883.

S. Sun , W. Pan , & L. L Wang . (2010). A comprehensive review of effect size reporting and interpreting practices in academic journals in education and psychology. Journal of Educational Psychology, 102, 9891004.

B Thompson . (2001). Significance, effect sizes, stepwise methods, and other issues: Strong arguments move the field. The Journal of Experimental Education, 70, 8093.

B. Thompson , & P. A Snyder . (1998). Statistical significance and reliability analyses in recent JCD research articles. Journal of Counseling and Development, 76, 436441.

T. Vacha-Haase , C. Ness , J. Nilsson , & D Reetz . (1999). Practices regarding reporting of reliability coefficients: A review of three journals. The Journal of Experimental Education, 67, 335341.

T. Vacha-Haase , & B Thompson . (2004). How to estimate and interpret various effect sizes. Journal of Counseling Psychology, 51, 473481.

A Valdman . (1998). A note from the editor: 20th anniversary of SSLA. Studies in Second Language Acquisition, 20, 463470.

J. C. Valentine , & H Cooper . (2008). A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: The study design and implementation assessment device (Study DIAD). Psychological Methods, 13, 130149.

H. Z Waring . (2009). Moving out of IRF (initiation-response-feedback): A single case analysis. Language Learning, 59, 796824.

C. S. Wells , & J. M Hintze . (2007). Dealing with assumptions underlying statistical tests. Psychology in the Schools, 44, 495502.

K. Wells , & J. H Littell . (2009). Study quality assessment in systematic reviews of research on intervention effects. Research on Social Work Practice, 19, 5262.

L. Wilkinson , & Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594604.

V. L Willson . (1980). Research techniques in AERJ articles: 1969 to 1978. Educational Researcher, 9, 510.

Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Studies in Second Language Acquisition
  • ISSN: 0272-2631
  • EISSN: 1470-1545
  • URL: /core/journals/studies-in-second-language-acquisition
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×