Hostname: page-component-54dcc4c588-scsgl Total loading time: 0 Render date: 2025-10-07T12:05:10.782Z Has data issue: false hasContentIssue false
  • English
  • Français

Study quality as an intellectual and ethical imperative: A proposed framework

Published online by Cambridge University Press:  12 April 2024

Luke Plonsky
Luke Plonsky*
Affiliation:
Northern Arizona University, USA
*
Email: lukeplonsky@gmail.com
Get access Rights & Permissions [Opens in a new window]

Abstract

What is quality in the context of applied linguistic research? Addressing this question is both an intellectual and ethical imperative for the field. Toward that end, I propose a four-part framework for understanding and evaluating study quality: Quality research is (a) methodologically rigorous, (b) transparent, (c) ethical, and (d) of value to society. The bulk of the paper is devoted to describing each of these four elements, first conceptually and then as observed in the field. I also articulate some of the many connections among the four elements within the framework. In addition, I suggest next steps for further addressing the notion of quality in terms of the framework itself as well as the ways it might be used to advance our field’s research, training, and legitimacy as an academic discipline that is still in some ways coming of age.

Keywords

study qualitymethodological rigorethics

Information

Type
Research Article
Information
Annual Review of Applied Linguistics , Volume 44 , March 2024 , pp. 4 - 18
Check for updates
Copyright
© The Author(s), 2024. Published by Cambridge University Press.

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Al-Hoorie, A. H., Hiver, P., & In’nami, Y. (in press). The validation crisis in the L2 motivational self system tradition. Studies in Second Language Acquisition.Google Scholar
Al-Hoorie, A. H., & Vitta, J. P. (2019). The seven sins of L2 research: A review of 30 journals’ statistical quality and their CiteScore, SJR, SNIP, JCR Impact Factors. Language Teaching Research, 23(6), 727–744.CrossRefGoogle Scholar
Andringa, S., & Godfroid, A. (2020). Sampling bias and the problem of generalizability in applied linguistics. Annual Review of Applied Linguistics, 40, 134–142.CrossRefGoogle Scholar
Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018). Journal article reporting standards for quantitative research in psychology: The APA publications and communications board task force report. American Psychologist, 73(1), 3–25.CrossRefGoogle ScholarPubMed
Arndt, H. L. (2023). Construction and validation of a questionnaire to study engagement in informal second language learning. Studies in Second Language Acquisition, 45(5), 1456–1480.CrossRefGoogle Scholar
Baralt, M., Mahoney, A. D., & Brito, N. (2020). Háblame Bebé: A phone application intervention to support Hispanic children’s early language environments and bilingualism. Child Language Teaching and Therapy, 36(1), 33–57.CrossRefGoogle Scholar
Bigelow, M., & Tarone, E. (2004). The role of literacy level in second language acquisition: Doesn’t who we study determine what we know? TESOL Quarterly, 38(4), 689–700.CrossRefGoogle Scholar
Burston, J., Athanasiou, A., & Giannakou, K. (2024). Quantitative experimental L2 acquisition MALL studies: A critical evaluation of research quality. ReCALL, 36(1), 22–39.CrossRefGoogle Scholar
Bylund, E., Khafif, Z., & Berghoff, R. (2023). Linguistic and geographic diversity in research on second language acquisition and multilingualism: An analysis of selected journals. Applied Linguistics.Google Scholar
Chapelle, C. (2021). Validity in language assessment. In Winke, P. & Brunfaut, T. (Eds.), The Routledge handbook of second language acquisition and language testing (pp. 11–20). Routledge.Google Scholar
Chapelle, C., & Duff, P. (2003). Some guidelines for conducting quantitative and qualitative research in TESOL. TESOL Quarterly, 37(1), 157–178.CrossRefGoogle Scholar
Chong, S. W. (2020). The role of research synthesis in facilitating research–pedagogy dialogue. ELT Journal, 74(4), 484–487.CrossRefGoogle Scholar
Cohen, J. (1997). The earth is round (p <.05). In Harlow, L. L., Mulaik, S. A. & Steiger, J. H. (Eds.), What if there were no significance tests? (pp. 21–35). Lawrence Erlbaum.Google Scholar
Cohen, A. D., & Macaro, E. (2013). Research methods in second language acquisition. In Macaro, E. (Ed.), The Bloomsbury companion to second language acquisition (pp. 107–133). Bloomsbury.Google Scholar
Coss, M., & Hwang, H.-B. (2024). Issues with pedagogical implications in applied linguistics research: A mixed-methods systematic evaluation. Research Methods in Applied Linguistics, 3(1), 100094.CrossRefGoogle Scholar
Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25(1), 7–29.CrossRefGoogle Scholar
De Costa, P. I. (Ed.) (2016). Ethics in applied linguistics research: Language researcher narratives. Routledge.Google Scholar
De Costa, P., Sterling, S., Lee, J., Li, W., & Rawal, W. (2021). Research tasks on ethics in applied linguistics. Language Teaching, 54(1), 58–70.CrossRefGoogle Scholar
Driver, M. Y. (in press). Measuring and understanding linguistic insecurity in heritage and foreign language contexts: Design and validation of a novel scale. Journal of Multilingual and Multicultural Development.Google Scholar
Ellis, R. (2021). A short history of SLA: Where have we come from and where are we going? Language Teaching, 54(2), 190–205.CrossRefGoogle Scholar
Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS One, 4(5), e5738.CrossRefGoogle ScholarPubMed
Gass, S., Fleck, C., Leder, N., & Svetics, I. (1998). Ahistoricity revisited: Does SLA have a history? Studies in Second Language Acquisition, 20(3), 407–421.CrossRefGoogle Scholar
Gass, S., Loewen, S., & Plonsky, L. (2021). Coming of age: The past, present, and future of quantitative SLA research. Language Teaching, 54(2), 245–258.CrossRefGoogle Scholar
Goetze, J., & Driver, M. (2022). Is learning really just believing? A meta-analysis of self-efficacy and achievement in SLA. Studies in Second Language Learning and Teaching, 12(2), 233–259.CrossRefGoogle Scholar
Gönülal, T., Loewen, S., & Plonsky, L. (2017). The development of statistical literacy in applied linguistics graduate students. International Journal of Applied Linguistics, 168(1), 4–32.CrossRefGoogle Scholar
Harding, L., & Winke, P. (2022). Innovation and expansion in language testing for changing times. Language Testing, 39(1), 3–6.CrossRefGoogle Scholar
Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 61–83.CrossRefGoogle ScholarPubMed
Hiver, P., & Al-Hoorie, A. H. (2020). Reexamining the role of vision in second language motivation: A preregistered conceptual replication of You, Dörnyei, and Csizér (2016). Language Learning, 70(1), 48–102.CrossRefGoogle Scholar
Hiver, P., Al-Hoorie, A. H., & Evans, R. (2022). Complex dynamic systems theory in language learning: A scoping review of 25 years of research. Studies in Second Language Acquisition, 44(4), 913–941.CrossRefGoogle Scholar
Hu, Y., & Plonsky, L. (2021). Statistical assumptions in L2 research: A systematic review. Second Language Research, 37(1), 171–184.CrossRefGoogle Scholar
In’nami, Y., Mizumoto, A., Plonsky, L., & Koizumi, R. (2022). Promoting computationally reproducible research in applied linguistics: Recommended practices and considerations. Research Methods in Applied Linguistics, 1(3), 100030.Google Scholar
Isaacs, T., & Chalmers, H. (in press). Reducing ‘avoidable research waste’ in applied linguistics research: Insights from healthcare research. Language Teaching.Google Scholar
Isbell, D., Brown, D., Chen, M., Derrick, D., Ghanem, R., Gutiérrez Arvizu, M. N., & Plonsky, L. (2022). Misconduct and questionable research practices: The ethics of quantitative data handling and reporting in applied linguistics. Modern Language Journal, 106(1), 172–195.CrossRefGoogle Scholar
Isbell, D. R., & Kim, J. (2023). Developer involvement and COI disclosure in high-stakes English proficiency test validation research: A systematic review. Research Methods in Applied Linguistics, 2(3), 100060.CrossRefGoogle Scholar
Klein, R. B. (2013). Beyond significance testing: Statistics reform in the behavioral sciences (2nd ed.). American Psychological Association.CrossRefGoogle Scholar
Kubanyiova, M. (2008). Rethinking research ethics in contemporary applied linguistics: The tension between macroethical and microethical perspectives in situated research. Modern Language Journal, 92(4), 503–518.CrossRefGoogle Scholar
Kubanyiova, M. (2013). Ethical debates in research on language and interaction. In Chapelle, C. A. (Ed.), The encyclopedia of applied linguistics (pp. 2001–2008). Blackwell.Google Scholar
Larson-Hall, J. (2017). Moving beyond the bar plot and the line graph to create informative and attractive graphics. Modern Language Journal, 101(1), 244–270.CrossRefGoogle Scholar
Larsson, T., Plonsky, L., Sterling, S., Kytö, M., Yaw, K., & Wood, M. (2023). On the frequency, prevalence, and perceived severity of questionable research practices. Research Methods in Applied Linguistics, 2(3), 100064.CrossRefGoogle Scholar
Li, S. (2015). The associations between language aptitude and second language grammar acquisition: A meta-analytic review of five decades of research. Applied Linguistics, 36(3), 385–408.CrossRefGoogle Scholar
Li, S. (2023). Working memory and second language writing: A systematic review. Studies in Second Language Acquisition, 45(S3), 647–679.CrossRefGoogle Scholar
Lindstromberg, S. (2022). P-curving as a safeguard against p-hacking in SLA research: A case study. Studies in Second Language Acquisition, 44(4), 1155–1180.CrossRefGoogle Scholar
Loewen, S., Gonulal, T., Isbell, D. R., Ballard, L., Crowther, D., Lim, J., & Tigchelaar, M. (2020). How knowledgeable are applied linguistics and SLA researchers about basic statistics? Data from North America and Europe. Studies in Second Language Acquisition, 42(4), 871–890.CrossRefGoogle Scholar
Loewen, S., & Hui, B. (2021). Small samples in instructed second language acquisition research. Modern Language Journal, 105(1), 187–193.CrossRefGoogle Scholar
Mackey, A. (2020). Interaction, feedback and task research in second language learning: Methods and design. Cambridge University Press.CrossRefGoogle Scholar
Mackey, A., & Gass, S. M. (2022). Second language research: Methodology and design (3rd ed.). Routledge.Google Scholar
Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P., & Glasziou, P. (2014). Biomedical research: Increasing value, reducing waste. Lancet, 383(9912), 101–104.CrossRefGoogle ScholarPubMed
Mahboob, A., Paltridge, B., Phakiti, A., Wagner, E., Starfield, S., Burns, A., & De Costa, P. I. (2016). TESOL quarterly research guidelines. TESOL Quarterly, 50(1), 42–65.CrossRefGoogle Scholar
Marsden, E. (2020). Methodological transparency and its consequences for the quality and scope of research. In McKinley, J. & Rose, H. (Eds.), The Routledge handbook of research methods in applied linguistics (pp. 15–28). Routledge.Google Scholar
Marsden, E., Alferink, I., Andringa, S., Bolibaugh, C., Collins, L., Jackson, C., & Plonsky, L. (2018a). Open Accessible Summaries in Language Studies (OASIS) [Database]. https://www.oasis-database.org.Google Scholar
Marsden, E., & Kasprowicz, R. (2017). Foreign language educators’ exposure to research: Reported experiences, exposure via citations, and a proposal for action. Modern Language Journal, 101(4), 613–642.CrossRefGoogle Scholar
Marsden, E., Mackey, A., & Plonsky, L. (2016). Breadth and depth: The IRIS repository. In Mackey, A. & Marsden, E. (Eds.), Advancing methodology and practice: The IRIS repository of instruments for research into second languages (pp. 1–21). Routledge.Google Scholar
Marsden, E., Morgan-Short, K., Thompson, S., & Abugaber, D. (2018). Replication in second language research: Narrative and systematic reviews and recommendations for the field. Language Learning, 68(2), 321–391.CrossRefGoogle Scholar
Merton, R. K. (1942). The normative structure of science. In Merton, R. K. (Ed.), The sociology of science: Theoretical and empirical investigations (pp. 267–278). University of Chicago Press.Google Scholar
Nicklin, C., & Plonsky, L. (2020). Outliers in L2 research in applied linguistics: A synthesis and data re-analysis. Annual Review of Applied Linguistics, 40, 26–55.CrossRefGoogle Scholar
Nicklin, C., & Vitta, J. P. (2021). Effect‐driven sample sizes in second language instructed vocabulary acquisition research. Modern Language Journal, 105(1), 218–236.CrossRefGoogle Scholar
Norouzian, R. (2020). Sample size planning in quantitative L2 research: A pragmatic approach. Studies in Second Language Acquisition, 42(4), 849–870.CrossRefGoogle Scholar
Norris, J. M. (2015). Statistical significance testing in second language research: Basic problems and suggestions for reform. Language Learning, 65(Supp. 1), 97–126.CrossRefGoogle Scholar
Norris, J. M., & Ortega, L. (2000). Effectiveness of L2 instruction: A research synthesis and quantitative meta-analysis. Language Learning, 50, 417–528.CrossRefGoogle Scholar
Norris, J. M., & Ortega, L. (2006). The value and practice of research synthesis for language learning and teaching. In Norris, J. M. & Ortega, L. (Eds.), Synthesizing research on language learning and teaching (pp. 3–50). Benjamins.CrossRefGoogle Scholar
Norris, J. M., & Ortega, L. (2012). Assessing learner knowledge. In Gass, S. M. & Mackey, A. (Eds.), The Routledge handbook of second language acquisition (pp. 573–589). Routledge.Google Scholar
Norris, J. M., Plonsky, L., Ross, S. J., & Schoonen, R. (2015). Guidelines for reporting quantitative methods and results in primary research. Language Learning, 65, 470–476.CrossRefGoogle Scholar
Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science, 349 (6251).Google Scholar
Ortega, L. (Ed.) (2005). Methodology, epistemology, and ethics in instructed SLA research SLA research: An introduction. Modern Language Journal, 89(3), 317–327.CrossRefGoogle Scholar
Ortega, L. (2009). Understanding second language acquisition. Routledge.Google Scholar
Ortega, L. (2012). Epistemological diversity and moral ends of research in instructed SLA. Language Teaching Research, 16(2), 206–226.CrossRefGoogle Scholar
Pavlenko, A., Hepford, E., & Jarvis, S. (2020). An illusion of understanding: How native and non-native speakers of English understand (and misunderstand) their Miranda rights. The International Journal of Speech, Language and the Law, 26(2), 181–207.CrossRefGoogle Scholar
Plonsky, L. (2011). Study quality in SLA: A cumulative and developmental assessment of designs, analyses, reporting practices, and outcomes in quantitative L2 research. [Unpublished doctoral dissertation]. Michigan State University.Google Scholar
Plonsky, L. (2013). Study quality in SLA: An assessment of designs, analyses, and reporting practices in quantitative L2 research. Studies in Second Language Acquisition, 35(4), 655–687.CrossRefGoogle Scholar
Plonsky, L. (2014). Study quality in quantitative L2 research (1990–2010): A methodological synthesis and call for reform. Modern Language Journal, 98(1), 450–470.CrossRefGoogle Scholar
Plonsky, L. (2023a). Introducing bibliometrics in applied linguistics. Studies in Second Language Learning and Teaching, 13(4), 721–726.CrossRefGoogle Scholar
Plonsky, L. (2023b). Sampling and generalizability in Lx research: A second order synthesis. Languages, 8(1), 75.CrossRefGoogle Scholar
Plonsky, L., Brown, D., Chen, M., Ghanem, R., Gutiérrez Arvizu, M. N., Isbell, D. R., & Zhang, M. (2024). “Significance sells”: Applied linguists’ view on questionable research practices. Research Methods in Applied Linguistics, 3, 100099.CrossRefGoogle Scholar
Plonsky, L., Hu, Y., Sudina, E., & Oswald, F. L. (2023). Advancing meta-analytic methods in L2 research. In Mackey, A. & Gass, S. (Eds.), Current approaches in second language acquisition research (pp. 304–333). Wiley Blackwell.CrossRefGoogle Scholar
Plonsky, L., Larsson, T., Sterling, S., Kytö, M., Yaw, K., & Wood, M. (in press). A taxonomy of questionable research practices in quantitative humanities. In Costa, P. I. De, Rabie-Ahmed, A. & Cinaglia, C. (Eds.), Ethical issues in applied linguistics scholarship. Benjamins.Google Scholar
Plonsky, L., Marsden, E., Crowther, D., Gass, S., & Spinner, P. (2020). A methodological synthesis of judgment tasks in second language research. Second Language Research, 36(4), 583–621.CrossRefGoogle Scholar
Porte, G., & McManus, K. (2019). Doing replication research in applied linguistics. Routledge.Google Scholar
Saito, K., Petrova, K., Suzukida, Y., Kachlicka, M., & Tierney, A. (2022). Training auditory processing promotes second language speech acquisition. Journal of Experimental Psychology: Human Perception and Performance, 48(12), 1410–1426.Google ScholarPubMed
Sato, M., & Loewen, S. (2022). The research–practice dialogue in second language learning and teaching: Past, present, and future. Modern Language Journal, 106(3), 509–527.CrossRefGoogle Scholar
Schmitt, N. (2019). Understanding vocabulary acquisition, instruction, and assessment: A research agenda. Language Teaching, 52(2), 261–274.CrossRefGoogle Scholar
Sterling, S., & Gass, S. (2017). Exploring the boundaries of research ethics: Perceptions of ethics and ethical behaviors in applied linguistics research. System, 70, 50–62.CrossRefGoogle Scholar
Sterling, S., Yaw, K., Larsson, T., Plonsky, L., & Kytö, M. (in preparation). A qualitative analysis of questionable research practices. Manuscript in preparation.Google Scholar
Sudina, E. (2021). Study and scale quality in second language survey research, 2009–2019: The case of anxiety and motivation. Language Learning, 71(4), 1149–1193.CrossRefGoogle Scholar
Sudina, E. (2023a). A primer on measurement invariance in L2 anxiety research. Annual Review of Applied Linguistics, 43, 140–146.CrossRefGoogle Scholar
Sudina, E. (2023b). Scale quality in second-language anxiety and WTC: A methodological synthesis. Studies in Second Language Acquisition, 45(5), 1427–1455.CrossRefGoogle Scholar
Tobia, K., Egbert, J., & Lee, T. R. (2023). Triangulating ordinary meaning (May 8, 2023). Georgetown Law Journal Online, 112, 23–54.Google Scholar
Vacha-Haase, T., & Thompson, B. (2004). How to estimate and interpret various effect sizes. Journal of Counseling Psychology, 51(4), 473–481.CrossRefGoogle Scholar
Vitta, J. P., Nicklin, C., & McLean, S. (2022). Effect size-driven sample-size planning, randomization, and multisite use in L2 instructed vocabulary acquisition experimental samples. Studies in Second Language Acquisition, 44(5), 1424–1448.CrossRefGoogle Scholar
Wei, R., Hu, Y., & Xiong, J. (2019). Effect size reporting practices in applied linguistics research: A study of one major journal. SAGE Open, 9(2).CrossRefGoogle Scholar
Wood, M., Sterling, S., Larsson, T., Plonsky, L., Kytö, M., & Yaw, K. (in press). Researchers training researchers: Ethics training in applied linguistics. TESOL Quarterly.Google Scholar
Yaw, K., Plonsky, L., Larsson, T., Sterling, S., & Kytö, M. (2023). Timeline: Research ethics in applied linguistics. Language Teaching, 56(4), 478–494.CrossRefGoogle Scholar