Hostname: page-component-76d6cb85b7-ntvhh Total loading time: 0 Render date: 2026-07-18T23:43:04.161Z Has data issue: false hasContentIssue false
  • English
  • Français

Analysis of rating scales: A pervasive problem in bilingualism research and a solution with Bayesian ordinal models

Published online by Cambridge University Press:  01 September 2021

João Veríssimo Open the ORCID record for João Veríssimo [Opens in a new window]
João Veríssimo*
Affiliation:
Potsdam Research Institute for Multilingualism, Potsdam, Germany Center of Linguistics, School of Arts and Humanities, University of Lisbon, Lisbon, Portugal
*
Address for correspondence: João Veríssimo, Faculdade de Letras da Universidade de Lisboa, Alameda da Universidade, 1600-214 Lisboa, Portugal. E-mail: jlverissimo@edu.ulisboa.pt
Get access Rights & Permissions [Opens in a new window]

Abstract

Research in bilingualism often involves quantifying constructs of interest by the use of rating scales: for example, to measure language proficiency, dominance, or sentence acceptability. However, ratings are a type of ordinal data, which violates the assumptions of the statistical methods that are commonly used to analyse them. As a result, the validity of ratings is compromised and the ensuing statistical inferences can be seriously distorted. In this article, we describe the problem in detail and demonstrate its pervasiveness in bilingualism research. We then provide examples of how bilingualism researchers can employ an appropriate solution using Bayesian ordinal models. These models respect the inherent discreteness of ratings, easily accommodate non-normality, and allow modelling unequal psychological distances between response categories. As a result, they can provide more valid, accurate, and informative inferences about graded constructs such as language proficiency. Data and code are publicly available in an OSF repository at https://osf.io/grs8x.

Keywords

rating scalesordinal modelsBayesian analysislanguage proficiencyacceptability judgements

Information

Type
Review Article
Information
Bilingualism: Language and Cognition , Volume 24 , Issue 5 , November 2021 , pp. 842 - 848
Check for updates
Copyright
Copyright © The Author(s), 2021. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Aust, F, & Barth, M (2020). papaja: Create APA manuscripts with R Markdown. R package version 0.1.0.9942. Retrieved from https://github.com/crsh/papajaGoogle Scholar
Birdsong, D, Gertken, LM, & Amengual, M (2012). Bilingual Language Profile: An easy-to-use instrument to assess bilingualism. [Measurement instrument]. Retrieved from https://sites.la.utexas.edu/bilingual/Google Scholar
Box, GEP (1976). Science and statistics. Journal of the American Statistical Association 71(356), 791–799. https://doi.org/gdm28wCrossRefGoogle Scholar
Bürkner, P.-C (2017). brms: An R package for Bayesian multilevel models using Stan. Journal of Statistical Software 80(1), 1–28. https://doi.org/gddxwpCrossRefGoogle Scholar
Bürkner, P.-C (2018). Advanced Bayesian multilevel modeling with the R package brms. The R Journal 10(1), 395–411. https://doi.org/gfxzpnCrossRefGoogle Scholar
Bürkner, P.-C, & Vuorre, M (2019). Ordinal regression models in psychology: A tutorial. Advances in Methods and Practices in Psychological Science 2(1), 77–101. https://doi.org/gfv26qCrossRefGoogle Scholar
Cho, J, & Slabakova, R (2014). Interpreting definiteness in a second language without articles: The case of L2 Russian. Second Language Research 30(2), 159–190. https://doi.org/ggktx6CrossRefGoogle Scholar
Christensen, RHB (2019). ordinal - Regression models for ordinal data. R package version 2019.12-10. Retrieved from https://CRAN.R-project.org/package=ordinalGoogle Scholar
Cohen, J (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
Dawes, J (2008). Do data characteristics change according to the number of scale points used? An experiment using 5-point, 7-point and 10-point scales. International Journal of Market Research 50(1), 61–104. https://doi.org/ggktxkCrossRefGoogle Scholar
DeCastellarnau, A (2018). A classification of response scale characteristics that affect data quality: A literature review. Quality & Quantity 52(4), 1523–1559. https://doi.org/gdqv89CrossRefGoogle ScholarPubMed
Douven, I (2018). A Bayesian perspective on Likert scales and central tendency. Psychonomic Bulletin & Review 25(3), 1203–1211. https://doi.org/gf5sjsCrossRefGoogle ScholarPubMed
Farhy, Y, & Veríssimo, J (2019). Semantic effects in morphological priming: The case of Hebrew stems. Language and Speech 62(4), 737–750. https://doi.org/ggkts3CrossRefGoogle ScholarPubMed
Flege, JE, Yeni-Komshian, GH, & Liu, S (1999). Age constraints on second-language acquisition. Journal of Memory and Language 41(1), 78–104. https://doi.org/dwfs84CrossRefGoogle Scholar
Glass, GV, McGaw, B, & Smith, ML (1981). Meta-analysis in social research. Beverly Hills: Sage Publications.Google Scholar
Hakuta, K, Bialystok, E, & Wiley, E (2003). Critical evidence: A test of the critical-period hypothesis for second-language acquisition. Psychological Science 14(1), 31–38. https://doi.org/ffjrhsCrossRefGoogle ScholarPubMed
Hopp, H (2009). The syntax–discourse interface in near-native L2 acquisition: Off-line and on-line performance. Bilingualism: Language and Cognition 12(4), 463–483. https://doi.org/dk5xtwCrossRefGoogle Scholar
Hulstijn, JH (2012). The construct of language proficiency in the study of bilingualism from a cognitive perspective. Bilingualism: Language and Cognition 15(2), 422–433. https://doi.org/ggktxvCrossRefGoogle Scholar
Kissling, EM (2018). Pronunciation instruction can improve L2 learners’ bottom-Up processing for listening. The Modern Language Journal 102(4), 653–675. https://doi.org/gfkf3jCrossRefGoogle Scholar
Krosnick, JA, & Presser, S (2010). Question and questionnaire design. In Marsden, P. V. & D., James (Eds.), Handbook of survey research (Second edition). Bingley, UK: Emerald.Google Scholar
Kuncel, RB (1977). The subject-item Interaction in itemmetric research. Educational and Psychological Measurement 37(3), 665–678. https://doi.org/bjrbmtCrossRefGoogle Scholar
Lemhöfer, K, & Broersma, M (2012). Introducing LexTALE: A quick and valid Lexical Test for Advanced Learners of English. Behavior Research Methods 44(2), 325–343. https://doi.org/c9f897CrossRefGoogle ScholarPubMed
Li, P, Sepanski, S, & Zhao, X (2006). Language History Questionnaire: A Web-based interface for bilingual research. Behavior Research Methods 38(2), 202–210. https://doi.org/fph9q8CrossRefGoogle ScholarPubMed
Liddell, TM, & Kruschke, JK (2018). Analyzing ordinal data with metric models: What could possibly go wrong? Journal of Experimental Social Psychology 79, 328–348. https://doi.org/gfdbv8CrossRefGoogle Scholar
Marian, V, Blumenfeld, HK, & Kaushanskaya, M (2007). The Language Experience and Proficiency Questionnaire (LEAP-Q): Assessing language profiles in bilinguals and multilinguals. Journal of Speech, Language, and Hearing Research 50(4), 940–967. https://doi.org/bt2xwbCrossRefGoogle ScholarPubMed
Matuschek, H, Kliegl, R, Vasishth, S, Baayen, RH, & Bates, D (2017). Balancing Type I error and power in linear mixed models. Journal of Memory and Language 94, 305–315. https://doi.org/gcx746CrossRefGoogle Scholar
McCullagh, P (1980). Regression models for ordinal data. Journal of the Royal Statistical Society: Series B (Methodological) 42(2), 109–127. https://doi.org/ggntw8Google Scholar
McElreath, R (2020). Statistical rethinking: A Bayesian course with examples in R and Stan (2nd ed.). CRC Press.CrossRefGoogle Scholar
McKelvey, RD, & Zavoina, W (1975). A statistical model for the analysis of ordinal level dependent variables. The Journal of Mathematical Sociology 4(1), 103–120. https://doi.org/dqfhppCrossRefGoogle Scholar
Puebla, C (2016). L2 proficiency survey. Unpublished raw data, Potsdam Research Institute for Multilingualism, University of Potsdam.Google Scholar
R Core Team. (2020). R: A language and environment for statistical computing. Vienna, Austria. Retrieved from https://www.R-project.org/Google Scholar
Schad, DJ, Betancourt, M, & Vasishth, S (2020). Toward a principled Bayesian workflow in cognitive science. Manuscript submitted for publication. Retrieved from https://arxiv.org/abs/1904.12765Google Scholar
Schad, DJ, Vasishth, S, Hohenstein, S, & Kliegl, R (2020). How to capitalize on a priori contrasts in linear (mixed) models: A tutorial. Journal of Memory and Language 110, 104038. https://doi.org/gf9tjpCrossRefGoogle Scholar
Schlenter, J (2019). Predictive language processing in late bilinguals (Doctoral dissertation). Universität Potsdam. https://doi.org/10.25932/publishup-43249CrossRefGoogle Scholar
Schwarz, N (1999). Self-reports: How the questions shape the answers. American Psychologist 54(2), 93–105. https://doi.org/fqrx56CrossRefGoogle Scholar
Tare, M, Golonka, E, Lancaster, AK, Bonilla, C, Doughty, CJ, Belnap, RK, & Jackson, SR (2018). The role of cognitive aptitudes in a study abroad language-learning environment. In Sanz, C & Morales-Front, A (Eds.), The Routledge Handbook of Study Abroad Research and Practice (1st ed.). Routledge, pp. 406–420 https://doi.org/10.4324/9781315639970-27CrossRefGoogle Scholar
Tomoschuk, B, Ferreira, VS, & Gollan, TH (2019). When a seven is not a seven: Self-ratings of bilingual language proficiency differ between and within language populations. Bilingualism: Language and Cognition 22(3), 516–536. https://doi.org/gfkm58CrossRefGoogle Scholar
Vasishth, S, Nicenboim, B, Beckman, ME, Li, F, & Kong, EJ (2018). Bayesian data analysis in the phonetic sciences: A tutorial introduction. Journal of Phonetics 71, 147–161. https://doi.org/gfzq3cCrossRefGoogle ScholarPubMed
Zell, E, & Krizan, Z (2014). Do people have insight into their abilities? A metasynthesis. Perspectives on Psychological Science 9(2), 111–125. https://doi.org/f5t93sCrossRefGoogle ScholarPubMed
Supplementary material: PDF

Veríssimo supplementary material

Veríssimo supplementary material

Download Veríssimo supplementary material(PDF)
PDF 87.8 KB