Hostname: page-component-5db6c4db9b-qvlvc Total loading time: 0 Render date: 2023-03-25T12:06:43.595Z Has data issue: true Feature Flags: { "useRatesEcommerce": false } hasContentIssue true

Analysis of rating scales: A pervasive problem in bilingualism research and a solution with Bayesian ordinal models

Published online by Cambridge University Press:  01 September 2021

João Veríssimo*
Potsdam Research Institute for Multilingualism, Potsdam, Germany Center of Linguistics, School of Arts and Humanities, University of Lisbon, Lisbon, Portugal
Address for correspondence: João Veríssimo, Faculdade de Letras da Universidade de Lisboa, Alameda da Universidade, 1600-214 Lisboa, Portugal. E-mail:


Research in bilingualism often involves quantifying constructs of interest by the use of rating scales: for example, to measure language proficiency, dominance, or sentence acceptability. However, ratings are a type of ordinal data, which violates the assumptions of the statistical methods that are commonly used to analyse them. As a result, the validity of ratings is compromised and the ensuing statistical inferences can be seriously distorted. In this article, we describe the problem in detail and demonstrate its pervasiveness in bilingualism research. We then provide examples of how bilingualism researchers can employ an appropriate solution using Bayesian ordinal models. These models respect the inherent discreteness of ratings, easily accommodate non-normality, and allow modelling unequal psychological distances between response categories. As a result, they can provide more valid, accurate, and informative inferences about graded constructs such as language proficiency. Data and code are publicly available in an OSF repository at

Review Article
Copyright © The Author(s), 2021. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)


Aust, F, & Barth, M (2020). papaja: Create APA manuscripts with R Markdown. R package version Retrieved from Scholar
Birdsong, D, Gertken, LM, & Amengual, M (2012). Bilingual Language Profile: An easy-to-use instrument to assess bilingualism. [Measurement instrument]. Retrieved from Scholar
Box, GEP (1976). Science and statistics. Journal of the American Statistical Association 71(356), 791799. Scholar
Bürkner, P.-C (2017). brms: An R package for Bayesian multilevel models using Stan. Journal of Statistical Software 80(1), 128. Scholar
Bürkner, P.-C (2018). Advanced Bayesian multilevel modeling with the R package brms. The R Journal 10(1), 395411. Scholar
Bürkner, P.-C, & Vuorre, M (2019). Ordinal regression models in psychology: A tutorial. Advances in Methods and Practices in Psychological Science 2(1), 77101. Scholar
Cho, J, & Slabakova, R (2014). Interpreting definiteness in a second language without articles: The case of L2 Russian. Second Language Research 30(2), 159190. Scholar
Christensen, RHB (2019). ordinal - Regression models for ordinal data. R package version 2019.12-10. Retrieved from Scholar
Cohen, J (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
Dawes, J (2008). Do data characteristics change according to the number of scale points used? An experiment using 5-point, 7-point and 10-point scales. International Journal of Market Research 50(1), 61104. Scholar
DeCastellarnau, A (2018). A classification of response scale characteristics that affect data quality: A literature review. Quality & Quantity 52(4), 15231559. ScholarPubMed
Douven, I (2018). A Bayesian perspective on Likert scales and central tendency. Psychonomic Bulletin & Review 25(3), 12031211. ScholarPubMed
Farhy, Y, & Veríssimo, J (2019). Semantic effects in morphological priming: The case of Hebrew stems. Language and Speech 62(4), 737750. ScholarPubMed
Flege, JE, Yeni-Komshian, GH, & Liu, S (1999). Age constraints on second-language acquisition. Journal of Memory and Language 41(1), 78104. Scholar
Glass, GV, McGaw, B, & Smith, ML (1981). Meta-analysis in social research. Beverly Hills: Sage Publications.Google Scholar
Hakuta, K, Bialystok, E, & Wiley, E (2003). Critical evidence: A test of the critical-period hypothesis for second-language acquisition. Psychological Science 14(1), 3138. ScholarPubMed
Hopp, H (2009). The syntax–discourse interface in near-native L2 acquisition: Off-line and on-line performance. Bilingualism: Language and Cognition 12(4), 463483. Scholar
Hulstijn, JH (2012). The construct of language proficiency in the study of bilingualism from a cognitive perspective. Bilingualism: Language and Cognition 15(2), 422433. Scholar
Kissling, EM (2018). Pronunciation instruction can improve L2 learners’ bottom-Up processing for listening. The Modern Language Journal 102(4), 653675. Scholar
Krosnick, JA, & Presser, S (2010). Question and questionnaire design. In Marsden, P. V. & D., James (Eds.), Handbook of survey research (Second edition). Bingley, UK: Emerald.Google Scholar
Kuncel, RB (1977). The subject-item Interaction in itemmetric research. Educational and Psychological Measurement 37(3), 665678. Scholar
Lemhöfer, K, & Broersma, M (2012). Introducing LexTALE: A quick and valid Lexical Test for Advanced Learners of English. Behavior Research Methods 44(2), 325343. ScholarPubMed
Li, P, Sepanski, S, & Zhao, X (2006). Language History Questionnaire: A Web-based interface for bilingual research. Behavior Research Methods 38(2), 202210. ScholarPubMed
Liddell, TM, & Kruschke, JK (2018). Analyzing ordinal data with metric models: What could possibly go wrong? Journal of Experimental Social Psychology 79, 328348. Scholar
Marian, V, Blumenfeld, HK, & Kaushanskaya, M (2007). The Language Experience and Proficiency Questionnaire (LEAP-Q): Assessing language profiles in bilinguals and multilinguals. Journal of Speech, Language, and Hearing Research 50(4), 940967. ScholarPubMed
Matuschek, H, Kliegl, R, Vasishth, S, Baayen, RH, & Bates, D (2017). Balancing Type I error and power in linear mixed models. Journal of Memory and Language 94, 305315. Scholar
McCullagh, P (1980). Regression models for ordinal data. Journal of the Royal Statistical Society: Series B (Methodological) 42(2), 109127. Scholar
McElreath, R (2020). Statistical rethinking: A Bayesian course with examples in R and Stan (2nd ed.). CRC Press.CrossRefGoogle Scholar
McKelvey, RD, & Zavoina, W (1975). A statistical model for the analysis of ordinal level dependent variables. The Journal of Mathematical Sociology 4(1), 103120. Scholar
Puebla, C (2016). L2 proficiency survey. Unpublished raw data, Potsdam Research Institute for Multilingualism, University of Potsdam.Google Scholar
R Core Team. (2020). R: A language and environment for statistical computing. Vienna, Austria. Retrieved from Scholar
Schad, DJ, Betancourt, M, & Vasishth, S (2020). Toward a principled Bayesian workflow in cognitive science. Manuscript submitted for publication. Retrieved from Scholar
Schad, DJ, Vasishth, S, Hohenstein, S, & Kliegl, R (2020). How to capitalize on a priori contrasts in linear (mixed) models: A tutorial. Journal of Memory and Language 110, 104038. Scholar
Schlenter, J (2019). Predictive language processing in late bilinguals (Doctoral dissertation). Universität Potsdam. Scholar
Schwarz, N (1999). Self-reports: How the questions shape the answers. American Psychologist 54(2), 93105. Scholar
Tare, M, Golonka, E, Lancaster, AK, Bonilla, C, Doughty, CJ, Belnap, RK, & Jackson, SR (2018). The role of cognitive aptitudes in a study abroad language-learning environment. In Sanz, C & Morales-Front, A (Eds.), The Routledge Handbook of Study Abroad Research and Practice (1st ed.). Routledge, pp. 406420 Scholar
Tomoschuk, B, Ferreira, VS, & Gollan, TH (2019). When a seven is not a seven: Self-ratings of bilingual language proficiency differ between and within language populations. Bilingualism: Language and Cognition 22(3), 516536. Scholar
Vasishth, S, Nicenboim, B, Beckman, ME, Li, F, & Kong, EJ (2018). Bayesian data analysis in the phonetic sciences: A tutorial introduction. Journal of Phonetics 71, 147161. ScholarPubMed
Zell, E, & Krizan, Z (2014). Do people have insight into their abilities? A metasynthesis. Perspectives on Psychological Science 9(2), 111125. ScholarPubMed
Supplementary material: PDF

Veríssimo supplementary material

Veríssimo supplementary material

Download Veríssimo supplementary material(PDF)