Skip to main content

VALIDATING GRAMMATICALITY JUDGMENT TESTS: Evidence from Two New Psycholinguistic Measures

  • Payman Vafaee (a1), Yuichi Suzuki (a2) and Ilina Kachisnke (a3)
  • Please note a correction has been issued for this article.

Several previous factor-analytic studies on the construct validity of grammaticality judgment tests (GJTs) concluded that untimed GJTs measure explicit knowledge (EK) and timed GJTs measure implicit knowledge (IK) (Bowles, 2011; R. Ellis, 2005; R. Ellis & Loewen, 2007). It has also been shown that, irrespective of the time condition chosen, GJTs’ grammatical sentences tap into IK, whereas their ungrammatical ones invoke EK (Gutiérrez, 2013). The current study examined these conclusions by employing two more fine-grained measures of IK: that is, a self-paced reading task and a word-monitoring task. The results of a confirmatory factor analysis revealed that manipulating GJTs’ time conditions and/or the grammaticality of the sentences does not render them distinct measures of EK and IK. The current work shows that GJTs are too coarse to be measures of IK, and that the different types of GJTs measure different levels of EK.

Corresponding author
*Correspondence concerning this article should be addressed to Payman Vafaee, 2116-D Susquehanna Hall, University of Maryland, College Park, MD 20742, USA. E-mail:
Hide All
We would like to thank professors Steve Ross, Mike Long, and Robert DeKeyser and the SSLA editors and anonymous reviewers for their thoughtful suggestions and constructive feedback. We are also grateful to the participants of our study. All errors and omissions are, of course, our own.
* Order of authors has been corrected since original publication. An erratum notice detailing this change was also published (DOI 10.1017/S0272263116000097).
Hide All
Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Mahwah, NJ: Lawrence Erlbaum.
Andringa, S., & Curcic, M. (2015). How explicit knowledge affects online L2 processing [Special issue]. Studies in Second Language Acquisition, 37(2), 237268. doi:10.1017/S0272263115000017
Bialystok, E. (1979). Explicit and implicit judgements of L2 grammaticality. Language Learning, 29(1), 81103.
Bialystok, E. (1994). Analysis and control in the development of second language proficiency. Studies in second language acquisition, 16(2), 157168.
Bley-Vroman, R. (1991). The logical problem of foreign language learning. Linguistic Analysis, 20(1–2), 349.
Bowles, M. A. (2011). Measuring implicit and explicit linguistic knowledge. Studies in Second Language Acquisition, 33(2), 247271.
Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York: Guilford Press.
Chraudron, C. (1983). Research on metalinguistic judgments: A review of theory, methods, and results. Language learning, 33(3), 343377.
Clahsen, H., & Felser, C. (2006). Grammatical processing in language learners. Applied Psycholinguistics, 27(1), 342.
DeKeyser, R. M. (1997). Beyond explicit rule learning. Studies in second language acquisition, 19(2), 195221.
DeKeyser, R. M. (2003). Implicit and explicit learning. In Doughty, C. J. & Long, M. H. (Eds.), The Handbook of Second Language Acquisition (pp. 312348). Oxford, UK: Blackwell.
DeKeyser, R. M. (2007). Skill acquisition theory. In Williams, B. V. J. (Ed.), Theories in second language acquisition (pp. 97114). Mahwah, NJ: Erlbaum.
DeKeyser, R. M. (2009). Cognitive-psychological processes in second language learning. In Long, M. H. & Doughty, C. J. (Eds.), The handbook of language teaching (pp. 119138). Oxford, UK: Wiley-Blackwell.
DeKeyser, R., & Criado, R. (2012). Automatization, skill acquisition, and practice in second language acquisition. In Chapelle, C. (Ed.), The encyclopedia of applied linguistics (pp. 45014504). Oxford, UK: Wiley-Blackwell.
Ellis, N. C. (1993). Rules and instances in foreign language learning: Interactions of explicit and implicit knowledge. European Journal of Cognitive Psychology, 5(3), 289318.
Ellis, N. C. (Ed.). (1994). Implicit and explicit learning of languages. London: Academic Press.
Ellis, N. C. (2005). At the interface: Dynamic interactions of explicit and implicit language knowledge. Studies in Second Language Acquisition, 27(2), 305352.
Ellis, N. C. (2008). The dynamics of second language emergence: Cycles of language use, language change, and language acquisition. The Modern Language Journal, 92(2), 232249.
Ellis, N. C. (2011). Implicit and explicit SLA and their interface. Implicit and Explicit Language Learning: Conditions, Processes, and Knowledge in SLA and Bilingualism, 3547.
Ellis, R. (1991). Grammaticality judgments and learner variability. In Burmeister, H. & Rounds, P. (Eds.), Variability in second language acquisition: Proceedings of the tenth meeting of the Second Language Acquisition Forum (pp. 2560). Eugene: Department of Linguistics, University of Oregon.
Ellis, R. (2004). The definition and measurement of L2 explicit knowledge. Language Learning, 54(2), 227275.
Ellis, R. (2005). Measuring implicit and explicit knowledge of a second language. Studies in Second Language Acquisition, 27(2), 141172.
Ellis, R. (2009). Implicit and explicit knowledge in second language learning, testing and teaching (Vol. 42). Bristol, UK: Multilingual Matters.
Ellis, R., & Loewen, S. (2007). Confirming the operational definitions of explicit and implicit knowledge in Ellis (2005): Responding to Isemonger. Studies in Second Language Acquisition, 29(1), 119126. doi:10.1017/S0272263107070052
Forster, K. I., & Forster, J. C. (2003). DMDX: A Windows display program with millisecond accuracy. Behavior Research Methods, Instruments, & Computers, 35(1), 116124.
Granena, G. (2013). Individual differences in sequence learning ability and second language acquisition in early childhood and adulthood. Language Learning, 63(4), 665703.
Gutiérrez, X. (2013). The construct validity of grammaticality judgment tests as measures of implicit and explicit knowledge. Studies in Second Language Acquisition, 35(3), 423449. doi:10.1017/S0272263113000041
Henning, G. (1987). A guide to language testing: Development, evaluation, research. Cambridge, MA: Newbury House.
Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural equation modeling: a multidisciplinary journal, 6(1), 155.
Hulstijn, J. H. (2001). Intentional and incidental second language vocabulary learning: A reappraisal of elaboration, rehearsal and automaticity. In Robinson, P. (Ed.), Cognition and second language instruction (pp. 258286). Cambridge, UK: Cambridge University Press.
Hulstijn, J. H. (2002). Towards a unified account of the representation, processing and acquisition of second language knowledge. Second Language Research, 18(3), 193223. doi:10.1191/0267658302sr207oa
Hulstijn, J. H. (2005). Theoretical and empirical issues in the study of implicit and explicit second-language learning. Studies in Second Language Acquisition, 27(2), 129140.
Isemonger, I. M. (2007). Operational definitions of explicit and implicit knowledge: Response to R. Ellis (2005) and some recommendations for future research in this area. Studies in Second Language Acquisition, 29(1), 101118.
Jiang, N. (2004). Morphological insensitivity in second language processing. Applied Psycholinguistics, 25(4), 603634.
Jiang, N. (2007). Selective integration of linguistic knowledge in adult second language learning. Language Learning, 57(1), 133.
Jiang, N., Hu, G., Lukyanchenko, A., & Cao, Y. (2010). Insensitivity to morphological errors in L2: Evidence from word monitoring. Paper presented at the SLRF 2010, October 14–17, 2010, College Park, MD.
Jiang, N., Novokshanova, E., Masuda, K., & Wang, X. (2011). Morphological congruency and the acquisition of L2 morphemes. Language Learning, 61(3), 940967.
Jöreskog, K. G. (1999). How large can a standardized coefficient be? Unpublished report. SSI Central, Inc. Retrieved from
Jöreskog, K. G., & Sorbom, D. (2012). LISREL 9.1 [Computer software]. Lincolnwood, IL: Scientific Software International.
Kaan, E. (2014). Predictive sentence processing in L2 and L1: What is different? Linguistic Approaches to Bilingualism, 4(2), 257282.
Kachinske, I., & Vafaee, P. (2014). Reexamining the validity of GJTs as measures of implicit knowledge: Reanalysis of Ellis & Loewen (2007), Bowles (2011), and Gutierrez (2013). Paper presented at the Second Language Research Forum (SLRF), University of South Carolina, Columbia, South Carolina.
Krashen, S. (1994). The input hypothesis and its rivals. In Ellis, N. (Ed.), Implicit and explicit learning of language (pp. 4577). London: Academic Press.
Leung, J. H. C., & Williams, J. N. (2011). The implicit learning of mappings between forms and contextually derived meanings. Studies in Second Language Acquisition, 33(1), 3355.
Leung, J. H. C., & Williams, J. N. (2014). Crosslinguistic differences in implicit language learning. Studies in Second Language Acquisition, 36(4), 733755.
MacCallum, R. C., Widaman, K. F., Preacher, K. J., & Hong, S. (2001). Sample size in factor analysis: The role of model error. Multivariate Behavioral Research, 36(4), 611637.
MacCallum, R. C., Widaman, K. F., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4(1), 8499.
Mueller, R. O., & Hancock, G. R. (2008). Best practices in structural equation modeling. In Osborne, J. (Ed.), Best practices in quantitative methods (pp. 488508). Thousand Oaks, CA: Sage.
Nunnally, J. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill.
Paciorek, A., & Williams, J. N. (2015). Implicit learning of semantic preferences of verbs [Special issue]. Studies in Second Language Acquisition, 37(2), 359382. doi:10.1017/S0272263115000108
Paradis, M. (1994). Neurolinguistic aspects of implicit and explicit memory. Implications for bilingualism. In Ellis, N. (Ed.), Implicit and explicit learning of languages (pp. 393419). London: Academic Press.
Paradis, M. (2009). Declarative and procedural determinants of second languages. Philadelphia, PA: John Benjamins.
Roberts, L., & Liszka, S. A. (2013). Processing tense/aspect-agreement violations on-line in the second language: A self-paced reading study with French and German L2 learners of English. Second Language Research, 29(4), 413439. doi:10.1177/0267658313503171
Segalowitz, N., & Hulstijn, J. (2005). Automaticity in bilingualism and second language learning. In Kroll, J. F. & de Groot, A. M. B. (Eds.), Handbook of bilingualism: Psycholinguistic approaches (pp. 371388). New York: Oxford University Press.
Suzuki, Y. (2015). Using new measures of implicit L2 knowledge to study the interface of explicit and implicit knowledge (Unpublished doctoral dissertation). University of Maryland, College Park.
Suzuki, Y., & DeKeyser, R. M. (2015). Comparing elicited imitation and word monitoring as measures of implicit knowledge. Language Learning, 65(4), 860895.
Ullman, M. T. (2001). A neurocognitive perspective on language: The declarative/procedural model. Nature Reviews Neuroscience, 2(10), 717726.
Velicer, W. F., & Fava, J. L. (1998). Affects of variable and subject sampling on factor pattern recovery. Psychological Methods, 3(2), 231.
Williams, J. N. (2005). Learning without awareness. Studies in Second Language Acquisition, 27(2), 269304. doi:10.1017/S0272263105050138
Williams, J. N. (2009). Implicit learning. In Ritchie, W. C. & Bhatia, T. K. (Eds.), New handbook of second language acquisition (pp. 319353). Bingley, UK: Emerald Group.
Zhang, R. (2015). Measuring university-level L2 learners’ implicit and explicit linguistic knowledge. Studies in Second Language Acquisition, 37, 457486.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Studies in Second Language Acquisition
  • ISSN: 0272-2631
  • EISSN: 1470-1545
  • URL: /core/journals/studies-in-second-language-acquisition
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Altmetric attention score

Full text views

Total number of HTML views: 26
Total number of PDF views: 589 *
Loading metrics...

Abstract views

Total abstract views: 1138 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 26th May 2018. This data will be updated every 24 hours.