Hostname: page-component-89b8bd64d-n8gtw Total loading time: 0 Render date: 2026-05-07T08:58:29.563Z Has data issue: false hasContentIssue false
  • English
  • Français

VALIDATING GRAMMATICALITY JUDGMENT TESTS

Evidence from Two New Psycholinguistic Measures

Published online by Cambridge University Press:  08 January 2016

Payman Vafaee ,
Yuichi Suzuki  and
Ilina Kachisnke
Payman Vafaee*
Affiliation:
University of Maryland
Yuichi Suzuki
Affiliation:
Kanagawa University
Ilina Kachisnke
Affiliation:
University of Maryland
*
*Correspondence concerning this article should be addressed to Payman Vafaee, 2116-D Susquehanna Hall, University of Maryland, College Park, MD 20742, USA. E-mail: payman.vafaee@gmail.com
Get access Rights & Permissions [Opens in a new window]

Abstract

Several previous factor-analytic studies on the construct validity of grammaticality judgment tests (GJTs) concluded that untimed GJTs measure explicit knowledge (EK) and timed GJTs measure implicit knowledge (IK) (Bowles, 2011; R. Ellis, 2005; R. Ellis & Loewen, 2007). It has also been shown that, irrespective of the time condition chosen, GJTs’ grammatical sentences tap into IK, whereas their ungrammatical ones invoke EK (Gutiérrez, 2013). The current study examined these conclusions by employing two more fine-grained measures of IK: that is, a self-paced reading task and a word-monitoring task. The results of a confirmatory factor analysis revealed that manipulating GJTs’ time conditions and/or the grammaticality of the sentences does not render them distinct measures of EK and IK. The current work shows that GJTs are too coarse to be measures of IK, and that the different types of GJTs measure different levels of EK.

Information

Type
Articles
Information
Studies in Second Language Acquisition , Volume 39 , Issue 1 , March 2017 , pp. 59 - 95
Check for updates
Copyright
Copyright © Cambridge University Press 2016 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

We would like to thank professors Steve Ross, Mike Long, and Robert DeKeyser and the SSLA editors and anonymous reviewers for their thoughtful suggestions and constructive feedback. We are also grateful to the participants of our study. All errors and omissions are, of course, our own.
* Order of authors has been corrected since original publication. An erratum notice detailing this change was also published (DOI 10.1017/S0272263116000097).

References

REFERENCES

Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Mahwah, NJ: Lawrence Erlbaum.Google Scholar
Andringa, S., & Curcic, M. (2015). How explicit knowledge affects online L2 processing [Special issue]. Studies in Second Language Acquisition, 37(2), 237–268. doi:10.1017/S0272263115000017 Google Scholar
Bialystok, E. (1979). Explicit and implicit judgements of L2 grammaticality. Language Learning, 29(1), 81–103.Google Scholar
Bialystok, E. (1994). Analysis and control in the development of second language proficiency. Studies in second language acquisition, 16(2), 157–168.Google Scholar
Bley-Vroman, R. (1991). The logical problem of foreign language learning. Linguistic Analysis, 20(1–2), 3–49.Google Scholar
Bowles, M. A. (2011). Measuring implicit and explicit linguistic knowledge. Studies in Second Language Acquisition, 33(2), 247–271.CrossRefGoogle Scholar
Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York: Guilford Press.Google Scholar
Chraudron, C. (1983). Research on metalinguistic judgments: A review of theory, methods, and results. Language learning, 33(3), 343–377.Google Scholar
Clahsen, H., & Felser, C. (2006). Grammatical processing in language learners. Applied Psycholinguistics, 27(1), 3–42.CrossRefGoogle Scholar
DeKeyser, R. M. (1997). Beyond explicit rule learning. Studies in second language acquisition, 19(2), 195–221.Google Scholar
DeKeyser, R. M. (2003). Implicit and explicit learning. In Doughty, C. J. & Long, M. H. (Eds.), The Handbook of Second Language Acquisition (pp. 312–348). Oxford, UK: Blackwell.Google Scholar
DeKeyser, R. M. (2007). Skill acquisition theory. In Williams, B. V. J. (Ed.), Theories in second language acquisition (pp. 97–114). Mahwah, NJ: Erlbaum.Google Scholar
DeKeyser, R. M. (2009). Cognitive-psychological processes in second language learning. In Long, M. H. & Doughty, C. J. (Eds.), The handbook of language teaching (pp. 119–138). Oxford, UK: Wiley-Blackwell.Google Scholar
DeKeyser, R., & Criado, R. (2012). Automatization, skill acquisition, and practice in second language acquisition. In Chapelle, C. (Ed.), The encyclopedia of applied linguistics (pp. 4501–4504). Oxford, UK: Wiley-Blackwell.Google Scholar
Ellis, N. C. (1993). Rules and instances in foreign language learning: Interactions of explicit and implicit knowledge. European Journal of Cognitive Psychology, 5(3), 289–318.Google Scholar
Ellis, N. C. (Ed.). (1994). Implicit and explicit learning of languages. London: Academic Press.Google Scholar
Ellis, N. C. (2005). At the interface: Dynamic interactions of explicit and implicit language knowledge. Studies in Second Language Acquisition, 27(2), 305–352.Google Scholar
Ellis, N. C. (2008). The dynamics of second language emergence: Cycles of language use, language change, and language acquisition. The Modern Language Journal, 92(2), 232–249.Google Scholar
Ellis, N. C. (2011). Implicit and explicit SLA and their interface. Implicit and Explicit Language Learning: Conditions, Processes, and Knowledge in SLA and Bilingualism, 35–47.Google Scholar
Ellis, R. (1991). Grammaticality judgments and learner variability. In Burmeister, H. & Rounds, P. (Eds.), Variability in second language acquisition: Proceedings of the tenth meeting of the Second Language Acquisition Forum (pp. 25–60). Eugene: Department of Linguistics, University of Oregon.Google Scholar
Ellis, R. (2004). The definition and measurement of L2 explicit knowledge. Language Learning, 54(2), 227–275.Google Scholar
Ellis, R. (2005). Measuring implicit and explicit knowledge of a second language. Studies in Second Language Acquisition, 27(2), 141–172.Google Scholar
Ellis, R. (2009). Implicit and explicit knowledge in second language learning, testing and teaching (Vol. 42). Bristol, UK: Multilingual Matters.Google Scholar
Ellis, R., & Loewen, S. (2007). Confirming the operational definitions of explicit and implicit knowledge in Ellis (2005): Responding to Isemonger. Studies in Second Language Acquisition, 29(1), 119–126. doi:10.1017/S0272263107070052 Google Scholar
Forster, K. I., & Forster, J. C. (2003). DMDX: A Windows display program with millisecond accuracy. Behavior Research Methods, Instruments, & Computers, 35(1), 116–124.Google Scholar
Granena, G. (2013). Individual differences in sequence learning ability and second language acquisition in early childhood and adulthood. Language Learning, 63(4), 665–703.Google Scholar
Gutiérrez, X. (2013). The construct validity of grammaticality judgment tests as measures of implicit and explicit knowledge. Studies in Second Language Acquisition, 35(3), 423–449. doi:10.1017/S0272263113000041 Google Scholar
Henning, G. (1987). A guide to language testing: Development, evaluation, research. Cambridge, MA: Newbury House.Google Scholar
Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural equation modeling: a multidisciplinary journal, 6(1), 1–55.Google Scholar
Hulstijn, J. H. (2001). Intentional and incidental second language vocabulary learning: A reappraisal of elaboration, rehearsal and automaticity. In Robinson, P. (Ed.), Cognition and second language instruction (pp. 258–286). Cambridge, UK: Cambridge University Press.Google Scholar
Hulstijn, J. H. (2002). Towards a unified account of the representation, processing and acquisition of second language knowledge. Second Language Research, 18(3), 193–223. doi:10.1191/0267658302sr207oa Google Scholar
Hulstijn, J. H. (2005). Theoretical and empirical issues in the study of implicit and explicit second-language learning. Studies in Second Language Acquisition, 27(2), 129–140.Google Scholar
Isemonger, I. M. (2007). Operational definitions of explicit and implicit knowledge: Response to R. Ellis (2005) and some recommendations for future research in this area. Studies in Second Language Acquisition, 29(1), 101–118.Google Scholar
Jiang, N. (2004). Morphological insensitivity in second language processing. Applied Psycholinguistics, 25(4), 603–634.Google Scholar
Jiang, N. (2007). Selective integration of linguistic knowledge in adult second language learning. Language Learning, 57(1), 1–33.Google Scholar
Jiang, N., Hu, G., Lukyanchenko, A., & Cao, Y. (2010). Insensitivity to morphological errors in L2: Evidence from word monitoring. Paper presented at the SLRF 2010, October 14–17, 2010, College Park, MD.Google Scholar
Jiang, N., Novokshanova, E., Masuda, K., & Wang, X. (2011). Morphological congruency and the acquisition of L2 morphemes. Language Learning, 61(3), 940–967.Google Scholar
Jöreskog, K. G. (1999). How large can a standardized coefficient be? Unpublished report. SSI Central, Inc. Retrieved from http://www.ssicentral.com/lisrel/techdocs/HowLargeCanaStandardizedCoefficientbe.pdf Google Scholar
Jöreskog, K. G., & Sorbom, D. (2012). LISREL 9.1 [Computer software]. Lincolnwood, IL: Scientific Software International. Google Scholar
Kaan, E. (2014). Predictive sentence processing in L2 and L1: What is different? Linguistic Approaches to Bilingualism, 4(2), 257–282.CrossRefGoogle Scholar
Kachinske, I., & Vafaee, P. (2014). Reexamining the validity of GJTs as measures of implicit knowledge: Reanalysis of Ellis & Loewen (2007), Bowles (2011), and Gutierrez (2013). Paper presented at the Second Language Research Forum (SLRF), University of South Carolina, Columbia, South Carolina.Google Scholar
Krashen, S. (1994). The input hypothesis and its rivals. In Ellis, N. (Ed.), Implicit and explicit learning of language (pp. 45–77). London: Academic Press.Google Scholar
Leung, J. H. C., & Williams, J. N. (2011). The implicit learning of mappings between forms and contextually derived meanings. Studies in Second Language Acquisition, 33(1), 33–55.Google Scholar
Leung, J. H. C., & Williams, J. N. (2014). Crosslinguistic differences in implicit language learning. Studies in Second Language Acquisition, 36(4), 733–755.Google Scholar
MacCallum, R. C., Widaman, K. F., Preacher, K. J., & Hong, S. (2001). Sample size in factor analysis: The role of model error. Multivariate Behavioral Research, 36(4), 611–637.Google Scholar
MacCallum, R. C., Widaman, K. F., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4(1), 84–99.Google Scholar
Mueller, R. O., & Hancock, G. R. (2008). Best practices in structural equation modeling. In Osborne, J. (Ed.), Best practices in quantitative methods (pp. 488–508). Thousand Oaks, CA: Sage.Google Scholar
Nunnally, J. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill.Google Scholar
Paciorek, A., & Williams, J. N. (2015). Implicit learning of semantic preferences of verbs [Special issue]. Studies in Second Language Acquisition, 37(2), 359–382. doi:10.1017/S0272263115000108 Google Scholar
Paradis, M. (1994). Neurolinguistic aspects of implicit and explicit memory. Implications for bilingualism. In Ellis, N. (Ed.), Implicit and explicit learning of languages (pp. 393–419). London: Academic Press.Google Scholar
Paradis, M. (2009). Declarative and procedural determinants of second languages. Philadelphia, PA: John Benjamins.Google Scholar
Roberts, L., & Liszka, S. A. (2013). Processing tense/aspect-agreement violations on-line in the second language: A self-paced reading study with French and German L2 learners of English. Second Language Research, 29(4), 413–439. doi:10.1177/0267658313503171 Google Scholar
Segalowitz, N., & Hulstijn, J. (2005). Automaticity in bilingualism and second language learning. In Kroll, J. F. & de Groot, A. M. B. (Eds.), Handbook of bilingualism: Psycholinguistic approaches (pp. 371–388). New York: Oxford University Press.Google Scholar
Suzuki, Y. (2015). Using new measures of implicit L2 knowledge to study the interface of explicit and implicit knowledge (Unpublished doctoral dissertation). University of Maryland, College Park.Google Scholar
Suzuki, Y., & DeKeyser, R. M. (2015). Comparing elicited imitation and word monitoring as measures of implicit knowledge. Language Learning, 65(4), 860–895.Google Scholar
Ullman, M. T. (2001). A neurocognitive perspective on language: The declarative/procedural model. Nature Reviews Neuroscience, 2(10), 717–726.Google Scholar
Velicer, W. F., & Fava, J. L. (1998). Affects of variable and subject sampling on factor pattern recovery. Psychological Methods, 3(2), 231.Google Scholar
Williams, J. N. (2005). Learning without awareness. Studies in Second Language Acquisition, 27(2), 269–304. doi:10.1017/S0272263105050138 Google Scholar
Williams, J. N. (2009). Implicit learning. In Ritchie, W. C. & Bhatia, T. K. (Eds.), New handbook of second language acquisition (pp. 319–353). Bingley, UK: Emerald Group.Google Scholar
Zhang, R. (2015). Measuring university-level L2 learners’ implicit and explicit linguistic knowledge. Studies in Second Language Acquisition, 37, 457–486.Google Scholar