Hostname: page-component-76c49bb84f-l6pkv Total loading time: 0 Render date: 2025-07-04T12:21:29.521Z Has data issue: false hasContentIssue false
  • English
  • Français

THE RELIABILITY AND VALIDITY OF PROCEDURAL MEMORY ASSESSMENTS USED IN SECOND LANGUAGE ACQUISITION RESEARCH

Published online by Cambridge University Press:  08 July 2021

Joshua Buffington Open the ORCID record for Joshua Buffington [Opens in a new window] ,
Alexander P. Demos Open the ORCID record for Alexander P. Demos [Opens in a new window]  and
Kara Morgan-Short Open the ORCID record for Kara Morgan-Short [Opens in a new window]
Joshua Buffington*
Affiliation:
University of Illinois at Chicago
Alexander P. Demos
Affiliation:
University of Illinois at Chicago
Kara Morgan-Short*
Affiliation:
University of Illinois at Chicago
*
*Correspondence concerning this article should be addressed to Joshua Buffington, Department of Psychology (m/c 285), University of Illinois at Chicago, 1007 W. Harrison St., Chicago, Illinois60607-7137. E-mail: bffngtn2@uic.edu; or Kara Morgan-Short, Department of Hispanic and Italian Studies (m/c 315), University of Illinois at Chicago, 601 S. Morgan St., Chicago, Illinois 60607. E-mail: karams@uic.edu
*Correspondence concerning this article should be addressed to Joshua Buffington, Department of Psychology (m/c 285), University of Illinois at Chicago, 1007 W. Harrison St., Chicago, Illinois60607-7137. E-mail: bffngtn2@uic.edu; or Kara Morgan-Short, Department of Hispanic and Italian Studies (m/c 315), University of Illinois at Chicago, 601 S. Morgan St., Chicago, Illinois 60607. E-mail: karams@uic.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

Evidence for the role of procedural memory in second language (L2) acquisition has emerged in our field. However, little is known about the reliability and validity of the procedural memory measures used in this research. The present study (N = 119) examined the reliability and the convergent and discriminant validity of three assessments that have previously been used to examine procedural memory learning ability in L2 acquisition, the dual-task Weather Prediction Task (DT-WPT), the Alternating Serial Reaction Time Task (ASRT), and the Tower of London (TOL). Measures of declarative memory learning ability were also collected. For reliability, the DT-WPT and TOL tasks met acceptable standards. For validity, an exploratory factor analysis did not provide evidence for convergent validity, but the ASRT and the TOL showed reasonable discriminant validity with declarative memory measures. We argue that the ASRT may provide the purest engagement of procedural memory learning ability, although more reliable dependent measures for this task should be considered. The Serial Reaction Time task also appears promising, although we recommend further consideration of this task as the present analyses were post hoc and based on a smaller sample. We discuss these results regarding the assessment of procedural memory learning ability as well as implications for implicit language aptitude.

Information

Type
Methods Forum
Information
Studies in Second Language Acquisition , Volume 43 , Special Issue 3: Implicit Language Aptitude: Conceptualizing the Construct, Validating the Measures, and Examining the Evidence , July 2021 , pp. 635 - 662
Check for updates
Open Practices
Open data
Copyright
© The Author(s), 2021. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

The research reported here came out of Joshua Buffington’s MA thesis. We would like to acknowledge the following people for their feedback on this work: Susan R. Goldman, James W. Pellegrino, and Members of the Cognition of Second Language Acquisition Laboratory. We are also grateful for the UIC Award for Graduate Research for funding part of this study.

The experiment in this article earned an Open Data badge for transparent practices. The materials are available at https://osf.io/ux4qs/.

References

REFERENCES

Antoniou, M., Ettlinger, M., & Wong, P. C. M. (2016). Complexity, training paradigm design, and the contribution of memory subsystems to grammar learning. PloS One, 11, Article e0158812. https://doi.org/10.1371/journal.pone.0158812 CrossRefGoogle ScholarPubMed
Batterink, L. J., Paller, K. A., & Reber, P. J. (2019). Understanding the neural bases of implicit and statistical learning. Topics in Cognitive Science, 11, 482–503. https://doi.org/10.1111/tops.12420 Google ScholarPubMed
Beauchamp, M. H., Dagher, A., Aston, J., & Doyon, J. (2003). Dynamic functional changes associated with cognitive skill learning of an adapted version of the Tower of London task. NeuroImage, 20, 1649–1660.CrossRefGoogle ScholarPubMed
Beauchamp, M. H., Dagher, A., Panisset, M., & Doyon, J. (2008). Neural substrates of cognitive skill learning in Parkinson’s disease. Brain and Cognition, 68, 134–143.CrossRefGoogle ScholarPubMed
Bollen, K. A. (2002). Latent variables in psychology and the social sciences. Annual Review of Psychology, 53, 605–634. https://doi.org/10.1146/annurev.psych.53.100901.135239 CrossRefGoogle ScholarPubMed
Brill-Schuetz, K. & Morgan-Short, K. (2014). The role of procedural memory in adult second language acquisition. Proceedings of the Annual Meeting of the Cognitive Science Society, 36, 260–265. https://escholarship.org/content/qt0dc7958r/qt0dc7958r.pdf Google Scholar
Buffington, J., & Morgan-Short, K. (2018). Construct validity of procedural memory tasks used in adult-learned language. Proceedings of the Annual Conference of the Cognitive Science Society, 40, 1422–1427. https://cogsci.mindmodeling.org/2018/papers/0276/0276.pdf Google Scholar
Buffington, J., & Morgan-Short, K. (2019). Declarative and procedural memory as individual differences in second language aptitude. In Wen, Z. E., Skehan, P., Biedroń, A., Li, S. & Sparks, R. L. (Eds.), Language aptitude: Advancing theory, testing, research and practice (pp. 215–237). Routledge.CrossRefGoogle Scholar
Carpenter, H., Morgan-Short, K., & Ullman, M. T. (2009). Predicting L2 using declarative and procedural memory assessments: A behavioral and ERP investigation. Presented at the Georgetown University Round Table, Washington, DC.Google Scholar
Carpenter, H. S. (2008). A behavioral and electrophysiological investigation of different aptitudes for L2 grammar in learners equated for proficiency level [Unpublished doctoral dissertation]. Georgetown University.Google Scholar
Carroll, J. B., & Sapon, S. M. (1959). Modern language aptitude test. Psychological Corporation.Google Scholar
Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155–159.CrossRefGoogle ScholarPubMed
Csabi, E., Benedek, P., Janacsek, K., Zavecz, Z., Katona, G., & Nemeth, D. (2016). Declarative and non-declarative memory consolidation in children with sleep disorder. Frontiers in Human Neuroscience, 9, Article 709. https://doi.org/10.3389/fnhum.2015.00709 CrossRefGoogle ScholarPubMed
Dagher, A., Owen, A. M., Boecker, H., & Brooks, D. J. (2001). The role of the striatum and hippocampus in planning: A PET activation study in Parkinson’s disease. Brain, 124, 1020–1032.CrossRefGoogle ScholarPubMed
DeKeyser, R. M. (2020). Skill acquisition theory. In VanPatten, B., Keating, G. D., & Wulff, S. (Eds.), Theories in second language acquisition (3rd ed., pp. 83–104). Routledge.CrossRefGoogle Scholar
Eichenbaum, H. (2012). The cognitive neuroscience of memory: An introduction (2nd ed.). Oxford University Press.Google Scholar
Eichenbaum, H., Otto, T., & Cohen, N. J. (1994). Two functional components of the hippocampal memory system. Behavioral and Brain Sciences, 17, 449–472. https://doi.org/10.1017/S0140525X00035391 CrossRefGoogle Scholar
Ettlinger, M., Bradlow, A. R., & Wong, P. C. M. (2014). Variability in the learning of complex morphophonology. Applied Psycholinguistics, 35, 807–831. https://doi.org/10.1017/S0142716412000586 CrossRefGoogle Scholar
Faretta-Stutenberg, M., & Morgan-Short, K. (2018). The interplay of individual differences and context of learning in behavioral and neurocognitive second language development. Second Language Research, 34, 67–101. https://doi.org/10.1177/0267658316684903 CrossRefGoogle Scholar
Foerde, K., Knowlton, B. J., & Poldrack, R. A. (2006). Modulation of competing memory systems by distraction. Proceedings of the National Academy of Sciences of the United States of America, 103, 11778–11783. https://doi.org/10.1073/pnas.0602659103 CrossRefGoogle ScholarPubMed
Foerde, K., Poldrack, R. A., & Knowlton, B. J. (2007). Secondary-task effects on classification learning. Memory & Cognition, 35, 864–874. https://doi.org/10.3758/BF03193461 CrossRefGoogle ScholarPubMed
Gabrieli, J. D. E. (1998). Cognitive neuroscience of human memory. Annual Review of Psychology, 49, 87–115. https://doi.org/10.1146/annurev.psych.49.1.87 CrossRefGoogle ScholarPubMed
Gebauer, G. F., & Mackintosh, N. J. (2007). Psychometric intelligence dissociates implicit and explicit learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 33, 34–54. https://doi.org/10.1037/0278-7393.33.1.34 Google ScholarPubMed
Gluck, M. A., Shohamy, D., & Myers, C. (2002). How do people solve the “weather prediction” task? Individual variability in strategies for probabilistic category learning. Learning & Memory, 9, 408–418. https://doi.org/10.1101/lm.45202 CrossRefGoogle ScholarPubMed
Godfroid, A., & Kim, K. M. (2021). The contributions of implicit-statistical learning aptitude to implicit second-language knowledge. Studies in Second Language Acquisition. Advance online publication. https:/doi.org/10.1017/s0272263121000085 CrossRefGoogle Scholar
Granena, G. (2013). Individual differences in sequence learning ability and second language acquisition in early childhood and adulthood. Language Learning, 63, 665–703. https://doi.org/10.1111/lang.12018 CrossRefGoogle Scholar
Granena, G. (2020). Implicit language aptitude. Cambridge University Press. https://doi.org/10.1017/9781108625616CrossRefGoogle Scholar
Hamrick, P. (2015). Declarative and procedural memory abilities as individual differences in incidental language learning. Learning and Individual Differences, 44, 9–15. https://doi.org/10.1016/j.lindif.2015.10.003 CrossRefGoogle Scholar
Hamrick, P., Lum, J. A. G., & Ullman, M. T. (2018). Child first language and adult second language are both tied to general-purpose learning systems. Proceedings of the National Academy of Sciences of the United States of America, 115, 1487–1492. https://doi.org/10.1073/pnas.1713975115 CrossRefGoogle ScholarPubMed
Hedenius, M., Ullman, M. T., Alm, P., Jennische, M., & Persson, J. (2013). Enhanced recognition memory after incidental encoding in children with developmental dyslexia. PloS One, 8, Article e63998. https://doi.org/10.1371/journal.pone.0063998 CrossRefGoogle ScholarPubMed
Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179–185. https://doi.org/10.1007/BF02289447 CrossRefGoogle ScholarPubMed
Howard, J. H. Jr., & Howard, D. V. (1997). Age differences in implicit learning of higher order dependencies in serial patterns. Psychology and Aging, 12, 634–656. https://doi.org/10.1037/0882-7974.12.4.634 CrossRefGoogle ScholarPubMed
Howard, J. H. Jr., & Howard, D. V. (2013). Aging mind and brain: Is implicit learning spared in healthy aging? Frontiers in Psychology, 4, Article 817. https://doi.org/10.3389/fpsyg.2013.00817 CrossRefGoogle ScholarPubMed
Howard, J. H. Jr., Howard, D. V., Dennis, N. A., & Kelly, A. J. (2008). Implicit learning of predictive relationships in three-element visual sequences by young and old adults. Journal of Experimental Psychology: Learning, Memory, and Cognition, 34, 1139–1157. https://doi.org/10.1037/a0012797 Google ScholarPubMed
Howard, M. C. (2016). A review of exploratory factor analysis decisions and overview of current practices: What we are doing and how can we improve? International Journal of Human-Computer Interaction, 32, 51–62.CrossRefGoogle Scholar
Janacsek, K., Shattuck, K. F., Tagarelli, K. M., Lum, J. A. G., Turkeltaub, P. E., & Ullman, M. T. (2020). Sequence learning in the human brain: A functional neuroanatomical meta-analysis of serial reaction time studies. NeuroImage, 207, Article 116387. https://doi.org/10.1016/j.neuroimage.2019.116387 CrossRefGoogle ScholarPubMed
Kaller, C. P., Rahm, B., Köstering, L., & Unterrainer, J. M. (2011). Reviewing the impact of problem structure on planning: A software tool for analyzing tower tasks. Behavioural Brain Research, 216, 1–8.CrossRefGoogle ScholarPubMed
Kaller, C. P., Unterrainer, J. M., & Stahl, C. (2012). Assessing planning ability with the Tower of London task: Psychometric properties of a structurally balanced problem set. Psychological Assessment, 24, 46–53. https://doi.org/10.1037/a0025174 CrossRefGoogle Scholar
Kalra, P. B., Gabrieli, J. D. E., & Finn, A. S. (2019). Evidence of stable individual differences in implicit learning. Cognition, 190, 199–211. https://doi.org/10.1016/j.cognition.2019.05.007 CrossRefGoogle ScholarPubMed
Knowlton, B. J., Mangels, J. A., & Squire, L. R. (1996). A neostriatal habit learning system in humans. Science, 273, 1399–1402. https://doi.org/10.1126/science.273.5280.1399 CrossRefGoogle ScholarPubMed
Knowlton, B. J., Squire, L. R., & Gluck, M. A. (1994). Probabilistic classification learning in amnesia. Learning & Memory, 1, 106–120. https://doi.org/10.1101/lm.1.2.106 CrossRefGoogle ScholarPubMed
Krus, D. J., & Helmstadter, G. C. (1993). The problem of negative reliabilities. Educational and Psychological Measurement, 53, 643–650. https://doi.org/10.1177/0013164493053003005 CrossRefGoogle Scholar
Lance, C. E., Butts, M. M., & Michels, L. C. (2006). The sources of four commonly reported cutoff criteria: What did they really say? Organizational Research Methods, 9, 202–220. https://doi.org/10.1177/1094428105284919 CrossRefGoogle Scholar
Lum, J. A., Conti-Ramsden, G., Page, D., & Ullman, M. T. (2012). Working, declarative and procedural memory in specific language impairment. Cortex, 48, 1138–1154. https://doi.org/10.1016/j.cortex.2011.06.001 CrossRefGoogle ScholarPubMed
Middleton, F. A., & Strick, P. L. (2000). Basal ganglia and cerebellar loops: Motor and cognitive circuits. Brain Research Reviews, 31, 236–250. https://doi.org/10.1016/S0165-0173(99)00040-5 CrossRefGoogle ScholarPubMed
Morgan-Short, K., Deng, Z., Brill-Schuetz, K. A., Faretta-Stutenberg, M., Wong, P. C. M., & Wong, F. (2015). A view of the neural representation of second language syntax through artificial language learning under implicit contexts of exposure. Studies in Second Language Acquisition, 37, 383–419. https://doi.org/10.1017/S0272263115000030 CrossRefGoogle Scholar
Morgan-Short, K., Faretta-Stutenberg, M., Brill-Schuetz, K., Carpenter, H., & Wong, P. C. M. (2014). Declarative and procedural memory as individual differences in second language acquisition. Bilingualism: Language and Cognition, 17, 56–72. https://doi.org/10.1017/S1366728912000715 CrossRefGoogle Scholar
Morgan-Short, K., & Ullman, M. T. (in press). Declarative and procedural memory in second language learning: Psycholinguistic considerations. In Godfroid, A. & Hopp, H. (Eds.), The Routledge handbook of second language acquisition and psycholinguistics. Routledge.Google Scholar
Morling, B. (2015). Research methods in psychology: Evaluating a world of information (2nd ed.). W. W. Norton & Company.Google Scholar
Nemeth, D., Janacsek, K., & Fiser, J. (2013). Age-dependent and coordinated shift in performance between implicit and explicit skill learning. Frontiers in Computational Neuroscience, 7, Article 147. https://doi.org/10.3389/fncom.2013.00147 CrossRefGoogle ScholarPubMed
Nissen, M. J., & Bullemer, P. (1987). Attentional requirements of learning: Evidence from performance measures. Cognitive Psychology, 19, 1–32.CrossRefGoogle Scholar
Nunally, J. C., & Bernstein, I. H. (1978). Psychometric theory. McGraw-Hill.Google Scholar
Ouellet, M., Beauchamp, M. H., Owen, A. M., & Doyon, J. (2004). Acquiring a cognitive skill with a new repeating version of the Tower of London task. Canadian Journal of Experimental Psychology/Revue Canadienne De Psychologie Expérimentale, 58, 272–288.CrossRefGoogle ScholarPubMed
Owen, A. M., James, M., Leigh, P. N., Summers, B. A., Marsden, C. D., Quinn, N. a., Lange, K. W., & Robbins, T. W. (1992). Fronto-striatal cognitive deficits at different stages of Parkinson’s disease. Brain, 115, 1727–1751.CrossRefGoogle ScholarPubMed
Paradis, M. (2009). Declarative and procedural determinants of second languages. John Benjamins Publishing Company.CrossRefGoogle Scholar
Pili-Moss, D., Brill-Schuetz, K. A., Faretta-Stutenberg, M., & Morgan-Short, K. (2020). Contributions of declarative and procedural memory to accuracy and automatization during second language practice. Bilingualism: Language and Cognition, 23, 639–651. https://doi.org/10.1017/S1366728919000543 CrossRefGoogle Scholar
Raiche, G., & Magis, D. (2020). nFactors: Parallel analysis and other nongraphical solutions to the Cattell scree test. Google Scholar
Raiche, G., Riopel, M., & Blais, J. G. (2006). Nongraphical solutions for the Cattell’s scree test. Paper presented at the annual meeting of the Psychometric Society.Google Scholar
R Core Team. (2019). R: A language and environment for statistical computing. R Foundation for Statistical Computing.Google Scholar
Reber, P. J. (2013). The neural basis of implicit learning and memory: A review of neuropsychological and neuroimaging research. Neuropsychologia, 51, 2026–2042. https://doi.org/10.1016/j.neuropsychologia.2013.06.019 CrossRefGoogle ScholarPubMed
Shallice, T. (1982). Specific impairments of planning. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences, 298, 199–209. https://doi.org/10.1098/rstb.1982.0082 Google ScholarPubMed
Siegelman, N., & Frost, R. (2015). Statistical learning as an individual ability: Theoretical perspectives and empirical evidence. Journal of Memory and Language, 81, 105–120. https://doi.org/10.1016/j.jml.2015.02.001 CrossRefGoogle ScholarPubMed
Song, S., Howard, J. H. Jr. & Howard, D. V. (2007). Implicit probabilistic sequence learning is independent of explicit awareness. Learning and Memory, 14, 167–176. https://doi.org/10.1101/lm.437407 CrossRefGoogle ScholarPubMed
Squire, L. R. (1994). Declarative and nondeclarative memory: Multiple brain systems supporting learning and memory. In Schacter, D. & Tulving, E. (Eds.), Memory systems 1994 (pp. 203–231). MIT Press.Google Scholar
Squire, L. R., & Dede, A. J. O. (2015). Conscious and unconscious memory systems. Cold Spring Harbor Perspectives in Biology, 7, Article a021667. https://doi.org/10.1101/cshperspect.a021667 CrossRefGoogle ScholarPubMed
Squire, L. R., Hamann, S., & Knowlton, B. J. (1994). Dissociable learning and memory systems of the brain. Behavioral and Brain Sciences, 17, 422–423. https://doi.org/10.1017/S0140525X00035330 CrossRefGoogle Scholar
Stark-Inbar, A., Raza, M., Taylor, J. A., & Ivry, R. B. (2017). Individual differences in implicit motor learning: Task specificity in sensorimotor adaptation and sequence learning. Journal of Neurophysiology, 117, 412–428. https://doi.org/10.1152/jn.01141.2015 CrossRefGoogle ScholarPubMed
Suzuki, Y. (2018). The role of procedural learning ability in automatization of L2 morphology under different learning schedules. Studies in Second Language Acquisition, 40, 923–937. https://doi.org/10.1017/S0272263117000249 CrossRefGoogle Scholar
Suzuki, Y., & DeKeyser, R. M. (2017). The interface of explicit and implicit knowledge in a second language: Insights from individual differences in cognitive aptitudes. Language Learning, 67, 747–790. https://doi.org/10.1111/lang.12241 CrossRefGoogle Scholar
Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6th ed.). Pearson.Google Scholar
Tagarelli, K. M., Ruiz, S., Vega, J. L. M., & Rebuschat, P. (2016). Variability in second language learning: The roles of individual differences, learning conditions, and linguistic complexity. Studies in Second Language Acquisition, 38, 293–316. https://doi.org/10.1017/S0272263116000036 CrossRefGoogle Scholar
Trafimow, D. (2015). A defense against the alleged unreliability of difference scores. Cogent Mathematics, 2, Article 1064626. https://doi.org/10.1080/23311835.2015.1064626 CrossRefGoogle Scholar
Trahan, D. E., & Larrabee, G. J. (1988). Continuous visual memory test. Psychological Assessment Resources.Google Scholar
Ullman, M. T. (2004). Contributions of memory circuits to language: The declarative/procedural model. Cognition, 92, 231–270. https://doi.org/10.1016/j.cognition.2003.10.008 CrossRefGoogle ScholarPubMed
Ullman, M. T. (2016). The declarative/procedural model: A neurobiological model of language learning, knowledge, and use. In Hickok, G., & Small, S. L. (Eds.), Neurobiology of language (pp. 953–968). Elsevier Inc. https://doi.org/10.1016/B978-0-12-407794-2.00076-6 CrossRefGoogle Scholar
Ullman, M. T. (2020). The declarative/procedural model. In VanPatten, B., Keating, G. D., & Wulff, S. (Eds.), Theories in second language acquisition (3rd ed., pp. 128–161). Routledge. https://doi.org/10.4324/9780429503986-7 CrossRefGoogle Scholar
Ullman, M. T., Earle, F. S., Walenski, M., & Janacsek, K. (2020). The neurocognition of developmental disorders of language. Annual Review of Psychology, 71, 389–417. https://doi.org/10.1146/annurev-psych-122216-011555 CrossRefGoogle ScholarPubMed
Unterrainer, J. M., Rahm, B., Kaller, C. P., Wild, P. S., Münzel, T., Blettner, M., Lackner, K., Pfeiffer, N., & Beutel, M. E. (2019). Assessing planning ability across the adult life span in a large population-representative sample: Reliability estimates and normative data for the Tower of London (TOL-F) task. Journal of the International Neuropsychological Society, 25, 520–529. https://doi.org/10.1017/S1355617718001248 CrossRefGoogle Scholar
Unterrainer, J. M., Rahm, B., Leonhart, R., Ruff, C. C., & Halsband, U. (2003). The Tower of London: The impact of instructions, cueing, and learning on planning abilities. Cognitive Brain Research, 17, 675–683. https://doi.org/10.1016/S0926-6410(03)00191-5 CrossRefGoogle Scholar
Van den Heuvel, O. A., Veltman, D. J., Groenewegen, H. J., Cath, D. C., van Balkom, A. J. L. M., van Hartskamp, J., Barkhof, F., & van Dyck, R. (2005). Frontal-striatal dysfunction during planning in obsessive-compulsive disorder. Archives of General Psychiatry, 62, 301–310. https://doi.org/10.1001/archpsyc.62.3.301 CrossRefGoogle ScholarPubMed
Willingham, D. B., Salidis, J., & Gabrieli, J. D. E. (2002). Direct comparison of neural systems mediating conscious and unconscious skill learning. Journal of Neurophysiology, 88, 1451–1460. https://doi.org/10.1152/jn.2002.88.3.1451 CrossRefGoogle ScholarPubMed
Yong, A. G., & Pearce, S. (2013). A beginner’s guide to factor analysis: Focusing on exploratory factor analysis. Tutorials in Quantitative Methods for Psychology, 9, 79–94.CrossRefGoogle Scholar
Supplementary material: File

Buffington et al. supplementary material

Buffington et al. supplementary material

Download Buffington et al. supplementary material(File)
File 61.8 KB