Hostname: page-component-76d6cb85b7-5qg8f Total loading time: 0 Render date: 2026-07-18T05:14:01.343Z Has data issue: false hasContentIssue false
  • English
  • Français

Rich data drive generalization: Lessons from machine learning for linguistics and cognitive science

Published online by Cambridge University Press:  01 July 2026

Andrew Kyle Lampinen Open the ORCID record for Andrew Kyle Lampinen [Opens in a new window]
Andrew Kyle Lampinen*
Affiliation:
Google DeepMind, USA lampinen@google.com
*
*Corresponding author.
Get access Rights & Permissions [Opens in a new window]

Abstract

The diversity of variation captured in data can strongly affect the generalization of a learning system – even when that variation occurs along axes orthogonal to the generalization in question. Thus, I argue that data richness both distinguishes current language models from prior linguistic models and may still underlie their remaining linguistic data inefficiency.

Information

Type
Open Peer Commentary
Information
Behavioral and Brain Sciences , Volume 49 , 2026 , e212
Check for updates
Copyright
© The Author(s), 2026. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Ahuja, K., Balachandran, V., Panwar, M., He, T., Smith, N. A., Goyal, N., & Tsvetkov, Y. (2025). Learning syntax without planting trees: Understanding hierarchical generalization in transformers. Transactions of the Association for Computational Linguistics, 13, 121–141.10.1162/tacl_a_00733CrossRefGoogle Scholar
Carvalho, W., & Lampinen, A. (2025). Naturalistic Computational Cognitive Science: Towards generalizable models and theories that capture the full range of natural behavior. arXiv preprint arXiv:2502.20349.Google Scholar
del Rio, F., Raymond-Sáez, A., Florea, D., Icarte, R. T., Hurtado, J., Calderon, C. B., & Soto, A. (2025). Data distributional properties as inductive bias for systematic generalization. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 25590–25601).Google Scholar
Gervain, J., Christophe, A., & Mazuka, R. (2020). Prosodic bootstrapping. The Oxford Handbook of Language Prosody, 563–573.Google Scholar
Goyal, P., Duval, Q., Seessel, I., Caron, M., Misra, I., Sagun, L., Joulin, A., & Bojanowski, P. (2022). Vision models are more robust and fair when pretrained on uncurated images without supervision. arXiv preprint arXiv:2202.08360.Google Scholar
Hill, F., Lampinen, A., Schneider, R., Clark, S., Botvinick, M., McClelland, J. L., & Santoro, A. (2020) Environmental drivers of systematicity and generalization in a situated agent. In Proceedings of the International Conference on Learning Representations.Google Scholar
Iverson, J. M., & Goldin-Meadow, S. (2005). Gesture paves the way for language development. Psychological Science, 16(5), 367–371.10.1111/j.0956-7976.2005.01542.xCrossRefGoogle ScholarPubMed
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2022, May). Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 8424–8445).CrossRefGoogle Scholar
Misra, K., & Mahowald, K. (2024). Language models learn rare phenomena from less rare phenomena: The case of the missing AANNs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 913–929). Association for Computational Linguistics.10.18653/v1/2024.emnlp-main.53CrossRefGoogle Scholar
Qin, T., Saphra, N., & Alvarez-Melis, D. (2025). Sometimes I am a tree: Data drives unstable hierarchical generalization. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing 2025 Nov (pp. 11733–11751).CrossRefGoogle Scholar
Raviv, L., Lupyan, G., & Green, S. C. (2022). How variability shapes learning and generalization. Trends in Cognitive Sciences, 26(6), 462–483.10.1016/j.tics.2022.03.007CrossRefGoogle ScholarPubMed
Sim, M. Y., Zhang, W. E., Dai, X., & Fang, B. (2025). Can vlms actually see and read? A survey on modality collapse in vision-language models. In Findings of the association for computational linguistics: ACL 2025 (pp. 24452–24470). Association for Computational Linguistics.10.18653/v1/2025.findings-acl.1256CrossRefGoogle Scholar
Tomasello, M., & Farrar, M. J. (1986). Joint attention and early language. Child development, 57(6), 1454–1463.10.2307/1130423CrossRefGoogle ScholarPubMed
Yedetore, A., & Kim, N. (2024). Semantic training signals promote hierarchical syntactic generalization in transformers. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 4059–4073).10.18653/v1/2024.emnlp-main.235CrossRefGoogle Scholar
Yu, Y., Khadivi, S., & Xu, J. (2022). Can data diversity enhance learning generalization?. In Proceedings of the 29th International Conference on Computational Linguistics (pp. 4933–4945).Google Scholar