Hostname: page-component-5d84bcc8dc-z9l2z Total loading time: 0 Render date: 2026-09-05T15:48:05.404Z Has data issue: false hasContentIssue false
  • English
  • Français

What Can Linguistics and Deep Learning Contribute to Each Other? Response to Pater

Published online by Cambridge University Press:  01 January 2026

Tal Linzen
Tal Linzen*
Affiliation:
Johns Hopkins University
*
Department of Cognitive Science, Johns Hopkins University, 3400 N. Charles St., Baltimore, MD [tal.linzen@jhu.edu]
Get access Rights & Permissions [Opens in a new window]

Abstract

Joe Pater's (2019) target article calls for greater interaction between neural network research and linguistics. I expand on this call and show how such interaction can benefit both fields. Linguists can contribute to research on neural networks for language technologies by clearly delineating the linguistic capabilities that can be expected of such systems, and by constructing controlled experimental paradigms that can determine whether those desiderata have been met. In the other direction, neural networks can benefit the scientific study of language by providing infrastructure for modeling human sentence processing and for evaluating the necessity of particular innate constraints on language acquisition.

Keywords

neural networkspsycholinguisticspoverty of the stimulusdeep learningsyntaxsentence processing

Information

Type
Perspectives
Information
Language , Volume 95 , Issue 1 , March 2019 , pp. e99 - e108
Copyright
Copyright © 2019 Linguistic Society of America

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Adger, David. 2003. Core syntax: A minimalist approach. Oxford: Oxford University Press.CrossRefGoogle Scholar
Badecker, William, and Kuminiak, Frantisec. 2007. Morphology, agreement and working memory retrieval in sentence production: Evidence from gender and case in Slovak. Journal of Memory and Language 56(1). 65–85. DOI: 10.1016/j.jml.2006.08.004.CrossRefGoogle Scholar
Bernardy, Jean-Philippe, and Lappin, Shalom. 2017. Using deep neural networks to learn syntactic agreement. Linguistic Issues in Language Technology 15(2). Online: http://csli-lilt.stanford.edu/ojs/index.php/LiLT/article/view/94/79.CrossRefGoogle Scholar
Bock, Kathryn, and Cutting, J. Cooper. 1992. Regulating mental energy: Performance units in language production. Journal of Memory and Language 31(1). 99–127. DOI: 10.1016/0749-596X(92)90007-K.CrossRefGoogle Scholar
Bock, Kathryn, and Miller, Carol A.. 1991. Broken agreement. Cognitive Psychology 23(1). 45–93. DOI: 10.1016/0010-0285(91)90003-7.CrossRefGoogle ScholarPubMed
Chomsky, Noam. 1957. Syntactic structures. The Hague: Mouton.CrossRefGoogle Scholar
Chomsky, Noam. 1971. Problems of knowledge and freedom. New York: Pantheon Books.Google Scholar
Chowdhury, Shammur Absar, and Zamparelli, Roberto. 2018. RNN simulations of grammaticality judgments on long-distance dependencies. Proceedings of the 27th International Conference on Computational Linguistics, 133-44. Online: http://aclweb.org/anthology/C18-1012.Google Scholar
Crain, Stephen, and Nakayama, Mineharu. 1987. Structure dependence in grammar formation. Language 63(3). 522-43. DOI: 10.2307/415004.CrossRefGoogle Scholar
Dyer, Chris, Kuncoro, Adhiguna, Ballesteros, Miguel; and Smith, A. Noah. 2016. Recurrent neural network grammars. Proceedings of NAACL-HLT 2016, 199–209. Online: https://www.aclweb.org/anthology/N16-1024.Google Scholar
Elman, Jeffrey L. 1991. Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning 7(2–3). 195–225. DOI: 10.1023/A:1022699029236.CrossRefGoogle Scholar
Ettinger, Allyson, Elgohary, Ahmed, Phillips, Colin; and Resnik, Philip. 2018. Assessing composition in sentence vector representations. Proceedings of the 27th International Conference on Computational Linguistics, 1790–1801. Online: http://aclweb.org/anthology/C18-1152.Google Scholar
Fossum, Victoria, and Levy, Roger. 2012. Sequential vs. hierarchical syntactic models of human incremental sentence processing. Proceedings of the 3rd Workshop on Cognitive Modeling and Computational Linguistics, 61–69. Online: http://aclweb.org/anthology/W12-1706.Google Scholar
Frank, Robert, and Mathis, Donald. 2007. Transformational networks. Proceedings of the 3rd Workshop on Psychocomputational Models of Human Language Acquisition. Online: http://blogs.umass.edu/brain-wars/files/2017/06/cogsci-2007.pdf.Google Scholar
Frank, Stefan L., and Bod, Rens. 2011. Insensitivity of the human sentence-processing system to hierarchical structure. Psychological Science 22(6). 829-34. DOI: 10.1177/0956797611409589.CrossRefGoogle ScholarPubMed
Goodkind, Adam, and Bicknell, Klinton. 2018. Predictive power of word surprisal for reading times is a linear function of language model quality. Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018), 10–18. Online: http://aclweb.org/anthology/W18-0102.Google Scholar
Gulordava, Kristina, Bojanowski, Piotr, Grave, Edouard, Linzen, Tal; and Baroni, Marco. 2018. Colorless green recurrent networks dream hierarchically. Proceedings of NAACL-HLT 2018, 1195–1205. DOI: 10.18653/v1/N18-1108.CrossRefGoogle Scholar
Hale, John, Dyer, Chris, Kuncoro, Adhiguna; and Brennan, Jonathan. 2018. Finding syntax in human encephalography with beam search. Proceedings of the 56th annual meeting of the Association for Computational Linguistics (long papers), 2727-36. Online: http://aclweb.org/anthology/P18-1254.Google Scholar
Haskell, Todd R., Thornton, Robert; and MacDonald, Maryellen C.. 2010. Experience and grammatical agreement: Statistical learning shapes number agreement production. Cognition 114(2). 151-64. DOI: 10.1016/j.cognition.2009.08.017.CrossRefGoogle ScholarPubMed
Kam, Xuân-Nga Cao, Stoyneshka, Iglika, Tornyova, Lidiya, Fodor, Janet D.; and Sakas, William G.. 2008. Bigrams and the richness of the stimulus. Cognitive Science 32(4). 771-87. DOI: 10.1080/03640210802067053.CrossRefGoogle ScholarPubMed
Kuncoro, Adhiguna, Dyer, Chris, Hale, John, Yogatama, Dani, Clark, Stephen; and Blunsom, Phil. 2018. LSTMs can learn syntax-sensitive dependencies well, but modeling structure makes them better. Proceedings of the 56th annual meeting of the Association for Computational Linguistics (long papers), 1426-36. Online: http://aclweb.org/anthology/P18-1132.Google Scholar
Lau, Jey Han, Clark, Alexander; and Lappin, Shalom. 2017. Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cognitive Science 41(5). 1202-47. DOI: 10.1111/cogs.12414.CrossRefGoogle ScholarPubMed
Lewis, John D., and Elman, Jeffrey L.. 2001. Learnability and the statistical structure of language: Poverty of stimulus arguments revisited. Proceedings of the Boston University Conference on Language Development (BUCLD) 26. 359-70.Google Scholar
Linzen, Tal, Dupoux, Emmanuel; and Goldberg, Yoav. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics 4. 521-35. Online: http://aclweb.org/anthology/Q16-1037.CrossRefGoogle Scholar
Linzen, Tal, and Leonard, Brian. 2018. Distinct patterns of syntactic agreement errors in recurrent networks and humans. Proceedings of the 40th annual conference of the Cognitive Science Society (CogSci 2018), 692-97. Online: http://mindmodeling.org/cogsci2018/papers/0147/0147.pdf.Google Scholar
Marvin, Rebecca, and Linzen, Tal. 2018. Targeted syntactic evaluation of language models. Proceedings of the 2018 conference on Empirical Methods in Natural Language Processing, 1192–1202. Online: http://aclweb.org/anthology/D18-1151.Google Scholar
McCoy, R. Thomas, Frank, Robert; and Linzen, Tal. 2018. Revisiting the poverty of the stimulus: Hierarchical generalization without a hierarchical bias in recurrent neural networks. Proceedings of the 40th annual conference of the Cognitive Science Society (CogSci 2018), 2093-98. Online: http://mindmodeling.org/cogsci2018/papers/0399/0399.pdf.Google Scholar
Pater, Joe. 2019. Generative linguistics and neural networks at 60: Foundation, friction, and fusion. Language 95(1). e41–e74.CrossRefGoogle Scholar
Pollack, Jordan B. 1990. Recursive distributed representations. Artificial Intelligence 46(1). 77–105. DOI: 10.1016/0004-3702(90)90005-K.CrossRefGoogle Scholar
Reali, Florencia, and Christiansen, Morten H.. 2005. Uncovering the richness of the stimulus: Structure dependence and indirect statistical evidence. Cognitive Science 29(6). 1007-28. DOI: 10.1207/s15516709cog0000_28.CrossRefGoogle ScholarPubMed
Roark, Brian. 2001. Probabilistic top-down parsing and language modeling. Computational Linguistics 27(2). 249-76. DOI: 10.1162/089120101750300526.CrossRefGoogle Scholar
Smith, Nathaniel J., and Levy, Roger. 2013. The effect of word predictability on reading time is logarithmic. Cognition 128(3). 302-19. DOI: 10.1016/j.cognition.2013.02.013.CrossRefGoogle ScholarPubMed
Tran, Ke, Bisazza, Arianna; and Monz, Christof. 2018. The importance of being recurrent for modeling hierarchical structure. Proceedings of the 2018 conference on Empirical Methods in Natural Language Processing, 4731-36. Online: http://aclweb.org/anthology/D18-1503.Google Scholar
van Schijndel, Marten, Exley, Andy; and Schuler, William. 2013. A model of language processing as hierarchic sequential prediction. Topics in Cognitive Science 5(3). 522-40. DOI: 10.1111/tops.12034.CrossRefGoogle Scholar
van Schijndel, Marten, and Linzen, Tal. 2018a. Modeling garden path effects without explicit hierarchical syntax. Proceedings of the 40th annual conference of the Cognitive Science Society (CogSci 2018), 2603-8. Online: http://mindmodeling.org/cogsci2018/papers/0496/0496.pdf.Google Scholar
van Schijndel, Marten, and Linzen, Tal. 2018b. A neural model of adaptation in reading. Proceedings of the 2018 conference on Empirical Methods in Natural Language Processing, 4704-10. Online: http://aclweb.org/anthology/D18-1499.Google Scholar
Vaswani, Ashish, Shazeer, Noam, Parmar, Niki, Uszkoreit, Jakob, Jones, Llion, Gomez, Aidan N., Lukasz, Kaiser; and Polosukhin, Illia. 2017. Attention is all you need. Advances in Neural Information Processing Systems (NIPS) 30. 5998–6008. Online: https://papers.nips.cc/paper/7181-attention-is-all-you-need.Google Scholar
Weiss, Gail, Goldberg, Yoav; and Yahav, Eran. 2018. On the practical computational power of finite precision RNNs for language recognition. Proceedings of the 56th annual meeting of the Association for Computational Linguistics (short papers), 740-45. Online: http://aclweb.org/anthology/P18-2117.Google Scholar
Williams, Adina, Drozdov, Andrew; and Bowman, Samuel R.. 2018. Do latent tree learning models identify meaningful structure in sentences? Transactions of the Association for Computational Linguistics 6. 253-67. DOI: 10.1162/tacl_a_00019.CrossRefGoogle Scholar
Xiang, Ming, Dillon, Brian; and Phillips, Colin. 2009. Illusory licensing effects across dependency types: ERP evidence. Brain and Language 108(1). 40–55. DOI: 10.1016/j.bandl.2008.10.002.CrossRefGoogle ScholarPubMed