A promise of LLMs: Modeling probabilistic constraint interaction

Like Futrell and Mahowald (F&M), we see one of the major promises of LLMs as a new tool for modeling soft constraint interactions. By including different constraints into an LLM’s architecture or training objective, one can simulate the causal effect of such constraints on the resulting model behavior and assess their effects individually or together. Crucially, because language models (LMs) can be trained on a human lifetime (or more) of linguistic input, one can run simulations at a level of ecological validity that was not available to previous generations of researchers.

In this response, we wish to deepen and extend the discussion in F&M by connecting it to allied fields. We also raise important phenomena from human language acquisition that we suggest should be included in testbeds for LLMs.

Why study constraint interaction? Looking beyond deep learning

F&M motivate the power and promise of graded constraint interaction by pointing to the literature on deep learning and Artificial Intelligence (AI). However, evidence in allied fields also supports the perspective that learning biases in biological systems can be gradual, probabilistic, and, in some cases, domain-general.

As one example, consider imprinting, the acquired attachment system in ducks and geese. Imprinting is commonly explained as resulting from the interaction of probabilistic biases or gradients of salience during learning (Bateson, Reference Bateson1979; Hess, Reference Hess1973). As soon as they can locomote, ducklings will follow moving objects, which produces attachment to the followed object. The onset of the ability to walk, therefore, opens a critical period for imprinting. The more intense the motor behavior, the stronger the imprinting, with no imprinting if the animal is passively carried and very strong imprinting if the animal must climb over hurdles on the imprinting track (Hess, Reference Hess1958). There are also gradients in the properties of attachment objects: a real duck or a sphere elicits stronger imprinting than other shapes. And there is a gradual decline in the strength of imprinting with hours after hatching. The end of the critical period is produced by the onset of fear of novel objects: ducklings begin to run away from a new object rather than follow it. In interaction in the natural world, these gradients produce strong and long-lasting attachment to duck-like moving objects that are followed within a short time after the development of locomotion. In this example, a fairly complex behavioral system has evolved successfully not by innately specifying the behavioral outcome but rather by the interaction of preferences and saliences that conspire with normal environmental input to achieve successful duck life.

Research on what begins and ends critical periods in many behavioral systems also shows that their neural underpinnings are an interacting set of gradient processes. Learning and synaptic sprouting begin when excitatory and inhibitory inputs to a network are in place; they end when myelin and the extracellular matrix develop, locking in the connections that are most functional. All of these processes are dynamic, not fixed – they are affected by the strength of the inputs they receive – so that “critical periods” can begin and end at a variety of times, depending on the environment (Hensch, Reference Hensch2005).

The natural world abounds with complex systems whose behavior is determined by interactions between graded, fuzzy constraints. We are optimistic that human language acquisition can ultimately find a similar constraint-based explanation and see ANN modeling as one important route towards realizing this goal.

Four challenges for LLM modeling

Below, we outline several challenges to which cognitive modeling can contribute and which we argue should be prioritized in the next decade.

Learning beyond the input: Most studies with LLMs train models on data that contain the full structural complexity of the human language to be acquired and show that the model makes the generalizations exhibited in the linguistic input. However, when their input does not contain full complexity or consistency, children modify language structures, introducing regularities and new structures not present in the original data (Austin et al., Reference Austin, Schuler, Furlong and Newport2022; Singleton & Newport, Reference Singleton and Newport2004). Such behavior is often absent in LLMs and, indeed, absent in adult learners. How can architectural constraints or constraints on learning mechanisms give rise to childlike regularization behavior?

Anything Goes? An important claim in the linguistics literature is that human learners will acquire only languages that observe the principles of natural languages (Bickerton, Reference Bickerton1984; Chomsky, Reference Chomsky1965, Reference Chomsky1981). This has been shown in artificial language learning experiments (Culbertson & Newport Reference Culbertson and Newport2015, Reference Culbertson and Newport2017; Newport Reference Newport2016) and for LLMs (Kalini et al., Reference Kallini, Papadimitriou, Futrell, Mahowald and Potts2024). However, this preference appears to be weaker in LMs than in people (Yang et al., Reference Yang, Aoyama, Yao and Wilcox2025). Children will dramatically change languages that violate linguistic universals to restore patterns common to human languages (Culbertson & Newport, Reference Culbertson and Newport2015, Reference Culbertson and Newport2017). What types of LLM mechanisms might reproduce these strong constraints?

Explaining Hierarchy: Our theory of hierarchy in language, roughly that people learn rules within a single context-free grammar, has not changed in 70 years. Yet LLMs obtain linguistic abilities without the computational capability to represent such grammars (Hahn, Reference Hahn2020). We must explain how and why soft constraints give rise to something equivalent to hierarchical LLMs, what this consists of, and whether it necessitates a new theory of hierarchy and structure in human language.

Human-Sized Data: While we are sympathetic to the authors’ “Contravariance Principle,” we believe that learning on 30 trillion words (LLMs) vs. 100 million words (roughly, humans) presents a vast difference in problem space, enough to question the contravariance principle. Linguists interested in computational modeling should run experiments on human-sized models (Warstadt & Bowman, Reference Warstadt, Bowman, Lappin and Bernardy2022; Wilcox et al., Reference Wilcox, Hu, Mueller, Warstadt, Choshen, Zhuang, Williams, Cotterell and Linzen2025). Thus far, “BabyLM” models fall short of people, implying that current architectures are not sufficient to make the correct linguistic generalizations from a plausible amount of training data (Warstadt et al., Reference Warstadt, Mueller, Choshen, Wilcox, Zhuang, Ciro, Mosquera, Paranjape, Williams, Linzen and Cotterell2023). How can soft constraints enable LLMs to learn from a biologically plausible data size?