2.1. Information theory for linguistic description

Information theory is fundamentally based on a mathematical equivalence between probability and description length. The description length of an object $ x $ is simply the length of a string that fully specifies $ x $ in some description language. Such description lengths have long been discussed as complexity metrics in linguistics and other fields (Rissanen Reference Rissanen1978, Grünwald Reference Grünwald2007, Voita & Titov Reference Voita and Titov2020), and formalisms (description languages) for linguistic phenomena may be evaluated and compared in terms of the description lengths they assign to linguistic data (Chomsky Reference Chomsky1965:43–47).

Suppose that you are observing a number of objects, one after another, and want to find a description language that you can use to write down descriptions of these objects such that the average description length per object is minimal. Without loss of generality, assume that the description language is binary—so that a description consists of a string of 0s and 1s. Shannon (Reference Shannon1948) showed that for an object $ x $ that appears with probability p(x), its description length in this optimal description language cannot be shorter than

(1) $$ S(x)\equiv -\mathrm{log}\ p(x), $$

a quantity called self-information or surprisal and measured in bits (with the logarithm taken to base 2, corresponding to a description language with two symbols). The average of surprisal over all possible objects x is called entropy and notated H[X].

(2) $$ H\left[X\right]\equiv \sum \limits_xp(x)S(x) $$

Surprisal measures the information content of an object x in terms of the description length for x in an optimal description language.

While surprisal is a useful quantity in its own right, it applies to only one variable, whereas linguistic description often involves characterizing interactions of multiple variables. For example, a linguist may be interested in the cooccurrence of bigrams or in characterizing cooccurrence patterns of stems and affixes. To this end, information theory gives us the concept of mutual information (MI), a quantity that describes the predictive relationship between two variables. Suppose we have two random variables X and Y, with possible outcomes labeled x and y, respectively, as well as a third random variable Z with possible outcomes labeled z. The MI of X and Y conditional on Z describes how well knowledge of X allows us to predict Y, given complete knowledge of Z.

(3) $$ I\left[X:Y|Z\right]\equiv \sum \limits_{x,y,z}p\left(x,y,z\right)\log \frac{p\left(x,y|z\right)}{p\left(x|z\right)p\left(y|z\right)} $$

If X is highly informative about Y given Z, then I[X : Y | Z] will be very high, and if X and Y are independent given Z, then we have I[X : Y | Z] = 0. Like surprisal, mutual information has an interpretation in terms of description length. It measures the extent to which knowledge of a variable Y enables a shorter description of a correlated variable X.

The connections above lead immediately to applications of information theory in describing the complexity of linguistic systems (for a selection of recent examples, see Bentz et al. Reference Bentz, Alikaniotis, Cysouw and Ferrer-i-Cancho2017, Koplenig et al. Reference Koplenig, Meyer, Wolfer and Müller-Spitzer2017, Pimentel et al. Reference Pimentel, McCarthy, Blasi, Roark and Cotterell2019). For example, Ackerman and Malouf (Reference Ackerman and Malouf2013) study the complexity of morphological paradigms. They argue that there are two relevant notions of complexity: enumerative complexity (e-complexity), and integrative complexity (i-complexity). The e-complexity is defined as the number of forms in the paradigm, while the i-complexity is meant to measure the difficulty that the system poses for the users of the language. The i-complexity of a cell in a paradigm is specifically the average conditional surprisal of forms expressing the grammatical features for that cell, given another form in the paradigm. If the forms within a cell are not predictable from the other form, then they will have high surprisal, and thus high i-complexity. Ackerman and Malouf (Reference Ackerman and Malouf2013) furthermore argue that real morphological paradigms can have unrestricted e-complexity as long as their i-complexity is sufficiently low. In follow-up work, Cotterell et al. (Reference Cotterell, Kirov, Hulden and Eisner2019) show that there is a tradeoff between e-complexity and i-complexity, measuring the i-complexity as the sum of the irregularity of the paradigm (using a measure of irregularity to be defined below).

In this article, we use a closely related measure, called ‘informational fusion’, to quantify the degree of fusion within a cell in a morphological paradigm. This measure was introduced in Rathi et al. Reference Rathi, Hahn and Futrell2021. For convenience, and as a concrete example of the use of information theory for linguistic description, we describe this measure below.

2.1.1. Informational fusion

Informational fusion intuitively captures the extent to which a set of features is expressed in a word form in a way that is unanalyzable. If a word form cannot be explained in terms of morphological processes corresponding to any subset of the features in question, then it should have high informational fusion. More precisely, informational fusion is the bits of information required to specify a form for a given set of features, beyond what is explained by any strict subset of those features.

Formally, the informational fusion of a surface form w in a language with respect to feature set σ and lemma ℓ is the surprisal of the surface form given all the forms for feature sets other than σ:

(4) $$ \varphi (w)=-\log\;p\left(w\;|\;\sigma, \ell, {\mathcal{L}}_{-\sigma}\right), $$

where ℒ_−σ is a data set consisting of all word forms in the language along with their features and lemmas, having removed all occurrences of feature set σ. The probability distribution p(∙ | σ, ℓ, ℒ_−σ), which we call the learner model, therefore represents all of the guesses one might make about the form for feature set σ without having ever seen a form with feature set σ, but rather only subsets of σ and unrelated forms. We call this distribution the learner model because it represents the guesses a language learner might make about the forms for an unseen cell in a paradigm. It is not intended to be a realistic model of the process of language learning, but only a model of uncertainty about an unseen cell of a paradigm.

If the form for feature set σ is highly predictable based on the rest of the forms in the language—for example, if it is the result of a composition of morphological processes that are in evidence elsewhere in the paradigm—then it will have low informational fusion. For example, the Hungarian form könyv-ek-et ‘book-pl-acc’ in Table 2 could plausibly be guessed by a learner on the basis of the rest of the paradigm, because a learner could pick up on the existence of an affixation process with suffix -ek for plural number and -et for accusative case. By contrast, if a form is not predictable, it will have high informational fusion. For the Latin form serv-ōs ‘servant-pl.acc’ in Table 1, a learner would have no grounds to guess the form serv-ōs based on the forms in the rest of the paradigm, so this form would come out with high informational fusion. More concretely, the informational fusion of Latin servōs as shown in Table 1 expands to the surprisal of the form given its features σ = ⟨acc, pl⟩ and lemma ℓ = serv- and given all of the other pairings of forms with lemmas and features in the language, so long as their features are not ⟨acc, pl⟩:

(5)

which we would expect to be very high because there are no patterns in the other paradigm cells that could be used to predict the form servōs.

Informational fusion is based on the measure of irregularity from Wu et al. Reference Wu, Cotterell and O’Donnell2019. The difference is that the irregularity measure involves holding out a particular lemma ℓ from the learner model’s data set, whereas informational fusion involves holding out feature set σ from the data set. Informational fusion differs from Ackerman and Malouf’s (Reference Ackerman and Malouf2013) i-complexity in that informational fusion is determined by the probability of the form given every other cell in the paradigm, whereas i-complexity is an average measure concerning the probability of the form given just one other cell.

Informational fusion captures an important aspect of morphological fusion that we believe has not yet been singled out in previous literature. It indicates the extent to which a form is logically predictable given subsets of its features. In contrast, in existing literature, fusion is often understood in opposition to agglutination, where agglutinative forms are understood as those in which morphemes corresponding to individual features are concatenated together. The focus on concatenation of morphemes, or lack thereof, means that it is hard to place systems of noncatenative morphology, such as Semitic root-and-pattern morphology, on a fusional–agglutinative axis. In contrast, informational fusion totally abstracts away from the ideas of ‘morphemes’ and concatenation thereof, and thereby avoids thorny questions of morpheme segmentation.

To see this, note that the concept ‘morpheme’ is never explicitly referred to in our above definition. Instead, we define informational fusion in terms of a learner model, which is a probability distribution on forms given features. This setup is entirely agnostic about the structure of the mapping from features to forms, and in principle, the learner could encompass any conceivable such mapping. Therefore, informational fusion measures the extent to which a form for feature set σ cannot be predicted in terms of any morphological processes or patterns corresponding to any strict subsets of σ—where morphological processes include not only affixation but also alternations, reduplication, suppletion, and in principle any number of logically possible but linguistically implausible mappings. The implementation of a specific learner model may make reference to the idea of morphemes, but the informational fusion concept on its own does not, and is therefore compatible with a variety of theories of morphology.

The fact that informational fusion does not rely explicitly on a notion of morpheme is important not only conceptually but also practically, because it means that the actual calculation of informational fusion does not need to rely on segmented data. However, while informational fusion does not require segmented data as input, it does require a specification of the learner model p(∙ | σ, ℓ, ℒ_−σ). It would be possible in principle to specify this distribution ‘by hand’, for example, by building in possible morphological processes explicitly and assigning them prior probabilities, then by fitting this distribution to paradigm data. For example, the learner model could be specified in a way that allows only affixation. Then informational fusion would come out to 0 for a fully agglutinative form and ∞ for a form that cannot be explained in terms of concatenated morphemes for its constituent features.

However, we have found that it is possible to get sensible estimates of informational fusion by specifying the learner model using a generic neural sequence-to-sequence transduction architecture (Sutskever et al. Reference Sutskever, Vinyals and Le2014); this architecture can in principle represent any relationship among strings. See Rathi et al. Reference Rathi, Hahn and Futrell2021 for an evaluation of the ability of informational fusion with a sequence-to-sequence learner model to reproduce traditional linguistic categorizations.

Informational fusion also abstracts away in principle from phonological processes: all that matters is whether the surface form is predictable given the features. If the learner model is aware of phonological processes in the relevant language, or is sufficiently powerful to infer them from the training data provided, then phonological processes alone will not make forms unpredictable, and informational fusion scores will be low.

The simplest interpretation of informational fusion is in terms of the description length for a form given a set of features. For example, in an almost completely systematic morphological system such as Hungarian nouns (see Table 2 above), the plural dative form can be predicted with high confidence based on observations of other cases and numbers, so its informational fusion value is nearly zero. This means that a full description of the Hungarian noun paradigm would have to allocate nearly zero bits for this particular cell of the paradigm given the rest.

The example of Hungarian also illustrates the ways in which the informational fusion measure is dependent on the feature system chosen by the analyst. Hungarian has (at least) six cases describing locations; these could be interpreted as six distinct atomic feature values, as in Table 2, or factorized into component features, as in Table 3.

Table 3.

Hungarian cases describing locations, for könyv- ‘book’.

Table 3. Long description

From left to right, the first column lists spatial relations: in, on, at. The second column shows static forms: könyvben for in, könyven for on, könyvnél for at. The third column displays motion toward forms: könyvbe for in, könyvre for on, könyvhez for at. The fourth column presents motion away forms: könyvből for in, könyvről for on, könyvtől for at. Each row aligns the spatial relation with its corresponding Hungarian case forms for the noun book.

If the six cases shown in Table 2 above are considered as six atomic features, and if we consider only singular forms, then all of the forms will come out with high informational fusion, because there is no basis on which to predict any one from any other. By contrast, if we consider the six cases to break down into two features, representing location type and motion type, then we would measure low informational fusion: for example, the form könyvről (‘on’, motion away) could be predicted from the from könyvre (‘on’, motion toward) on the basis of an analogy to könyvből (‘in’, motion away) and könyvbe (‘in’, motion toward). The form könyvtől (‘at’, motion away) would likely come out as partially fusional, with low but nonzero informational fusion, because the -ől element is predictable but the -t- is not. In fact, the -t- is a regular remnant of an older, now nonproductive locative case in Hungarian.

This example shows a general property of informational fusion: informational fusion for a fine-grained system is lower than informational fusion for a more coarse-grained version of the same system. When the six cases are analyzed into component features, more structure in the paradigm can be discovered, so the informational fusion can go down. This property is in keeping with the intuitive description of informational fusion as ‘the extent to which a form cannot be predicted on the basis of any strict subset of its features’. In this sense, the measurement of informational fusion does depend on the feature system selected by the analyst, but is not entirely relative: one cannot legitimately measure higher informational fusion when passing from a more coarse-grained to a more fine-grained feature system. More generally, one cannot achieve arbitrary values of informational fusion by choosing a feature system.

The sum of average informational fusion values for all of the forms in a paradigm constitutes a lower bound on the description length of the whole system. In turn, this description length translates into difficulty for a learner trying to learn a paradigm and for a speaker trying to recall forms in the paradigm.

2.2. Efficiency-based explanation

Above, we discussed the use of information-theoretic concepts for the description of linguistic phenomena such as fusion. Here, we turn to a further, stronger use of information theory: as part of an explanatory theory for linguistic phenomena, in which the universals and variation in linguistic systems are explained in terms of constraints on various kinds of information-theoretic complexity.

A growing body of research argues that linguistic universals and phenomena can be explained in terms of pressures of communicative efficiency (e.g. Christiansen & Chater Reference Christiansen and Chater2008, Haspelmath Reference Haspelmath and Biberauer2008, Jaeger & Tily Reference Jaeger and Tily2011, Fedzechkina et al. Reference Fedzechkina, Jaeger and Newport2012, Cotterell et al. Reference Cotterell, Kirov, Hulden and Eisner2019, Gibson et al. Reference Gibson, Futrell, Piantadosi, Dautriche, Mahowald, Bergen and Levy2019). Futrell, Levy, and Gibson (Reference Futrell, Levy and Gibson2020) title this idea the Efficiency Hypothesis, stating that phenomena of language result from a tradeoff between communication and complexity, where an important part of complexity is the difficulty of on-line production and comprehension under cognitive information-processing constraints. This notion of efficiency is closely tied to information-theoretic approaches, because both parts of the efficiency hypothesis—communication and cognitive information-processing constraints—can be formalized using information-theoretic concepts and are subject to information-theoretic bounds.

As an example of the utility of information theory in understanding constraints in language processing, we review the Surprisal theory of on-line language comprehension (Hale Reference Hale2001, Levy Reference Levy2008). This theory holds that the processing difficulty of word w_t (or any linguistic unit of any size; see Smith & Levy Reference Smith and Levy2013) in context w ₁ … w _t−1 is proportional to its surprisal according to the comprehender’s expectations.

(6) $$ S=-\log\;p\left({w}_t\;|\;{w}_1\hskip2pt \dots \hskip2pt {w}_{t-1}\right) $$

When the expectation distribution p(w_t | w ₁ … w _t−1) is estimated using n-gram models, probabilistic context-free grammars, or neural models, surprisal is a highly accurate predictor of word-by-word reading times (Demberg & Keller Reference Demberg and Keller2008, Smith & Levy Reference Smith and Levy2013, Frank & Ernst Reference Frank and Ernst2019, Wilcox et al. Reference Wilcox, Gauthier, Hu, Qian and Levy2020, Rathi Reference Rathi2021) and has the ability to capture numerous phenomena studied in the sentence-processing literature, such as garden path and antilocality effects (Hale Reference Hale2001, Levy Reference Levy2008) (although see van Schijndel & Linzen Reference van Schijndel and Linzen2021 and Staub Reference Staub2024 for difficulties).

We note that the expectation distribution p(w_t | w ₁ … w _t−1) in surprisal theory is a very different kind of thing from the learner model p(w | σ, ℓ, ℒ_−σ) used as part of the idea of informational fusion above. The expectation distribution p(w_t | w ₁ … w _t−1) represents a comprehender’s expectations about words that might follow in a particular context; it is related to statistical cooccurrence patterns among words in linguistic experience and will relate to factors such as token frequency in corpora. In contrast, the learner model p(w | σ, ℓ, ℒ_−σ) represents a learner’s or speaker’s beliefs about unseen mappings from features to forms.

Within the surprisal theory paradigm, Futrell, Gibson, and Levy (Reference Futrell, Gibson and Levy2020) argue that the context accessible to the processor is best thought of as a lossy memory representation m_t of the true context:

(7) $$ SM=-\log\;p\left({w}_t\;|\;{m}_t\right), $$

where the memory representation m_t = M(w ₁ … w _t−1) is given by some memory-encoding function M. This generalization allows the theory to capture processing phenomena such as structural forgetting and dependency locality effects (Futrell, Gibson, & Levy Reference Futrell, Gibson and Levy2020) and the way that choices of lexical items modulate the difficulty of multiple center-embedding (Hahn et al. Reference Hahn, Futrell, Levy and Gibson2022).

The empirical success of surprisal theory motivates the use of surprisal as a complexity metric for linguistic explanation. Minimization of surprisal translates reliably into observably easier linguistic processing; it is not a purely theoretical complexity metric. Minimization of surprisal will form part of our theory of morphological paradigms.

This previous work has focused on explaining sizes of paradigms and distributions of irregular forms within them. In contrast, our work focuses on explaining the organization of morphemes and morphological processes within a word. We build on Hahn, Degen, and Futrell (Reference Hahn, Degen and Futrell2021), who use the memory–surprisal tradeoff to explain the ordering of morphemes in agglutinative languages (as well as general word order). We focus on extending this idea to explain morphological fusion as well as other subtle morphological phenomena.

2.2.1. The memory–surprisal tradeoff

The memory–surprisal tradeoff describes the complexity of incremental language processing in terms of two factors: (i) the difficulty of predicting upcoming material, and (ii) the difficulty of maintaining memory representations of past material. These two factors trade off: a high-fidelity representation of past material enables accurate predictions about future material, but at the cost of higher investment of memory resources. The Efficient Tradeoff Hypothesis (ETH; Hahn, Degen, & Futrell Reference Hahn, Degen and Futrell2021) holds that languages are structured so that favorable tradeoffs of these two factors are possible: that is, so that upcoming material is highly predictable given only small amounts of information stored in memory.

The core idea of the memory–surprisal tradeoff is that with more information in the memory representation m_t, the average surprisal achieved will be lower. We can quantify the average amount of information stored in the memory state as H_M, the entropy of the memory state:

(8) $$ {H}_M=-\sum \limits_mp(m)\log p(m), $$

where the sum runs over all possible memory states, and the probability distribution p(m) is the probability that the memory state is m at any given time. The memory–surprisal tradeoff curve quantifies the lowest achievable average surprisal S_M given a particular average amount of information stored in memory H_M. With a steeper curve, lower processing difficulty can be achieved while storing less information in memory. Thus, a steeper curve is efficient, in the sense that it is possible to achieve a desired level of on-line processing difficulty at the cost of less memory resources.

Given this setting, Hahn, Degen, and Futrell (Reference Hahn, Degen and Futrell2021:729) formalize the ETH as follows: ‘the order of elements in natural language is characterized by a distinctively steeper memory–surprisal trade-off curve compared to other possible orders’. For example, in Figure 1, Language A has a steeper curve than Language B and would thus be more efficient, as we would expect a natural language to be. The idea of ‘steepness’ is quantified by calculating the Area Under the Curve (AUC), where a lower AUC corresponds to a steeper curve. Hahn, Degen, and Futrell (Reference Hahn, Degen and Futrell2021) furthermore show that an efficient tradeoff is generally achieved when languages follow the Information Locality Principle (Futrell Reference Futrell2019, Futrell & Hahn Reference Futrell and Hahn2022, Futrell & Hahn Reference Futrell and Hahn2026, Mansfield & Kemp Reference Mansfield, Kemp, O’Shannessy and Gray2025), which states that linguistic units that predict each other will be close in linear order (cf. Behaghel Reference Behaghel1932, Givón Reference Givón and Haiman1985).

Figure 1.

Sample memory–surprisal tradeoff curves of two hypothetical languages, Language A and Language B. The curve for Language A is steeper, and thus we would say it is more efficient in terms of the cognitive resources required.

While the ETH was defined in previous work as a theory of word and morpheme order, we use it here as a theory of how grammatical information is packaged into morphemes, for example, as a theory of why multiple features might be expressed in a single morpheme, and which features are likely to be fused in this way. We hypothesize that attested morphological systems (conceived of generally as mappings from sets of features to word forms) achieve more favorable memory–surprisal tradeoffs than alternative systems. The information locality principle carries over into this domain in a modified form: we generally predict that features are likely to be expressed by a single morpheme when they statistically depend on each other.

Our general approach, therefore, is to calculate the memory–surprisal tradeoff for attested morphological systems and to compare against other possible systems. We hypothesize that the attested systems will enable more efficient tradeoffs, quantified as the AUC of the memory–surprisal tradeoff.

2.2.2. Estimation of the memory–surprisal tradeoff

Remarkably, we can calculate a bound on the memory–surprisal tradeoff curve that is independent of any assumptions about the representational form of the incremental memory state m_t, requiring only that the memory state m_t at each time depends only on the previous input w _t−1 and the previous memory state m _t−1. A memory representation with this property is called autoregressive. In this extremely general setting, we can calculate a lower bound on the memory–surprisal tradeoff curve by fitting a series of incremental language models that use successively more context to predict upcoming material. This lower bound not only provides a useful calculational tool, but also elucidates the connection between the memory–surprisal tradeoff and information locality.

To calculate the lower bound, first we need a quantity I_t, which is the mutual information between units at a distance of t units:

(9) $$ It={S}_{t-1}-1-{S}_t, $$

where S_t is the average surprisal of linguistic units for a language model that sees a ‘context window’ of size t, that is, a (t + 1)-gram model. Then for all timescales T, for any memory-encoding function M whose entropy satisfies

(10) $$ {H}_M\le \sum \limits_{t=1}^T{tI}_t, $$

the average surprisal is lower-bounded as

(11) $$ {S}_M\ge S+\sum \limits_{t=T+1}^{\infty }{I}_t, $$

where S is surprisal given the full context, as in equation 6. The lower bound on the whole curve can be computed by first calculating values for the mutual informations I ₀, I ₁, I ₂, … and then using these to calculate H_M and S_M for timescales T = 0, 1, 2, … . The intuition behind equation 10 is that bringing I_t bits of predictive information from t time steps in the past to bear on predicting the next token requires keeping these I_t bits in memory for t time steps, resulting in a memory cost of tI_t. In the studies below, we determine I_t using n-gram models with Kneser-Ney smoothing trained on a training set, and estimate average surprisal S_t as cross-entropy on a test set. To mitigate overfitting with larger values of t, we estimate Ŝ_t as min_τ≤t S_t, where S_t is the cross-entropy on the t’th-order Markov model.

The optimal ordering of units can then be determined via minimization of the AUC of the tradeoff determined in this way. To do so, we follow the methodology of Hahn, Degen, and Futrell (Reference Hahn, Degen and Futrell2021), using an adaptation of Gildea and Jaeger’s (Reference Gildea and Jaeger2015) hill-climbing method. For our purposes and given data sets available to us, we measure the memory–surprisal tradeoff at the level of abstract sequences of features. Forms are represented as a sequence consisting of a root and a series of features: for example, the English form goes is represented as root-3-sg-prs, indicating that it expresses the features of third person, singular, and present tense. This representation abstracts away from any ambiguity that might exist in word forms, and also abstracts away from the length of individual morphemes.

2.3. Why information theory?

We conclude this section with a brief discussion of the advantages of information theory for characterizing and explaining morphological paradigms. Information theory is not currently a standard tool in linguistic analysis, but we believe it can be and should be.

The primary advantage of information theory is that it is a fully formed mathematical framework that simultaneously allows for very general deductive inferences while also grounding out in empirically observable variables. As an example of the deductive power of information theory, consider the bound on surprisal as a function of memory entropy in equation 11: this bound holds for all conceivable autoregressive memory-encoding functions. This level of generality is not typical in linguistics and psycholinguistics, where theories are often formulated in terms of concrete representations and mechanisms. In contrast, the memory–surprisal tradeoff describes inevitable effects of memory on language users’ expectations, applying to all possible representations or mechanisms for prediction.

While enabling powerful deductions, information theory can operate while dealing only minimally with unobservable concepts and quantities. All of the information-theoretic quantities in the current work are based on probability distributions that can be estimated from data. In particular, the memory–surprisal tradeoff for word forms can be estimated from word-frequency data, and informational fusion can be estimated from paradigm tables.

3.1. Advantage of fusion: lowering local surprisal

We consider two languages representing minimal examples of agglutination and fusion, as shown in Table 4. Each language encodes two binary features X ₁ and X ₂ into forms that consist of two-character strings drawn from an alphabet {A,B,C,D}. To make the example more concrete, one could take X ₁ to be a voice feature with values active and passive, and X ₂ to be a tense feature with values present and past.

Table 4.

Two toy languages, agglutinative L _agg and fusional L _fus, which are both mappings from two underlying binary features to forms. For example, the third row represents the case where the first underlying feature has the value passive and the second underlying feature has the value present; this is expressed in L _agg as BC and in L _fus as BD. In the forms of the agglutinative language, the first character (A or B) corresponds to the voice feature and the second character (C or D) corresponds to the tense feature. In the forms of the fusional language, the values of the second character (C or D) cannot be identified with values of the tense feature. Probabilities are chosen to create nonzero mutual information between the two features.

Table 4. Long description

Starting from the top row, each entry lists two underlying binary features: voice and tense. The agglutinative language L sub agg encodes these as two-character forms, where the first character (A or B) represents voice and the second character (C or D) represents tense. For example, passive plus present yields BC. The fusional language L sub fus also uses two-character forms, but the second character does not directly correspond to tense. Each row displays the feature combination, the agglutinative form, and the fusional form. Probabilities are assigned to ensure nonzero mutual information between features. The table visually distinguishes how the mapping from features to forms differs between the two language types.

In the agglutinative language L _agg, there is a systematic relationship between individual characters and individual features: in the second position in the string, C means the second (tense) feature has value 0 (present) and D means that it has value 1 (past). Effectively, each character in this language is a morpheme corresponding to one feature.

In contrast, in the fusional language L _fus, there is no systematic relationship between the second feature and any character in the string. Here, the characters C and D cannot be analyzed as morphemes corresponding to feature values present or past, because their meaning depends on other features.

These are the simplest possible toy languages that exhibit a difference between agglutination and fusion while controlling for alphabet size and the lengths of strings. The agglutinative language L _agg represents an identity function applied to the input features; the fusional language L _fus applies a Controlled-NOT operation. The languages also come with probability distributions on the input features, and these are chosen so that the two features have nonzero mutual information.

For examples such as these, we consistently find memory–surprisal tradeoffs such as those shown in Figure 2, favoring the fusional language. Furthermore, the higher the MI between the two features, the greater the advantage of the fusional language.

Figure 2.

Memory–surprisal tradeoffs for the agglutinative language L _agg (solid line) and the fusional language L _fus (dashed line) as shown in Table 4 for different levels of MI between the two input features. As the MI of the input features increases, efficiency of the fusional language increases.

Figure 2. Long description

The graph consists of three panels arranged horizontally. Each panel represents a different mutual information value: left panel M I equals 0.05 bits, center panel M I equals 0.19 bits, right panel M I equals 0.46 bits. In each panel, the x axis is labeled Memory in bits, ranging from 0 to approximately 1.25, and the y axis is labeled Surprisal in bits, ranging from 0.75 to 2.00. Two lines are plotted in each panel: a solid line for the agglutinative language and a dashed line for the fusional language. In the left panel, both lines overlap, showing a linear decrease in surprisal as memory increases. In the center panel, the dashed fusional line shifts below the solid agglutinative line, indicating lower surprisal for the fusional language at the same memory values. In the right panel, this separation increases further, with the fusional line consistently below the agglutinative line. The legend below the panels identifies the solid line as agglutinative and the dashed line as fusional. As mutual information increases from left to right, the efficiency of the fusional language improves relative to the agglutinative language.

Why do we get this pattern in the tradeoff curves? The answer becomes clear if we examine surprisal as a function of the size of the context considered, as shown in Figure 3. Here we see that the difference between the two languages arises entirely from highly local surprisal, when no context is available—that is, when the context window has size t = 0. For the agglutinative language, the surprisal at t = 0 is always two bits, reflecting the fact that the four symbols of the vocabulary are equiprobable in the absence of context. For the fusional language, by contrast, the local surprisal can be lower, reflecting the fact that the symbols C and D are not equiprobable: the symbol D appears only for the lower-probability meanings, while the symbol C appears only for the higher-probability meanings.

Figure 3.

Average surprisal S_t under a (t + 1)-gram model, for the agglutinative language L _agg (solid line) and the fusional language L _fus (dotted line). The fusional language achieves lower surprisal at t = 0, that is, local surprisal.

Figure 3. Long description

There are three panels arranged horizontally. Each panel has surprisal in bits on the y-axis from 0.75 to 2.00 and amount of context t on the x-axis from 0 to 2. The left panel is labeled M I equals 0.05 bits, the middle M I equals 0.19 bits, and the right M I equals 0.46 bits. In all panels, two lines are plotted: a solid line for agglutinative and a dashed line for fusional. In the left panel, both lines decrease linearly from 2.00 at t equals 0 to 1.00 at t equals 1, then remain flat. In the middle panel, the fusional line starts slightly below the agglutinative at t equals 0, both decrease to 1.00 at t equals 1, then flatten. In the right panel, the fusional line starts further below the agglutinative at t equals 0, both decrease to 1.00 at t equals 1, then flatten. The legend below the panels identifies line types.

The structure of this result reveals the key advantage of fusion: it enables a part of a string to covary with two or more input features, rather than just one. Therefore, the local surprisal of that part of the string will better reflect the probability of the meaning as a whole, rather than just the probability of one feature. A general formal version of this argument is found in Appendix Section A1.

3.2. Advantage of agglutination: lowering memory requirements

Having established that the memory–surprisal tradeoff favors fusion of features with high mutual information, we now turn to circumstances in which the memory–surprisal tradeoff favors agglutination. We find that agglutination is favorable when features have low mutual information, that is, when they would be placed far from each other in an optimal ordering.

We consider toy languages in a setting with three underlying binary features X ₁, X ₂, X ₃, with the mutual information I[X ₂ : X ₃] positive and all other mutual informations between features set to 0. For convenience we can identify X ₁ with voice, X ₂ with aspect (perfect or imperfect), and X ₃ with tense. We consider three toy languages where forms consist of three characters Y ₁Y ₂Y ₃: (i) a fully agglutinative language where each character is a one-to-one function of its corresponding feature, called L _agg, (ii) a language that fuses the independent features X ₁ (voice) and X ₂ (aspect), called L _fuse-low, and (iii) a language that fuses the correlated features X ₂ (aspect) and X ₃ (tense), called L _fuse-high. The languages are shown in Table 5.

Table 5.

Three languages expressing three binary features. For ease of understanding, the three features are identified as voice (with values active and passive), aspect (with values perfect and imperfect), and tense (with values present and past). The probabilities are chosen so that tense and aspect features have MI of 0.19 bits, while voice and tense have MI of 0 bits. In the language L _agg, each character in the form corresponds to one input feature. In the language L _fuse-low, the first two characters jointly express the voice and aspect features (which have zero mutual information). In the language L _fuse-high, the second and third characters jointly express the aspect and tense features (which have high mutual information).

Table 5. Long description

The table has five columns: Probability, Features (X sub 1, X sub 2, X sub 3), L sub agg form, L sub fuse-low form, and L sub fuse-high form. From the top row downward: Row 1 shows probability 3/16, features active, perfect, present, and forms ABC in all three languages. Row 2 has 1/16, active, perfect, past, and ABE in all languages. Row 3 has 1/16, active, imperfect, present, with AFC in L sub agg and L sub fuse-low, but AFE in L sub fuse-high. Row 4 has 3/16, active, imperfect, past, with AFE in L sub agg and L sub fuse-low, but AFC in L sub fuse-high. Row 5 has 3/16, passive, perfect, present, with GBC in L sub agg and L sub fuse-high, but GFC in L sub fuse-low. Row 6 has 1/16, passive, perfect, past, with GBE in L sub agg and L sub fuse-high, but GFE in L sub fuse-low. Row 7 has 1/16, passive, imperfect, present, with GFC in L sub agg, GBC in L sub fuse-low, and GFE in L sub fuse-high. Row 8 has 3/16, passive, imperfect, past, with GFE in L sub agg, GBE in L sub fuse-low, and GFC in L sub fuse-high. The forms in each language reflect different patterns of feature fusion, as described in the caption.

The memory–surprisal tradeoff curves for the three languages are shown in Figure 4. We find that the language L _fuse-low, which fuses the independent features X ₁ and X ₂, has a less favorable memory–surprisal tradeoff curve than the agglutinative language L _agg. The difference is in evidence in the high-memory range, where the improperly fused language requires more memory to achieve the same level of surprisal as the agglutinative language. In contrast, the language that fuses high-MI features, L _fuse-high, is superior to the agglutinative language L _agg and to L _fuse-low.

Figure 4.

Memory–surprisal tradeoffs for the agglutinative language L _agg (solid line), the language fusing high-MI features (light dashed line), and the language fusing low-MI features (heavy dashed line) as shown in Table 5 for different levels of MI between the two input features. Note that the lines often cover each other.

Figure 4. Long description

From left to right, each panel is labeled at the top with M I equals 0.05 bits, M I equals 0.19 bits, and M I equals 0.46 bits. The x-axis in all panels is labeled Memory in bits, ranging from 0.0 to 1.5. The y-axis is labeled Surprisal in bits, ranging from 1.0 to 2.5. Each panel contains three lines: a solid line for agglutinative, a light dashed line for fusing high-M I features, and a heavy dashed line for fusing low-M I features. In the leftmost panel, all three lines overlap, showing a linear decrease in surprisal as memory increases. In the middle panel, the lines begin to diverge slightly, with the agglutinative and high-M I fusion lines nearly overlapping and the low-M I fusion line slightly below. In the rightmost panel, the agglutinative and high-M I fusion lines remain close, but the low-M I fusion line is distinctly lower, indicating lower surprisal for the same memory. The legend at the bottom identifies line types. The main trend is that as M I increases, the difference between language types becomes more pronounced, especially for low-M I fusion.

Whereas previously we found that fusion of high-MI features is favored because it can lower local surprisal, we find here that fusion of low-MI features is disfavored because it increases long-range surprisal. The curves of surprisal as a function of the amount of context t are shown in Figure 5. Here we see that although L _agg and L _fuse-low are equivalent in terms of local surprisal (at t = 0), they differ at t = 1, when one character of context is taken into account. Here, the surprisal for L _fuse-low is higher than the alternative languages.

Figure 5.

Average surprisal S_t under a (t + 1)-gram model, for the three languages of Table 5. Note that the lines often cover each other.

Figure 5. Long description

From left to right, three panels each display a line graph with surprisal in bits on the y-axis from 0 to 2.5 and amount of context t on the x-axis from 0 to 3. The left panel is labeled M I equals 0.05 bits, the center M I equals 0.19 bits, and the right M I equals 0.46 bits. In all panels, surprisal drops steeply from about 2.5 to 1.0 as context increases from 0 to 1, then remains flat. Each panel shows three lines: a solid line for agglutinative, a dashed line for fusing high-M I features, and a dash-dotted line for fusing low-M I features. The lines in each panel closely overlap. The legend at the bottom identifies line styles for each language type.

Intuitively, the long-range surprisal for L _fuse-low is high because in order to predict the character Y ₃, one must know the value of the underlying feature X ₂, but this cannot be reconstructed from only the previous character Y ₂. In order to maximally reduce the surprisal of the character Y ₃, one must expend more memory to look at both Y ₁ and Y ₂. Essentially, fusion of these independent features imposes a memory burden: more memory is required to make accurate predictions because features cannot be recovered from small amounts of context. In Appendix Section A2, we complement the simulation-based argument above with a formal argument showing that fusion is generally detrimental to long-range surprisal.

3.3. Advantage of category clustering: lowering local surprisal

Category clustering is a basic property of morphological systems where affixes are arranged in ‘slots’—for example, a morpheme indicating that a verb is first person will typically appear in the same position as a (mutually exclusive) morpheme indicating that it is second person (Mansfield et al. Reference Mansfield, Stoll and Bickel2020). Here, we demonstrate that category clustering makes the memory–surprisal tradeoff more favorable because it decreases local surprisal.

We simulated two languages, L _clustered with category clustering and L _nonclustered without. In these languages, two binary input features X ₁ (which could represent, for example, a voice feature with values active or passive) and X ₂ (which could represent, for example, a tense feature with values past or present) are expressed in strings drawn from the vocabulary {A,B,C,D} (see Table 6). In the version without category clustering L _nonclustered, the voice feature X ₁ is ordered before tense in two of the four combinations, and after it in the others. In the version with category clustering L _clustered, the tense feature X ₂ consistently followed voice X ₁. We assigned equal frequency to all four forms.

Table 6.

Feature-to-morpheme mappings for language L _clustered (with category clustering) and language L _nonclustered (without category clustering). In language L _clustered, the voice feature always precedes the tense feature; in language L _nonclustered, there is no way to predict the order of categories based on one feature alone.

Table 6. Long description

On the left panel, language L sub clustered displays a linear mapping where the voice feature consistently precedes the tense feature in morpheme order. Each morpheme is labeled, with arrows indicating the fixed sequence: voice first, then tense. On the right panel, language L sub nonclustered shows morpheme mappings without a predictable order; voice and tense features appear in varying positions, with arrows connecting features to morphemes in a non-linear, scattered arrangement. Category clustering is visually indicated in the left panel by grouped morphemes, while the right panel lacks such grouping. All labels and arrows are rendered in clear, contrasting colors to distinguish feature types and mapping paths.

Figure 6 shows that, while the tradeoffs are equivalent for low and very high memory capacities, the version with category clustering leads to overall more efficient tradeoffs. This happens because feature values are mutually exclusive, so the presence of one tense morpheme (similarly voice) excludes the presence of another tense morpheme. Thus, morphemes expressing mutually exclusive feature values are always informative about each other, and consistent ordering is more efficient. The formal argument explicating the result is found in Appendix Section A3.

Figure 6.

The memory–surprisal tradeoff favors category clustering. The dashed line is the tradeoff curve for the language L _clustered, which features category clustering, and the solid line is the tradeoff curve for the language L _nonclustered, which does not. We can see that the tradeoff for L _clustered is more efficient at high memory capacities, as the surprisal is lower.

3.4. Discussion

The simulations and arguments above demonstrate that the efficient tradeoff hypothesis introduced by Hahn, Degen, and Futrell (Reference Hahn, Degen and Futrell2021) can make predictions not only about word and morpheme order, but also about fusion and category clustering. Regarding fusion, an optimal tradeoff is found when features with high mutual information are expressed in a fusional way and features with low mutual information are expressed through agglutination. This happens because for two highly correlated features, it is useful to fuse them, because local surprisal would otherwise be high; whereas for two uncorrelated features, it is damaging to fuse them, because then more memory is required to make predictions about a third feature.

The overall prediction is that fusion of features with high mutual information is favored, and fusion of features with low mutual information is disfavored. Similarly, when deriving optimal orders for features, the best orders are those that place features with high mutual information close together (Hahn, Degen, & Futrell Reference Hahn, Degen and Futrell2021). Thus we have a link between proximity in optimal ordering under the ETH and fusion: we predict that features that are ordered close to each other in an optimal ordering will also be those features that are likely to be expressed fusionally. This generalization from order to fusion is reminiscent of Givón’s (Reference Givón and Haiman1985) Proximity Principle, which claims that elements that are related to each other conceptually in an utterance will be placed close together in terms of both time (order) and space (simultaneous expression, i.e. fusion).

4.1. Data sources

For the following studies, we estimated the memory–surprisal tradeoff curves for each language using the methods described in Section 2.2. In order to estimate the mutual information quantities required for calculation of the average surprisal bound S_M (equation 11), we extracted frequency data for feature bundles from Universal Dependencies Version 2 (Nivre et al. Reference Nivre, Bosco, Choi, de Marneffe, Dozat, Farkas, Foster, Ginter, Goldberg and Hajič2015).

4.2. Study 1: polyexponence

Across languages, polyexponence is more common for some groups of features than others. Here we focus on cases where two or three features are expressed together as one morpheme. We refer to these phenomena as double and triple exponence. In the double case, fusion is very common for person and number in verbs, and case and number in adjectives and nouns. The most prominent examples of triple exponence are the fusion of tense, aspect, and mood (TAM) markers and the fusion of person, number, and gender (PNG) markers in verbal paradigms (Bickel & Nichols Reference Bickel, Nichols, Dryer and Haspelmath2013). Assuming that the ETH is able to predict cases of fusion, we would then expect that these doubly and triply exponential features are adjacent (or at least close together) in the optimal ordering under the memory–surprisal tradeoff—that is, the morpheme order that leads to the minimum AUC.

Using a set of several languages with the relevant features, we evaluated this hypothesis by determining the optimal morpheme order for each language by minimizing the memory–surprisal AUC. We had twenty-three data points for person/number (Ancient Greek, Arabic, Armenian, Catalan, German, Greek, Hebrew, Hungarian, Icelandic, Italian, Latin, Latvian, Polish, Portuguese, Russian, Slovak, Slovenian, Spanish, Turkish, Ukrainian, Urdu, Uyghur, and Wolof), ten for case/number (Ancient Greek, Armenian, German, Greek, Icelandic, Latin, Latvian, Russian, Urdu, and Uyghur), fifteen for PNG (Ancient Greek, Arabic, German, Greek, Hebrew, Icelandic, Italian, Latin, Latvian, Polish, Russian, Slovak, Spanish, Ukrainian, and Urdu), and thirteen for TAM (Ancient Greek, Armenian, Greek, Hungarian, Latin, Polish, Russian, Slovak, Slovenian, Turkish, Ukrainian, Uyghur, and Wolof).

For the double exponence case, we calculated the distance in the optimal order between the two features being considered, and then normalized this by the number of feature categories in the language. This way, if the language has many features, we control for chance variation in the distance between features.

For the triple exponence case, instead of the distance between the features considered, we determined the standard deviation of their three ranks in optimal ordering. A low standard deviation indicates that the features are clustered together in optimal ordering; the lowest possible standard deviation would be 0.816, assuming the features are adjacent. We then normalized this value as above. We performed these procedures for all possible combinations of two and three features, limiting our data to those combinations that had at least seven languages represented.

Results are shown in Figures 7 and 8. All sets of doubly and triply exponential features had very low normalized standard deviations compared to random sets of three features, with p < 0.001 for person/number, case/number, TAM, and PNG by one-sample t-test. The results indicate that these features tend to cluster together in the optimal order, confirming the prediction of the ETH.

Figure 7.

Average normalized difference in position of two-feature combinations with greater than seven data points. Person/number and case/number are labeled; diamond indicates mean, and open circle indicates median. Error bars indicate 95% confidence interval.

Figure 8.

Normalized standard deviation in position of three-feature combinations with greater than seven data points. PNG and TAM are labeled; diamond indicates mean, and open circle indicates median. Error bars indicate 95% confidence interval.

Figure 8. Long description

The plot displays average separation in optimal ordering on the y-axis, ranging from 0.2 to 0.6. The violin plot is centered horizontally. At the lower region, a filled circle labeled TAM is positioned near 0.2. Slightly above, another filled circle labeled PNG is placed. At the center of the violin, a diamond symbol marks the mean, and an open circle above it marks the median. Horizontal error bars extend from the mean, indicating the 95 percent confidence interval. The violin plot is widest near the center and narrows at the top and bottom, reflecting the distribution of the data.

4.3. Study 2: suppletion

In addition to exponence, we consider suppletion, a phenomenon in which a given grammatical feature is expressed through unpredictable changes in the root, rather than affixation or introflection (Veselinova Reference Veselinova, Dryer and Haspelmath2013). For example, in English, the root go takes two forms depending on the tense: go in the future and present, and went in the past. Suppletion has been documented in several languages, but is usually quite rare and used for only a few roots (Bybee Reference Bybee1985, Markey Reference Markey1985, Aski Reference Aski1995, Fertig Reference Fertig1998). Across languages, some features are more likely than others to be involved in suppletion. Specifically, Moskal (Reference Moskal2015) finds that nominal suppletion is more likely to be triggered by number than by case. Here, we examine whether the ETH can predict this crosslinguistic trend across a sample of languages.

We argue that suppletion can be considered as fusion between the root and the feature being inflected for. Because the root changes form unpredictably based on the feature, we can say that the root and feature are being expressed jointly through fusion, as opposed to agglutination or nonconcatenative processes. Interestingly, Greenberg’s (Reference Greenberg and Greenberg1963) universal 39 states that ‘where morphemes of both number and case are present and both follow or both precede the noun base, the expression of number almost always comes between the noun base and the expression of case’. This indicates that even in nonsuppletive forms, case suffixes tend to come after number suffixes in linear order.

If we apply the ETH to this phenomenon, we should see that features that trigger suppletion (namely number) are closer to the root in optimal ordering than features that do not trigger suppletion (namely case), as determined by minimization of the area under the memory–surprisal curve. We again estimate the memory–surprisal curve for each language, as in Section 2.2, and use this method to determine the optimal ordering of feature categories. Then, we compare the rank of case and number to determine which tends to be closer to the root.

We examined seventeen languages, and of those, fifteen languages have number closer to the root than case in the optimal ordering. The languages that exhibited this pattern were Arabic, Armenian, Basque, Czech, Estonian, German, Greek, Finnish, Hungarian, Latin, Polish, Romanian, Slovak, Slovenian, Turkish; the languages that did not were Russian and Urdu. To validate the significance of this result, we ran a binomial mixed-effects model, with language family as a random intercept (with a total of five families: Afro-Asiatic, Basque, Indo-European, Turkic, and Uralic). Coding the number-before-case order as 1, we find a positive random intercept of size 2.01 with p < 0.01, corresponding to a probability of 0.88 that number is closer to the root crosslinguistically. We conclude that the ETH-optimal ordering is therefore consistent with naturally occurring patterns of suppletion in these languages. Note, however, that our conclusions are based on a small sample of languages, which are unevenly distributed across families (the majority being Indo-European), and as such it is possible that this effect does not hold in other languages or families.

4.4. Study 3: pairwise informational fusion

In Sections 4.2 and 4.3, we considered fusion as a binary choice: features are either fusional or agglutinative. However, in Section 2.1 we explained how fusion is better considered as a gradable measure. Here, we explore whether the memory–surprisal tradeoff can more generally predict degrees of morphological fusion in real languages. We use informational fusion as a graded empirical measure of the extent to which multiple features are expressed in a single morpheme. We show that degrees of fusion for pairs of features in four languages can be predicted by the ETH.

While general informational fusion φ(w) is concerned with the fusion of all features in a given surface form, we are primarily interested in pairs of features. Therefore, we modify the idea of informational fusion to define the pairwise informational fusion φ₂(w) for a form w that exhibits a pair of features f ₁, f ₂ ∈ σ.

(12) $$ {\unicode{x03C6}}_2(w)=-\mathrm{log}\hskip2pt p(w\hskip2pt |\hskip2pt {\mathcal{L}}_{-({f}_1,{f}_2)},\sigma, \ell ) $$

Then, for any given feature pair f ₁, f ₂, we determine the average φ₂(w) across all forms w that express f ₁, f ₂ to find a summary measure for the feature pair. This measure is indicated as $ {\overline{\unicode{x03C6}}}_2\left({f}_1,{f}_2\right) $ . Thus, we find an average pairwise fusion value for all feature pairs in ℒ. (Interestingly, the idea of pairwise fusion can be generalized for any n ≥ 2, where φ _n(w) is the informational fusion of n features.) Note that if there are only two categories that can be expressed (as is the case with Latin nouns, for example, which express number and case), pairwise fusion is the same as informational fusion. Below, we apply pairwise informational fusion to attested forms, not to any randomly reordered languages.

For example, consider the Spanish verb paradigm in Table 7. Examining the feature pair (impf, sg), we can see that all forms exhibiting both features are easily predictable from the rest of the paradigm as a result of regular and distinct morphological processes; for example, amabas ‘you loved’ (amar-2.sg.impf) is formed from the addition of ba before the person/number marker -s, a process that can be learned from the imperfect plural forms. The feature pair (impf, sg) receives a correspondingly low pairwise fusion of ~2.71 bits, as estimated by the sequence-to-sequence learner model. Meanwhile, it is difficult to predict the forms exhibiting the feature pair (1, pl): the key suffix -mos cannot be predicted on the basis of anything in the rest of the paradigm. We thus find a very large pairwise fusion of ~46.08 bits for the features (1, pl).

Table 7.

Indicative mood forms of the Spanish verb amar ‘to love’. Imperfect singular forms and first-person plural forms are in bold.

Table 7. Long description

From left to right, the first column lists tense abbreviations: prs for present, impf for imperfect, pret for preterite, fut for future, cond for conditional. The next three columns display singular forms labeled 1st, 2nd, and 3rd person; the following three columns show plural forms labeled 1st, 2nd, and 3rd person. Present tense forms are amo, amas, ama, amamos (bold), amáis, aman. Imperfect forms are amaba (bold), amabas (bold), amaba (bold), amábamos (bold), amabais, amaban. Preterite forms are amé, amaste, amó, amamos (bold), amasteis, amaron. Future forms are amaré, amarás, amará, amaremos (bold), amaréis, amarán. Conditional forms are amaría, amarías, amaría, amaríamos (bold), amaríais, amarían. Imperfect singular forms and all first-person plural forms are bolded.

Here, we wish to determine the relationship between the optimal order of morphemes under the ETH and the degree of fusion between pairs of features. More precisely, we predict that pairs of features with high informational fusion will not be far apart in optimal order: there will be an inverse correlation between the degree of fusion and the distance between features in optimal ordering.

4.4.1. Methods

Measuring informational fusion requires specification of a learner model that predicts unseen cells in a paradigm. For this learner model, we use an LSTM seq2seq model with attention (Sutskever et al. Reference Sutskever, Vinyals and Le2014, Bahdanau et al. Reference Bahdanau, Cho and Bengio2015, Kann & Schütze Reference Kann and Schütze2016). This is a generic neural-network architecture designed for estimating arbitrary stochastic functions from strings to strings. In our setting, the LSTM takes as input the feature combination σ, part-of-speech tag, and lemma ℓ (in characters), and produces the form w in characters as output. The input is represented as a string: for example, for a noun with σ = ⟨gen, pl⟩ and ℓ = serv-, the input string is s e r v n gen pl, and the target output string is s e r v ō r u m. We then estimate the surprisal of the form as:

(13) $$ -\log p(w\,|\,\ell, \sigma )=-\sum_t\log {p}_{\unicode{x03B8}}({w}_t\,|\,{w}_{<t},\ell, \sigma ), $$

where p _θ is the LSTM probability for character w_t given the input string and the previously generated characters w_<t. We use batch size 512, embedding dimension 128, and learning rate 0.001, and trained for ten passes through the training data with early stopping. We also used teacher-forcing to constrain the output space of the target. Models are not used in the analysis if the average cross-entropy loss on the final epoch exceeded 0.1.

The learner model is trained on paradigm data from UniMorph 3.0 (Sylak-Glassman et al. Reference Sylak-Glassman, Kirov, Yarowsky and Que2015, McCarthy et al. Reference McCarthy, Kirov, Grella, Nidhi, Xia, Gorman, Vylomova, Mielke, Nicolai and Silfverberg2020). This data is generally extracted from Wiktionary, an online open-source dictionary that contains data from several languages. For Arabic, we transliterate the UniMorph data into the Latin script using the ALA-LC standard (this is an implementation detail for our seq2seq model). In each language, for every possible pair of features f ₁, f ₂, we estimate their pairwise fusion using the sequence-to-sequence learner model. To calculate the fusion of a given feature combination (f ₁, f ₂), we average the pairwise informational fusion of all forms exhibiting those features in the data set.

We then compare pairwise fusion to the optimal ordering of morphemes as determined by the ETH. To do so, we calculate the ordering that would correspond to the lowest AUC (see Section 2.2). This ordering is on the level of individual features (see Table 8). We then convert this to the level of categories by taking the by-feature ranks and averaging them by category, and then reordering by these averaged ranks (see Table 9). This step controls for noise in the fine-grained order, as the languages we examined are known to exhibit category clustering.

Table 8.

Optimal ordering of features in Spanish as determined by minimization of the area under the memory–surprisal curve.

Table 8. Long description

The table has three columns: Category, Feature, and Rank. From top to bottom, the rows are as follows. VerbForm fin is ranked 0. Mood cond is 1, Mood imp is 2. Tense pst is 3, Tense impf is 4, Tense fut is 5, Tense prs is 6. Mood ind is 7, Mood subj is 8. Person 1st is 9, Person 3rd is 10, Person 2nd is 11. Number pl is 12, Number sg is 13. The order reflects minimization of the area under the memory–surprisal curve for Spanish.

Table 9.

Optimal ordering of categories in Spanish as determined by averaging the optimal ordering of Table 8 by category.

Table 9. Long description

The table consists of two columns. The left column is labeled Category and the right column is labeled Rank. From the top row downward, the categories are VerbForm with rank zero, Mood with rank one, Tense with rank two, Person with rank three, and Number with rank four. Each category is aligned left and each rank is centered. The ordering reflects the averaged optimal sequence for Spanish as determined by prior analysis.

With this optimal category ordering, we determine the difference in rank between pairs of features and compared this to their pairwise fusion. For example, if we were to compare the fusion of subj and sg in the Spanish rankings above, we would determine their pairwise fusion and compare this to the difference in rank between Mood and Number, that is, 3.

4.4.2. Results

We examine variation in fusion in the verbal paradigms of four languages: Arabic, Latin, Portuguese, and Spanish. We chose these languages in particular because they both satisfy restrictions on data and exhibit high levels of variation in fusion. Computation of both fusion and optimized ordering requires a large amount of annotated data, which presently exists for only a handful of languages. The corresponding Universal Dependencies treebanks were Arabic PADT, Latin PROIEL, Portuguese Bosque, and Spanish AnCora. In our estimation of the memory–surprisal tradeoff, we do not include participles, which are morphologically distinct from verbs in these languages despite being marked as verbs in Universal Dependencies.

Even if we have enough data, if a language does not exhibit substantial variation in fusion, there is little that the memory–surprisal tradeoff can explain. Indeed, we elect to use verbal paradigms over nominal paradigms precisely because they exhibit much more diversity in form: for example, Latin verbs tend to be highly fusional in the present tense (ama-mus) and highly agglutinative in the perfect (ama-ver-a-nt). We plot pairwise informational fusion values in Figure 9. Note that most paradigms are highly concentrated around a specific level of fusion. Meanwhile, many languages that do exhibit variation do not have enough data from Universal Dependencies to accurately estimate the memory–surprisal tradeoff curve. For this reason, we restrict this study to the languages listed above. This notably limits the generality of this set of results; we are ultimately testing three Romance languages and one Afro-Asiatic language.

Figure 9.

Pairwise informational fusion values for each language studied, grouped by part of speech; participles are not included in verbal form distributions. We observe that many paradigms have little to no variation to explain.

Figure 9. Long description

The chart consists of 36 vertical boxplots, each labeled along the x-axis with a language and part of speech, ordered from Arabic adjectives to Turkish nouns. The y-axis is labeled Fusion in bits, ranging from 0 to 50. Each boxplot displays the distribution of fusion values for that language and part of speech. Some boxplots, such as Arabic adjectives and Latin verbs, show high median fusion and wide interquartile ranges, while others, like Armenian adjectives and Polish nouns, have low medians and minimal spread. Outliers are present in several categories, and some boxplots are compressed near zero, indicating little to no variation. The overall trend reveals that fusion values and their variability differ substantially by both language and part of speech.

We compare informational fusion values for feature pairs to their difference in optimal order by means of a scatterplot. Figure 10 reveals that in all four languages, there is an ‘empty’ upper-right quadrant, meaning that no pair of features exhibits high fusion while simultaneously being far apart in optimal order. To further quantify this result, we test for significance using a nonparametric permutation test for the area under the Pareto curve (the curve shown by the black step curve in the plots) as in Cotterell et al. Reference Cotterell, Kirov, Hulden and Eisner2019. We can notate a given plot as a collection of points {(x ₁, y ₁), (x ₂, y ₂), …}; to run the permutation test, we randomly permute the y-values in this set, creating a new set {(x ₁, y _σ(1)), (x ₂, y _σ(2)), …}, where σ is a random shuffling transformation. We then determine the AUC for both the real curve and the stochastically permuted curve. We iterate this process 10,000 times and determine the p-value as the probability that the area under the empirical curve is greater than that of the random curve. We find p < 0.05 for all languages. Therefore, our results support the idea that the ETH can predict gradable morphological fusion.

Figure 10.

Tradeoffs between difference in rank and average fusion. Each point represents two features (f ₁, f ₂), plotting $ \left({\overline{\unicode{x03C6}}}_2\left({f}_1,{f}_2\right),R\left({f}_1,{f}_2\right)\right) $ . Step curve indicates Pareto curve. All tradeoffs are significant (p < 0.01) by permutation test for the area under the Pareto curve.

Figure 10. Long description

Panel a, top-left, labeled Arabic p less than 0.001, plots difference in rank on the y-axis from 0 to 6 and average fusion on the x-axis from 0 to 50. Black dots represent data points, and a stepwise Pareto curve connects the lowest points at each fusion value. Panel b, top-right, labeled Latin p less than 0.001, shows y-axis from 0 to 8 and x-axis from 0 to 50, with a similar distribution of dots and a stepwise Pareto curve. Panel c, bottom-left, labeled Portuguese p less than 0.005, has y-axis from 0 to 6 and x-axis from 0 to 50, with dots and a stepwise Pareto curve. Panel d, bottom-right, labeled Spanish p less than 0.05, has y-axis from 0 to 4 and x-axis from 0 to 50, with dots and a stepwise Pareto curve. In all panels, the step curve traces the lower envelope of the data, indicating the Pareto frontier. All tradeoffs are statistically significant as indicated by the p-values.

4.5. Summary and discussion

Examining crosslinguistic data, we found that we can use the memory–surprisal tradeoff to predict which features are often fused: in particular, features are fused when they are placed close together in the optimal ordering according to the memory–surprisal tradeoff. This idea was found to pick out features such as number and case for double exponence; tense, aspect, and mood for triple exponence; and number and the root for suppletion. Furthermore, the memory–surprisal tradeoff predicts the gradable degree of fusion for pairs of features as quantified using informational fusion. These results indicate broad support for the ETH as a factor that influences fusion crosslinguistically.

5.1. Generative approaches to morphology

In the generative literature, at least in Minimalist frameworks, morphology is typically considered as part of syntax (Kiparsky Reference Kiparsky1982, Lieber Reference Lieber1992, Halle & Marantz Reference Halle, Marantz, Hale and Keyser1993). In particular, in Halle and Marantz’s (Reference Halle, Marantz, Hale and Keyser1993) Distributed Morphology (DM), morphemes are treated as syntactic nodes. These morphemes then go through operations that result in complete words; in this respect, DM is a theory of word formation. Because morphemes are considered to be syntactic nodes, these operations are the same as those responsible for the realization of syntax. Thus, word formation is analogous to sentence formation. In this vein, Baker (Reference Baker1985) proposes the Mirror Principle, which, informally, argues that morpheme order reflects word order. However, the principle makes claims only about the underlying structure of words and sentences, before linearization through movement operations. The memory–surprisal tradeoff makes claims about the general ordering of units within language, and thus can be applied to both the morpheme and word level, therefore making it theoretically compatible with the mirror principle.

In the DM framework, morphological fusion reflects a merger of adjacent nodes (Halle & Marantz Reference Halle, Marantz, Hale and Keyser1993) (see also Sproat Reference Sproat1985, Marantz Reference Marantz, Hammond and Noonan1988, Noyer Reference Noyer1992). Thus, in DM, morphemes can be fused if they are adjacent at some point in the syntactic derivation. In terms of predicting which morphemes fuse, DM thus can primarily draw on the correspondence to order in syntactic derivations, so that elements may tend to be fused morphologically when they are adjacent in syntactic derivations. In this sense, DM predicts a correspondence between closeness in order and fusion, similar to what we have argued based on the ETH in this article. A difference is that the ETH, as a general theory of order in language, simultaneously predicts which features are likely to be close to each other/fused and provides a first-principles explanation of ultimately why pairs of features are ordered together in terms of processing efficiency.

An interesting difference between the ETH and generative approaches is the level of representation under consideration. The ETH aims to predict the final linearized surface order from only the sets of features realized in the surface form and the frequencies of those sets of features in usage. This is motivated by the fact that the ETH is grounded in theories of on-line language comprehension, and it is the surface order of morphemes that is the directly observable input of this process. Meanwhile, DM and other generative approaches model this final structure in terms of underlying orders and derivation processes. In such theories, constraints on possible paradigms are fundamentally constraints on derivations of surface forms, whereas the ETH directly constrains the surface forms. In this sense, the ETH is philosophically similar to Harmonic Grammar (Smolensky & Legendre Reference Smolensky and Legendre2006) and Optimality Theory (Prince & Smolensky Reference Prince and Smolensky1993) in that it is fundamentally about constraints on the output of derivational processes, not constraints on the structure of the derivational process itself.

5.2. Meaning-based theories of order and fusion

As formulated in Hahn, Degen, & Futrell Reference Hahn, Degen and Futrell2021, the ETH is a theory of morphological and syntactic order. Previous work (Bybee Reference Bybee1985, Rice Reference Rice2000) has argued that morpheme order is related to semantics. A prominent family of proposals argues that morpheme order is determined by the order in which the meanings of morphemes combine; that is, when the meaning of a morpheme has narrower scope, it will be ordered closer to the root (e.g. Rice Reference Rice2000, Caballero Reference Caballero2010, Korotkova & Lander Reference Korotkova and Lander2010, Narrog Reference Narrog2010). By referring to the order of semantic composition, this view is broadly compatible with approaches viewing morphology as a part of syntax, though it references the surface order, like the ETH. Scope-based theories predict order, and might be taken to predict that morphemes fuse when they are adjacent in their semantic scope, but otherwise do not seem to make predictions regarding fusion.

Another meaning-based theory, articulated by Bybee (Reference Bybee1985), posits that affix order is dictated by ‘semantic relevance’ with the root; that is, affixes that are more relevant to the semantics of the root are positioned closer to it. For example, tense-aspect-mood (TAM) markers are typically placed closer to the verb compared to, for example, agreement markers, since TAM alters a verb’s semantics to a greater degree than does agreement. Similar arguments can be made for other universal generalizations about the position of affixes relative to the root. For example, Greenberg’s universal 39 (Greenberg Reference Greenberg and Greenberg1963) states that number affixes are closer to the noun than case affixes. Using the idea of semantic relevance, we can argue that this is because number affixes change the referent of the noun, while case affixes alter only the noun’s syntactic position. Bybee (Reference Bybee1985:36–41) also applies relevance to explain patterns in fusion with the stem, noting that changes in verb stems are much more frequently related to aspect, rarely related to tense and mood, and almost never related to person and number of the subject. Conversely, the verb stem is more likely to impact the choice of aspect, voice, or valence allomorphs than the choice of person or number allomorphs. This ordering is in agreement with the relevance-based ordering of affix morphemes that she posited.

We believe that the theory we present here captures a similar intuition to Bybee’s Relevance Principle, but where ‘relevance’ is operationalized as mutual information (see also Culbertson et al. Reference Culbertson, Schouwstra and Kirby2020). Unlike the idea of relevance, which is somewhat subjective, mutual information is a quantity that can be estimated from corpus data straightforwardly.

A second related idea is that combinations of elements that are highly frequent in language use come to be chunked and ultimately grammaticalize (e.g. Bybee Reference Bybee2006a,Reference Bybeeb, Bybee & Napoleão de Souza Reference Bybee, de Souza, Aleksandar and Łukasz2021), leading to opaque or suppletive forms (Hay Reference Hay2001, Bybee Reference Bybee2006b). For the organization of morphological paradigms, this idea makes the prediction that feature pairs that often cooccur may come to be expressed in ways that cannot be reduced to expressions for the individual features. Indeed, this conceptually relates to the predictions made by the ETH, as feature categories have high mutual information precisely when specific pairs of feature values cooccur more frequently than one would expect based on their individual frequencies (equation 3).