5.1 Translation quality

We first look at a number of metrics that are used to assess general MT quality. We calculate the widely used lexical metrics BLEU (Papineni et al. Reference Papineni, Roukos, Ward and Zhu2002), TER (Snover et al. Reference Snover, Dorr, Schwartz, Micciulla and Makhoul2006), and chrF (Popović Reference Popović2015) using the sacreBLEU (Post Reference Post2018) implementation and, following the advice of Freitag et al. (Reference Freitag, Rei, Mathur, kiu Lo, Stewart, Avramidis, Kocmi, Foster, Lavie and Martins2022), also report on two neural-based metrics, namely, BLEURT (Sellam, Das, and Parikh Reference Sellam, Das and Parikh2020) (specifically, implementation BLEURT-20-D12, a 12-layer distilled model that is $\sim 3.5X$ smaller and 1 order of magnitude faster than the original BLEURT-20) and COMET (Rei et al. Reference Rei, Stewart, Farinha and Lavie2020) (specifically, the default model WMT $_{2}$ 0-COMET-DA), which are reported to correlate better with human judgements.

BLEU considers word-level precision with regard to a reference translation with a penalty for brevity, while chrF works at character n-gram level. In turn, TER, also working at word level, is an edit-distance metric that calculates the minimum number of edits (additions, deletions, substitutions, and shifts) required to transform the MT output into the reference sentence. From a completely different perspective, BLEURT is a pre-trained evaluation model that uses the BERT language model (Devlin et al. Reference Devlin, Chang, Lee and Toutanova2019), synthetic data to represent a diverse range of lexical, syntactic, and semantic dissimilarities and a set of human evaluations. COMET is also a reference-based neural regression model built on top of XLM-R (large) (Conneau et al. Reference Conneau, Khandelwal, Goyal, Chaudhary, Wenzek, Guzmán, Grave, Ott, Zettlemoyer and Stoyanov2020) and trained on human judgments of translation quality. As can be noted, the chosen metrics cover a considerable set of approaches, which serves to monitor the level of quality from different perspectives for sounder conclusions.

For the outputs of the baseline MT and each of the following generations, we calculate scores using the target side of the test set as a reference, which represents the traits of the original Spanish, and also the source in the case of COMET. Results gathered in Table 2 show that neural metrics and BLEU point to the same tendency: as the MT generations advance, the scores drop slightly and rather steadily.Footnote ^c This implies that the output of the systems is more and more different from the reference each time. chrF seems to contradict this. However, the character-based nature of this metric might be having an effect here. As we will see in Section 5.2, the number of types used by each MT generation decreases, in particular, types with low frequencies, which may contain character sequences that are rarer. As a result, the text might be getting more and more homogeneous also in terms of character sequences. Therefore, the quantity of character sequences that are correct (more standard) can be higher in each generation, while at the same time, the text being more and more different from the reference translation. We see another exception with TER, which indicates that, albeit minimally, fewer changes are needed to transform the MT outputs into the reference as the generations advance. Further qualitative research would be necessary at this point in order to understand this behaviour, although the homogenisation effect attributed to MT might be at play again. With regard to the level of quality, it should be noted that absolute numbers seem to point to a rather strong baseline and strong subsequent generations.

Table 2.

Global quality automatic scores as reported by BLEU, TER, chrF, BLEURT, and COMET, where the output of each MT generation (Gen.), MT $_{0}$ to MT $_{10}$ , is compared against the Spanish side of the test set. Scores are provided within a 0–100 range, with the best-scoring MT generation for each metric in bold. The $\downarrow$ symbol indicates that lower values of the metric correspond to better performance

5.2 Lexical richness and density

To study linguistic diversity, we first report type and token information and then calculate four widely used metrics, specifically, type/token ratio (TTR) (Templin Reference Templin1957), Yule’s K (Yule Reference Yule1944a), and the measure of textual lexical diversity (MTLD) (McCarthy and Jarvis Reference McCarthy and Jarvis2010), followed by lexical density (LD) (Toral Reference Toral2019).

The assumption behind these lexical diversity indices, which are based on type (unique words) and token (all instances of the types) distributions and ratios, is that the higher the number of different words in a text, the richer the text is and, as a consequence, better and requiring a higher language competence to produce it. TTR is the simplest of the metrics. It is calculated by dividing the number of types in a text by the number of tokens.

Yule’s K focuses on the frequency distribution of words in a text. It measures how constant a text is in terms of the appearance of new words; that is, it aims to capture how often words repeat and to what extent the vocabulary is spread across different frequency levels. It is calculated as follows:

(1) \begin{equation} K=10^4 \cdot \left (\frac {\sum _{i=1}^{\infty } i^2 f_i-N}{N^2}\right ) \end{equation}

where $\sum _{i=1}^{\infty } i^2 f_i$ is the sum of the squares of the frequencies of each word and $N$ is the total number of words in the text.

Lower values of Yule’s K indicate a richer vocabulary, meaning the text uses a wider variety of words and has less repetition. Conversely, higher values point to a less rich vocabulary, meaning the text has more repetition of the same words.

In turn, MTLD calculates the average length of word sequences in a text that maintain a given level of lexical diversity. It does this by setting a threshold for the diversity and splitting the text into segments based on when this threshold is crossed (we use the default value of .720 TTR). The text is read word by word, and the TTR is calculated cumulatively. When the TTR falls below the threshold of 0.72, a segment is marked, and the calculation restarts from the next word. Next, the average length of all segments is calculated. This average length is the MTLD score, indicating how many words, on average, can be read before the lexical diversity falls below the threshold. Higher MTLD values indicate greater lexical diversity, as it takes more words to reach the threshold, while lower MTLD values indicate lower lexical diversity, as the threshold is reached more quickly.

According to McCarthy and Jarvis (Reference McCarthy and Jarvis2010), lexical diversity indices suffer from two key problems: their sensitivity to text length and the assumption of textual homogeneity. The issue with text length lies in the fact that tokens increase linearly as the text expands, whereas types, that is, the number of different words, gradually decrease given the degree of repeated tokens that is necessary to maintain a meaningful and understandable development of a narrative. In fact, some researchers have taken this as an opportunity to account for the representativeness of a topic in a text or corpus (Morse Reference Morse1995; McEnery Reference McEnery2003). In reference to the assumption of homogeneity, the authors point that the level of diversity required by specific rhetorical purposes and strategies is different, and therefore, an unsteady distribution of types might not necessarily reflect weaker text production skills.

These vulnerabilities lead us to consider their appropriateness for our experimental setup. First, the metrics are not usually tested on texts as large as our test set, and as we mentioned, all metrics are dependent on length to a higher or lower degree. Second, our test set consists of isolated sentences instead of internally coherent full-length texts with a narrative thread that reflects natural type/token distributions.

The previous notwithstanding, we concluded that the two issues discussed above do not seem crucial in our experiments because of the static nature of our test set (we use the same 1 million English sentences to obtain the Spanish translation of the different MT generations) and because our aim is to consider relative changes across generations. However, to reduce the effect the mentioned issues might have, we calculate the scores by applying a batch approach as suggested by Hess et al. (Reference Hess, Sefton and Landry1986) and McCarthy and Jarvis (Reference McCarthy and Jarvis2007), among others. Concretely, we have divided the test set into batches of 1,000 sentences and calculated all metrics for each one of them and report their averages.Footnote ^d

The results of the different metrics are displayed in Table 3. Let us focus on the type and token counts first. The former inform about the number of different words used throughout the texts, while the latter account for all instances of such words. These are indicative of the breadth of the vocabulary used, its frequency, and also text length. The original Spanish text, consisting of 1 million sentences, contains 17,345,663 tokens and 299,158 types. The counts for both tokens and types decrease in the Spanish translation produced by the baseline MT (MT generation 0, MT $_{0}$ ). Tokens are reduced by 1.41%, meaning that the length of the text produced by MT is slightly shorter than in the original. The drop in types is more considerable as 17.15% do not appear in the Spanish translation with respect to the original version.

Table 3.

Lexical diversity and density scores as reported by type and token counts, type-token ratio (TTR), measure of textual lexical diversity (MTLD), Yule’s K, and lexical density (LD) for the Spanish side of the test set and the output of each MT generation, MT $_{0}$ to MT $_{10}$ (Gen.), and where the generation with the highest diversity for each metric is in bold. The $\downarrow$ symbol indicates that lower values of the metric correspond to better performance

The subsequent MT generations also see a tendency of reduced tokens and types (0.03% and 0.94%, respectively, for MT $_{1}$ with reference to MT $_{0}$ ), but the drop is substantially more limited, with even a couple of exceptions where the counts increase. In the case of types, we see a maximum drop of 2,339 units (MT $_{1}$ ) and a minimum of 98 (MT $_{9}$ ). Note that MT $_{10}$ sees a small increase of 68 types. In turn, the number of tokens also tends to shrink, but between a maximum of 31,518 words (MT $_{2}$ ) and a minimum of 4,723 words (MT $_{1}$ ). We observe two cases (MT $_{4}$ and MT $_{9}$ ), where the token count increases, even when the types decrease.

Overall, these counts seem to indicate that there is certain diversity or language traits that the MT system cannot learn and/or reproduce, but once the traits of the MT language are determined, the loss is drastically reduced going forward. The Spanish with MT traits is, unsurprisingly, more suitable for the subsequent engines to imitate.

Let us now turn to lexical diversity indices. Results show inconclusive trends: all three metrics, TTR, MTLD, and Yule’s K, show a drop from the original to the baseline MT (MT $_{0}$ )—and an increase in Yule’s K, given that it accounts for uniformity, and therefore, it is a lower score that indicates a higher diversity (see Table 3). From the baseline MT onward, TTR increases slightly (MT $_{0}$ to MT $_{1}$ ) but stabilises early. MTLD scores also stay within the 68–71 range, even when the fluctuation is more varied (it increases until MT $_{3}$ and then decreases until MT $_{6}$ ; it goes upward for MT $_{7}$ and then down again for MT $_{7}$ –MT $_{10}$ ). In contrast to the previous metrics, Yule’s K shows a rather steady increase in diversity.

We calculated one final metric for this linguistic level: LD. It is calculated as the ratio of content words (nouns, proper nouns, verbs, adjectives, and adverbs) to total words (tokens) and can be taken as an approximation of the informativeness of a text (Toral Reference Toral2019). The higher the number of content words, the more information a text is supposed to include. A higher ratio indicates that the text is, indeed, more informative but also that it becomes more and more compact in the sense that fewer function words are used to arrange the content words within the sentence. The results in Table 3 show an apparent slight increase in LD, indicating that, in proportion, the occurrences of function words are actually decreasing as MT generations advance.

Figure 2.

Normalised token counts for content words across MT generations, MT₀ to MT $_{10}$ , where NOUN refers to nouns, PROPN to proper nouns, VERB to lexical verbs, ADJ to adjectives, and ADV to adverbs.

Figure 3.

Normalised token counts for function words across MT generations, MT $_{0}$ to MT $_{10}$ , where DET refers to determiners, PRON to pronouns, ADP to prepositions, AUX to auxiliary verbs, CCONJ to coordinating conjunctions, and SCONJ to subordinating conjunctions.

What lexical metrics seem to reveal is that there is a tendency to use a reduced vocabulary as MT generations advance. We further inspected this trend by focusing on POS categories. We tagged the original Spanish text and the MT proposals with Universal POS tagsFootnote ^eE using spaCyFootnote ^f model es_core_news_sm-3.7.0. We then counted the occurrences of each category for the original test set and the output of the MT baseline and generations. This allows us to observe whether the vocabulary reduction happens across all grammatical categories similarly or whether there are certain categories that suffer more.

Figures 2 and 3 show the progression of the content words and the function words, respectively. We normalised the results by dividing the counts for each MT output by the counts of the original test. If we look at content words, we see that except for proper nouns (PROPN), all categories tend to increase, albeit slightly. We see certain nuances when paying attention to each of the categories. Nouns (NOUN) undergo a rather steady increase starting from MT $_{0}$ . And so seems the case of verbs (VERB) from MT $_{0}$ and MT $_{2}$ , but it seems to level from then onwards. What is interesting about verbs is that this category suffers a drop from the original text to MT $_{0}$ before it starts picking up, and ten generations later, it is still behind original counts. Proper nouns experience the same drop, which is even sharper and continues in this direction for a few generations before it starts increasing slowly. Interestingly, adjectives (ADJ) and adverbs (ADV) are more frequent in the machine-translated texts, and adjectives, in particular, increase as generations advance.

If we turn to closed categories, that is, function words, we observe that all categories suffer an initial more pronounced drop from the original text to MT $_{0}$ and then decrease at different rates. Prepositions (ADP) and coordinating conjunctions (CCONJ) vary slightly, while determiners (DET) and pronouns (PRON) do so more sharply. The progression of subordinating conjunctions (SCONJ) is more uneven. Even when it never reaches original occurrences, it fluctuates as MT generations advance. One exception to the downward trend is that of auxiliaries (AUX), that is, the elements that accompany lexical verbs to express grammatical information such as person, number, tense, mood, aspect, voice, or evidentiality. They sustain an increase from the original text to MT $_{0}$ and then start decreasing. After ten MT generations, their occurrence rate is still higher than in the original. A possible reason for this increase might be that MT systems might have used verb phrases requiring auxiliaries more frequently compared to the original Spanish.

From the review of POS counts, we can conclude that all open categories except proper nouns increase as the MT generations advance, and function words tend to decrease. This might indicate that the language becomes increasingly compact, which corroborates the LD results in Table 3. This can happen either by favouring syntactic constructions that compress content words and require fewer function words, for example, noun clusters over complements, which include prepositions and determiners for each of the elements involved. It can also mean that language becomes more explicit and lacking referencing by means of pronouns. Of course, the higher density of content words can also be due to incorrect clusterings and omissions of compulsory function words. Yet, each category is affected differently, which calls for a more qualitative study to fully understand the in-category changes.

Token-based counts hide the diversity of the vocabulary in terms of the actual number of different words that are used. Therefore, in order to check whether all POS categories follow the same trends as overall token counts, we investigated word counts further and looked into the counts of types for words grouped by POS tag. Results are displayed in Figures 4 and 5 for content words and function words, respectively. As before, we normalised the results by dividing the counts for each MT output by the counts of the original test set.

Figure 4.

Normalised type counts for content words across MT generations, MT $_{0}$ to MT $_{10}$ , where NOUN refers to nouns, PROPN to proper nouns, VERB to lexical verbs, ADJ to adjectives, and ADV to adverbs.

Figure 5.

Normalised type counts for function words across MT generations, MT $_{0}$ to MT $_{10}$ , where DET refers to determiners, PRON to pronouns, ADP to prepositions, AUX to auxiliary verbs, CCONJ to coordinating conjunctions, and SCONJ to subordinating conjunctions.

The first thing to notice is that the loss of types, which happens much more markedly from the original test set to the MT $_{0}$ baseline, as we already saw in Table 3, occurs across all POS categories. All categories decrease between 10% and 30%. Specifically, in the case of open categories, except for proper nouns, they all decrease between 20% and 28%, and in the case of closed categories between 12% and 29%. The trend for open categories seems to show that noun, verb, and adjective types are reduced very slightly after the initial drop at MT $_{0}$ towards a point of stability. Adverbs drop slightly lower with the baseline MT $_{0}$ but then increase slightly until they reach a very similar range as the other categories.

If we combine these trends with those of the token counts (see Figure 2), we can say that in the case of nouns, adjectives, and adverbs, even when the texts display a continuous slightly higher number of occurrences across generations, there is an initial loss of types with the MT $_{0}$ baseline, but after that, their number remains rather steady. This means that those types that have got through MT $_{0}$ will appear more frequently than in the original test set. The verbs also follow this trend, but we did observe a slight initial drop in token counts too. We see a rather unexpected behaviour with proper nouns, as the frequency of tokens decreases up until MT $_{4}$ while types undergo an initial upward trend and only stabilise after MT $_{4}$ . This could indicate that MT leads to proper nouns being used less frequently while they become more diverse. Further exploration would be necessary to find out how specifically MT handles this word category.

In reference to closed categories, we observe that the type counts for coordinated conjunctions decrease gradually, while the remaining categories stay within a certain range with more marked fluctuations from generation to generation. Remember that in terms of token counts (see Figure 3), we observed that coordinated conjunctions stayed rather stable, which means that certain types appear more and more often and others disappear as generations advance. In the case of pronouns and determiners, which saw a steady drop, we see that there is some stability with regard to types. Therefore, these categories disappear as syntactic elements but not in terms of variety. Auxiliaries undergo an increase in overall occurrences, but their types decrease from the very beginning and there seems to be a tendency towards their reduction, although this is not constant.

We have observed that to a certain degree, all type categories decrease, especially when going through the MT $_{0}$ baseline. In order to learn whether the frequency of occurrences for the types changes as MT generations advance, we calculate the number of types across six frequency bands: Band 1 includes the number of words that appear over 100,000 times, Band 2 99,999–10,000 times, Band 3 9,999–1,000 times, Band 4 999–100 times, Band 5 99–10 times, and Band 6 9–1 times. The results, displayed in Table 4, clearly show that the words that have a higher risk to disappear from the language are those in the lowest frequency band. This is in line with the results by Vanmassenhove et al. (Reference Vanmassenhove, Shterionov and Way2019). What is more, it seems that types remain rather stable with frequencies of over 1,000 and some slight loss is observed for those with 100 to 1,000 occurrences. Band 5 with 10–99 occurrences displays a contrasting trend as it shows a higher occurrence for the baseline MT $_{0}$ with respect the original language. This then decreases and increases slightly seemingly in a randomly manner, yet never as low as the original. It seems to pick an upward trend after MT $_{5}$ , even when, within our range of generations, we do not see it surpass the values of MT $_{0}$ . The obvious decline is sustained by the types with frequencies below 10. Note that in this band, it is the drop between the original test and the baseline MT $_{0}$ that suffers considerably (a drop of over 45,000 types), but then, the decline is markedly slower.

Table 4.

Counts of types across frequency bands for the Spanish side of the test set and the output of each MT generation, MT $_{0}$ to MT $_{10}$ (Gen.), where Band 1 refers to words that appear over 100,000 times, Band 2 99,999–10,000 times, Band 3 9,999–1,000 times, Band 4 999–100 times, Band 5 99–10 times, and Band 6 9–1 times

5.3 Morphological richness

To account for morphological variety, we consider the inflexional diversity of lemmas. We follow the work of Vanmassenhove et al. (Reference Vanmassenhove, Shterionov and Gwilliam2021), where the authors adopt the Shannon entropy (Shannon Reference Shannon1948) and Simpson’s diversity index (Simpson Reference Simpson1949) to compute the entropy of the inflexional paradigms of lemmas. This could account for the richness of diversity.Footnote ^g The authors claim that, in line with the conceptualisation of Shannon, the entropy of the inflexional paradigm of a specific lemma could be computed by taking the base frequency of that lemma (frequency of all word forms associated with that lemma) and the probabilities of all the word forms within the inflexional paradigm of that particular lemma as shown in Equation 2:

(2) \begin{equation} p(w f \mid \unicode{x1D4C1})=\frac {\operatorname {count}(w f)}{\sum _{w f^{*} \in l} \operatorname {count}\left (w f^{*}\right )} \end{equation}

where $p(w f \mid \unicode{x1D4C1})$ is computed as the fraction of the counts of the wordform, $\operatorname {count}(w f)$ , to the count of all wordforms for the lemma $\unicode{x1D4C1}$ and $\in$ indicates the wordforms of a given lemma $\unicode{x1D4C1}$ .

Based on this idea, Shannon entropy is calculated as per Equation 3:

(3) \begin{equation} H(\unicode{x1D4C1})=-\sum _{w f \in \unicode{x1D4C1}} p(w f \mid \unicode{x1D4C1}) \log p(w f \mid \unicode{x1D4C1}) \end{equation}

where $H(\unicode{x1D4C1})$ denotes the entropy of the lemma $\unicode{x1D4C1}$ .

This idea can also be followed for Simpson’s diversity index ( $D$ ), which accounts for the evenness of the diversity, and it is calculated as per Equation 4:

(4) \begin{equation} D(\unicode{x1D4C1})=\frac {1}{\sum _{w f \in \ell } p(w f \mid \unicode{x1D4C1})^{2}} \end{equation}

We then calculate the average Shannon entropy and Simpson’s diversity index for all lemmas $\unicode{x1D4C1}$ to obtain a representative score for the Spanish side of the original test set and each MT generation. Both metrics serve as approximations of morphological diversity, with higher values indicating greater diversity and lower values indicating less diversity.

The scores of both metrics usually range between 0 and 1, but for ease of readability, we multiply the scores presented in Table 5 by 100 to present them in the range of [0–100]. We can observe that according to these two metrics, similar to what we observed with lexical metrics, there seems to be a drop in diversity for MT $_{0}$ compared to the original Spanish (represented as a lower score for Shannon entropy and a higher score for Simpson’s diversity index). Then, the MT traits across generations are very stable. This might indicate that while a degree of morphological reduction seems to be occurring when the language goes through the first MT cycle, the grammatical information may not be suffering any further considerable loss as generations progress.

Table 5.

Morphological variety scores as reported by Shannon Entropy and Simpson’s Diversity Index for the Spanish side of the test set and the output of each MT generation, MT $_{0}$ to MT $_{10}$ (Gen.), where the generation with the highest diversity for each metric appears in bold. The $\downarrow$ symbol indicates that lower values of the metric correspond to better performance

5.4 Structural similarity

In this section, we study to what extent the language of each of the MT generations has the same structural traits as the original Spanish. To do so, we explore the results of two metrics, namely, perplexity and posTER (Marchisio et al. Reference Marchisio, Freitag and Grangier2022).

To start, let us consider perplexity. We first build a language model, both at the word level and at POS level, trained using 8 million sentences from the original Spanish corpus, thus representing unprocessed language, which coincides with the batch used to train the baseline MT system. To train our neural model, we follow the architecture used to build GPT-2 (Radford et al. Reference Radford, Wu, Child, Luan, Amodei and Sutskever2019), which is basically a decoder-only transformer. We then score the capacity of the LM to predict the language of the original Spanish test set and the output of the baseline MT and successive MT generations. The lower the perplexity, the better the LM is at predicting the language in the test sets, and therefore, the better it captures the features of the original language.

Results in Table 6 show that, both at word level and at POS level, MT $_{0}$ sees an improvement with respect to the original language, and then it degrades slowly and steadily. In ten MT generations, the perplexity does not reach the level of uncertainty it displays with the original language. The fact that the Spanish of the original test set—same language generation with which the LM has been trained—is more difficult to predict for the LM than the subsequent MT generation outputs seems counter-intuitive at first glance.

Table 6.

Structural similarity scores as reported by the perplexity measure and posTER metric, for the Spanish side of the test set and the output of each MT generation, MT $_{0}$ to MT $_{10}$ (Gen.), where P:ES represents the perplexity of the the Spanish LM at the word level, P:ES-POS represents perplexity of the Spanish POS LM, and P:EN-POS represents the perplexity of the English POS LM. The $\downarrow$ symbol indicates that lower values of the metric correspond to better performance. The generation with the best scores for each metric appears in bold

We hypothesise that the naturally occurring Spanish is more difficult to predict maybe due to its higher diversity. Then, when the language goes through the MT architecture for the first time, it might create a less diverse and more homogeneous language, which might be easier to predict by the LM. The trend of the metric in reference to the MT generations is not surprising if we assume that a level of degradation, which deviates from the original Spanish, is introduced by each system.

We also trained an English LM at the POS level to test whether the distortion we observe across MT generations is due to the impact of the source language to some extent; that is, the Spanish output of the MT generations is acquiring traits of the English source through interference. To start, results clearly show that the English LM is worse at predicting Spanish than the Spanish LM, which is a good sign. Next, we also observe that the behaviour of the metric shows the same trend as with the Spanish LMs, that is, the LM is better at predicting machine-translated Spanish than the original Spanish. The perplexity increases gradually for the subsequent MT generations, indicating that it is increasingly difficult for the English LM to predict the POS sequencing of the Spanish outputs. We can derive that the Spanish language is changing across generations and that such change is not identified as resulting in a POS sequencing that resembles English patterns more closely regardless of the level of interference that might (or might not) be happening. Therefore, the distortion introduced by the MT generations cannot be associated to source language interference without further analysis.

Let us focus on the second metric. posTER was proposed by Marchisio et al. (Reference Marchisio, Freitag and Grangier2022) as a measure to calculate structural similarity at POS level. The authors calculate the edit distance between translations to account for differences between versions. We apply this measure by using the POS-tagged, original Spanish test set as a reference and the translations of each of the MT systems as hypotheses.

Results in Table 6 show rather gradual yet minimal distortion according to this metric. From MT $_{0}$ to MT $_{10}$ , which represent the lowest and highest posTER scores, the difference in the percentage of POS-level changes would be 1.23. Yet, as the authors (Marchisio et al. Reference Marchisio, Freitag and Grangier2022) themselves suggest, the seventeen POS categories used by the Universal Dependency tag set might be too coarse to identify more subtle changes and conceive actual deviations.