2.1.1. Transparency and reproducibility

Using commercial MT services, researchers have no control over the specific version of the model used for translation, as they are “closed source.” This prevents others from replicating their results later because the MT system used originally may have changed in the meantime (Chan et al., Reference Chan, Zeng, Wessler, Jungblut, Welbers, Bajjalieh, van Atteveldt and Althaus2020). This problem does not arise when using open-source MT models. They are typically versioned and available from publicly accessible platforms or repositories, such as the Hugging Face Model hub.Footnote ⁵ Researchers can therefore document the exact version of the model they have used, making research using open-source MT models reproducible.

Second, open-source MT models are transparent. The research teams providing them typically document the parallel corpora and model architecture used to train their models. Furthermore, it is a well-established best practice to report model performance in predefined test sets (benchmarks). This enables researchers to make informed decisions about which MT model to use in a specific application (cf. Licht and Lind, Reference Licht and Lind2023), for example, by assessing the reliability of the available models in the languages they want to translate.Footnote ⁶ In contrast, the information available about the performance of commercial services’ MT models often does not meet scientific standards, nor is it transparent what data has been used for training.

2.1.2. Cost efficiency and resource requirements

Using commercial services to machine-translate large text corpora can be expensive, as one pays for each translated character. For example, Google charges users of their Cloud Translation service U.S. $ 20 per 1 million characters, often implying three to four-digit sums for the translation.Footnote ⁷

Accordingly, researchers’ reliance on commercial services creates an undesirable barrier for those with limited budgets (Baden et al., Reference Baden, Pipal, Schoonvelde and van der Velden2022). Open-source MT models can lower this barrier. They are freely available for download and use, and researchers thus do not have to pay translation fees. Instead, the only financial cost arising when using open-source MT models results from the energy consumed for computing and, if necessary, from using cloud servers. Overall, this cost efficiency makes open-source MT models an attractive option for researchers on a tight budget and may even help level the playing field for smaller research teams.

Table 1 provides a cost comparison between two popular commercial MT services (Google Translate and DeepL) and a popular open-source MT model (M2M 418 M). The two main costs that MT incurs are money and time. For commercial translation services (APIs), financial costs are calculated on a per-character basis. For open-source models, the costs are based on GPU costs, which are easily accessible through services like Google Colab.Footnote ⁸ Our cost estimates show that, financially, using open-source models is significantly cheaper than commercial services. Translating 18 million characters costs more than EUR 300 via an API and less than EUR 3 with an open-source model.

Table 1.

Comparison of costs in Euro and compute time arising when translating a fixed amount of text with commercial services or the M2M open-source MT model

^a We disregard the time elapsed for sending API requests.

^b Estimates based on relative efficiency relative to A100 GPU.

However, using open-source MT models requires more computation time and expertise. Especially on older GPUs, translations with open-source models can take many hours to complete. Moreover, running the required software on a GPU requires some additional expertise (e.g., moderate Python programming skills). To tilt the balance further in favor of the open-source approach, we thus provide an interactive Google Colab-based online translation application, code base, and tutorial.Footnote ⁹ We have designed our app to ease the use of open-source MT models for researchers with no Python programming skills. Our accompanying tutorial and code base provide researchers with basic Python programming experience with a template that they can adapt to their needs. Given that the level of technical prowess in the (computational) social science community is steadily increasing (cf. Baden et al., Reference Baden, Pipal, Schoonvelde and van der Velden2022), these resources should lower the above-discussed barriers to using open-source MT models.

2.1.3. Advantage over generative LLMs

Although not a point of comparison in this paper, we find it worth pointing out that open-source MT models also compare favorably in terms of cost and resource efficiency to another alternative: using generative large language models (LLMs) for translation. LLMs such as GPT, LlaMa 3, & Co. can be prompted to follow specific task instructions, including translation between languages (cf. Lyu et al., Reference Lyu, Du, Xu, Duan, Wu, Lynn, Aji, Wong and Wang2024). Furthermore, similar to open-source MT models, many LLMs are “open-weights”Footnote ¹⁰ and thus also facilitate transparent and reproducible research. However, using LLMs for MT compares negatively to using the open-source MT models we consider in terms of resource requirements. Competitive LLMs have many more parameters than the open-source MT models we consider. For example, even the relatively large M2M model has only 1.2 billion parameters compared to the 32 to 70 billion parameters of medium-sized multilingual LLMs. This means that processing and generating a token with an LLM expends much more energy—and thus emits more CO₂—than using a specialized MT model. The size of LLMs, moreover, likely makes local deployment infeasible for most applied researchers, which means that they have to fall back to using a commercial provider’s API or hardware.

3.1.1. Empirical strategy

Like de Vries et al. (Reference Vries, Schoonvelde and Schumacher2018), we compare the reliability of MT for topic modeling by comparing topic model outputs to those of a model fitted directly to texts translated by human experts. The intuition of this comparison is the following. If we find higher discrepancies in the model fitted to human translations when we use an open-source MT model for MTs instead of Google Translate, this would indicate that relying on open-source models for translation impairs the reliability of LDA topic modeling. This finding would support the conclusion that open-source MT models’ translation quality is insufficient for applied bag-of-words text analysis. However, if the topic model fitted on open-source MT models’ translations fares no worse (or even better) than the benchmark model compared to an equivalent topic model fitted on translations obtained with Google Translate, we would conclude that open-source MT models enable comparatively reliable topic modeling of machine-translated texts.

Accordingly, we evaluate whether using open-source MT models for translation instead of Google Translate negatively affects topic models’ measurements compared to the human translation benchmark. To compare the quality of open-source MT, we translated the parallel corpora from the respective language into English with OPUS-MT (Tiedemann and Thottingal, Reference Tiedemann and Thottingal2020).Footnote ¹² We then pre-processed the data in the same way as de Vries et al. (Reference Vries, Schoonvelde and Schumacher2018) did, created a TDM, and fitted an LDA topic model with 90 topics (using the same random seed, burn-in time, and number of iterations). Finally, we compare the inputs and outputs of topic models fitted to OPUS-MT or Google Translate translations to the ones obtained from human expert-translated texts.Footnote ¹³ We used the same metrics as de Vries et al. (Reference Vries, Schoonvelde and Schumacher2018): The similarity between documents’ bag-of-words representations. The similarity between the documents’ estimated topic proportions. And the similarity between the estimated topics’ document compositions. Overall, our empirical strategy allows us to evaluate whether, compared to the commercial Google Translate translations, the translations obtained from OPUS-MT led to stronger discrepancies vis-à-vis the expert translation benchmark.

To compare Google Translate and OPUS-MT, we compute how much the measurements obtained with the LDA topic models fitted to their translations differ from those obtained with the LDA topic model fitted to expert translations. We then assess how much these deviations from the “gold standard” model differ between the commercial and open-source MT-based topic models. However, statistical significance in difference-in-means tests does not answer the question of whether this particular difference is substantial.Footnote ¹⁴ We thus also rely on equivalence testing (Lakens et al., Reference Lakens, Scheel and Isager2018). In equivalence testing, researchers define the smallest effect size of interest before analyzing the data. The observed difference in means is then compared with this threshold to examine whether an effect size falls within a range of values considered equivalent to a null effect. If the test statistic and its confidence interval exceed the range of values defined as negligible, the equivalence hypothesis is rejected. We set the equivalence bound to ± 0.01.

3.2.1. Data

We have compiled a large benchmark of human-labeled multilingual political text datasets for our analyses. Our benchmark includes five datasets (Theocharis et al., Reference Theocharis, Barberá, Fazekas, Popa and Parnet2016; Lehmann and Zobel, Reference Lehmann and Zobel2018; Düpont and Rachuj, Reference Düpont and Rachuj2022; Poljak, Reference Poljak2023; Ivanusch and Regel, Reference Ivanusch and Regel2024). As shown in Table 4, these datasets jointly cover 31 languages from the Indo-European, Uralic, and Turkic language families and three domains of political communication (party manifestos, parliamentary speech, and social media).

Table 4.

Datasets

^a Additional analyses conducted including speeches from Croatia written in Bosnian (bs) and Croatian (hr).

^b Additional analyses conducted including tweets written in Greek (el).

We have obtained MTs into English for all non-English texts in our datasets. The CMP Translations corpus contributed by Ivanusch and Regel (Reference Ivanusch and Regel2024) contains English translations of sentences generated with DeepL, and we have added translations from two open-source MT models (M2M 1.2B and OPUS-MT).Footnote ¹⁶ For all other datasets, we have obtained English translations from two commercial services (DeepL and Google Translate) and three open-source MT models (M2M 418 M, M2M 1.2B, and OPUS-MT). In addition, all but the Lehmann and Zobel (Reference Lehmann and Zobel2018) data come with Google Translate translations that the datasets’ owners obtained when compiling them.Footnote ¹⁷

The texts in the five datasets included in our benchmark have been coded on multiple dimensions. As shown in Table 5, our benchmark thus covers various target concepts, ranging from policy topics to negative campaigning to incivility. We use this variation within and across datasets to define 15 classification tasks (see column two in Table 5). We adopt this comparative approach to facilitate the generalizability of our findings. By including tasks focusing on concepts with varying levels of difficulty in corpora from different domains, we ensure that our findings on the “reliability cost” of using open-source MT are not specific to a single dataset, target concept, or classification task.

Table 5.

Tasks overview

3.2.2. Empirical strategy

We proceed in two steps. First, we analyze the similarities of translations obtained with open-source and commercial MT models. This enables us to understand how using an open-source MT model instead of a commercial one might impact the inputs used to fine-tune a classifier.

Second, we turn to our main research question of whether using an open-source MT model instead of a commercial one to translate the texts taken as inputs when fine-tuning impacts a classifier’s predictions and reliability negatively. For this analysis, we fine-tune one text classifier per MT model, dataset, and task covered by our benchmark (see Table 5) on a training data split. This results in four classifiers per task in the CMP Translations corpus and six (five) classifiers per task for the other datasets with(out) old Google Translate translations.Footnote ¹⁸ In addition, we have fine-tuned one multilingual classifier per task to enable comparisons with the alternative strategy of aligning texts by embedding instead of translation (cf. Licht and Lind, Reference Licht and Lind2023).Footnote ¹⁹

We then use these classifiers to predict the labels of held-out texts in the corresponding test set splits. Holding constant the random seed, data splits, and training hyper-parameters used when fine-tuning,Footnote ²⁰ the only factor that varies between classifiers fine-tuned for the same task is the translation model used to translate the source texts into English. We can thus attribute differences in classifiers’ predictions and out-of-sample classification performances to the MTs used to fine-tune them.

Specifically, we compare classifiers at two levels of analysis. First, we compare classifiers’ language- and label-class-specific reliability in labeling held-out texts using the F1 score.Footnote ²¹ Second, we analyze the agreement and discrepancies at the level of predicted labels of the test set examples.

3.2.3. Similarities between open-source and machine commercial translations

A first step to understanding how using an open-source MT model instead of a commercial one might impact the predictions and classification reliability of a classifier fine-tuned using these translations as inputs is to assess how similar or dissimilar the translations generated with alternative models are.

We address this question based on a sample of 500 sentences per language in our full-sentence sample of the CMP Translations corpusFootnote ²² by computing the similarity between translations obtained with M2M (1.2B), respectively, OPUS-MT, to the translations obtained with DeepL for each sentence in this sample.Footnote ²³ To measure the similarity between two translations of the same source sentence obtained with different MT models, we rely on BERTScore (Zhang et al., Reference Zhang, Kishore, Felix, Weinberger and Artzi2020). BERTScore is a method for estimating the semantic similarity of two sentences by comparing the contextualized embeddings of their tokens obtained with a pre-trained transformer model.Footnote ²⁴

Figure 2 summarizes the distribution of BERTScore F1 scores for translations obtained with the M2M (1.2B) and OPUS-MT models compared to DeepL translations by language (y-axis) and open-source MT model (plot panels). Numbers plotted above box plots report average BERTScore F1 scores per language and open-source MT model. The closer the BERTScore F1 is to 1.0, the more similar a translation pair is.

Figure 2.

Distribution of similarities of open-source MT models’ translations to DeepL translations by language and open-source MT model in sample of 500 sentences per language sampled from the CMP Translations corpus. Translation similarity measured with BERTScore at translation pair level. Note that no OPUS-MT translation to English was obtained for Greek, Lithuanian, Norwegian, Portuguese, Romanian, and Slovenian due to translation direction limitations.

Figure 2 shows that both open-source MT models typically generate translations highly similar to DeepL translations, as translations’ similarities are mostly between 0.9 and 1.0. This indicates that the two open-source MT models typically generate translations that are semantically highly similar to those produced by DeepL.

However, three observations are notable. First, as further illustrated in Figure C1 in the Supplementary Material, for each of the two open-source MT models we analyze, there are some languages whose translations’ similarities to DeepL translations are, on average, above the cross-language average (i.e., grand mean) of all languages, and some languages whose average translation similarities are below this cross-language average. For example, the similarity of the M2M (1.2B) model’s translations of Spanish and Swedish sentences to their DeepL translations is above average, while its Finnish, Hungarian, and Turkish translations are below average. Second, this sorting of languages into above- and below-average groups is partially consistent across open-source MT models (e.g., Hungarian, Spanish, Swedish, and Turkish). Assuming that DeepL yields on average higher quality translations, this suggests that the two open-source MT models share some strengths and weaknesses. Third, while the estimated pairwise similarity is above 0.9 for the vast share of sentences, there are a few “outlier” data points with BERTScore F1 scores below the lower whisker of the box plot for each language and open-source MT model. These are cases of high semantic discrepancy between open-source MT and DeepL translations, and we will pay greater attention to these outliers in an analysis reported in Section 3.4.

3.2.4 (No) reliability cost of open-source machine translation

Next, we compare the reliability of classifiers fine-tuned using open-source MT models’ translations to classifiers fine-tuned using commercial MT models’ translations. Specifically, we compare language- and label-class-specific F1 scores between classifiers trained on translations generated with different MT models within dataset, task, and language through regression modeling and complement these analyses with equivalence testing. While there is much heterogeneity in classifiers’ performance across datasets, tasks, and languages, our experimental setup allows us to focus on the translation model (type) as a factor shaping classifiers’ behavior. However, a detailed breakdown of classifiers’ F1 scores by dataset, task, label class, and/or language can be found in Table C9, Figures C2–C6, and Tables C10–C14 in the Supplementary Material.

We first present the results of the regressions that model the overall effect of the (type of) MT model used to translate texts on classifiers’ language- and label-class-specific F1 scores, and add interactions with source language indicators in a second step to assess heterogeneous effects across language.Footnote ²⁵ To account for uncertainty in estimates of classifiers’ test set F1 scores, we bootstrap F1 estimates from each classifier’s test set predictions and use these bootstrapped scores in our analysis. Furthermore, to account for heterogeneity in F1 scores across datasets, tasks, languages, and label classes (cf. Figures C2–C6), all our regressions include dataset, task, source language, and label class fixed effects and cluster standard errors by source language, tasks, label class, and translation model.

Specifically, we estimate the following regressions:

(1)\begin{equation}{\textrm{F}}{1_d},t,c,l = {^{{\beta }}}0{ + {^{\beta }}}1{\textrm{MT model type }}{ + {^{\delta }}}d{ + {^{\theta }}}t{ + {^{\kappa }}}c{ + {^{\lambda }}}l{ +} ^d,t,c,l\end{equation}

(2)\begin{equation}{\textrm{F}}{1_d},t,c,l{ = {^{\beta }}}0{ + {^{\beta }}}1{\textrm{MTmodel}}{ + {^{\delta }}}d{ + {^{\theta }}}t{ + {^{\kappa }}}c{ + ^{{\lambda }}}l{ +} ^d,t,c,l\end{equation}

(3)\begin{equation}{\text{F}}{1_d},t,c,l{ = {^{\beta }}}0{ + {^{\beta }}}1{\text{MT model }}{ + {^{\beta }}}2{\text{language}}\end{equation}

\begin{equation*}{ + {{\beta}} {\text3{(MT model \times language)}}}\end{equation*}

\begin{equation*}{{ + \delta d + \theta t + \kappa c + \lambda l + \epsilon \ d,t,c,l}}\end{equation*}

where F1 score _{d , t, c, l} represents a bootstrapped F1 score for dataset d, task t, label class c, and language l; in equation (1), “MT model type” is a categorical indicator differentiating between commercial MT-based classifiers (reference category), open-source MT-based translations, and multilingual Transformer-based classifiers; in equations (2) and (3), “MT model” indicates the MT model used for translation (using DeepL as the reference category); in equation (3), “language” indicates the source language; and δ, θ, κ, and λ are the dataset, task, label class, and source language fixed effects estimates, respectively.Footnote ²⁶

If for model (1) our estimate of β₁ for the open-source MT-based classifier category is negative, this would indicate that using an open-source MT model for translation instead of a commercial one results, on average, in a lower F1 score, indicating poorer reliability in labeling held-out texts. The magnitude of the coefficient estimate, in turn, indicates how much worse.

We first turn to the overall effect of open-source versus commercial MT (equation 1). Model 1 in Table 6 estimates the average difference in classifiers’ bootstrapped language and label-class-specific F1 scores when using an open-source MT model instead of a commercial one. This difference is estimated to be negative and statistically significant (t = −9.57, p < 0.000). However, the estimated magnitude is only 0.009, that is, less than a difference of 0.01 units on the [0, 1] F1 score scale.

Table 6.

OLS coefficient estimates of the effect of using open-source vs. commercial machine translation models for translating input texts on classifiers’ language-specific out-of-sample classification performance (F1 score)

*** p < 0.001; ** p < 0.01; *p < 0.05.

The F1 score is measured on a scale from 0 to 1. A coefficient estimate of, for example, +0.01 (+0.001) represents an average increase of the F1 score by 0.01 (0.001), that is, one (a tenth of one) F1 score point. All models include data set, task/outcome, and language fixed effects.

Standard errors clustered by data set, task/outcome, language, and, in case of tasks with more than two labels, by label class.

Thus, when fine-tuning a supervised text classifier, researchers should expect a reduction in its out-of-sample classification reliability if they use an open-source instead of a commercial MT model for translation. However, they can expect that this reduction will be negligibly small, considering that even classifiers fine-tuned on different folds of the same dataset (Licht, Reference Licht2023; Laurer et al., Reference Laurer, Van Atteveldt, Casas and Welbers2024) or with different seeds (Wang, Reference Wang2023) usually exhibit higher levels of variability in test set F1 scores than our estimate of 0.009.

Equivalence tests (Lakens et al., Reference Lakens, Scheel and Isager2018) support this conclusion (see Table C17 in the Supplementary Material). The difference between the average bootstrapped F1 scores of commercial and open-source MT-based classifiers is 0.006 and within the equivalence bound of [−0.02, 0.02] (t = −5.338, p < 0.000). Moreover, Model 1 in Table C15 in the Supplementary Material shows that these findings hold when we omit classifiers fine-tuned with input text translations generated with older versions of Google Translate from the comparison. And Model 2 in Table C15 presents evidence that the estimated F1 score difference of 0.009 reported in Model 1 in Table 6 drops by 53% if we remove classifiers fine-tuned with input text translations generated with the small (418B) M2M model from the comparison.

Model 2 in Table 6 adds nuance to these findings. It compares classifiers fine-tuned with DeepL translations to ones fine-tuned using one of the other MT model’s translations. This shows that using the open-source OPUS-MT instead of DeepL does not significantly reduce classifiers’ test set F1 scores. In the case of the M2M (1.2B) model, the difference is negative and statistically significant, but with a magnitude of only 0.006 on the [0, 1] F1 score scale, again small and arguably practically negligible.Footnote ²⁷ Furthermore, Model 2 in Table 6 shows that even the “worst” alternative of using the smaller M2M (418 M) model only reduces classifiers’ test set F1 score by 0.014 compared to using DeepL.

Turning to cross-language heterogeneity, Figure 3 reports predicted values of the regression model fitted with equation (3) on page 19.Footnote ²⁸ It shows that our finding of a very small reliability cost of open-source MT in supervised text classification holds for most high- and low-resource languages included in our benchmark.Footnote ²⁹ In high-resource languages (top panel), open-source MT-based classifiers are predicted to perform, on average, slightly worse for Danish, German, Spanish, Dutch, and Swedish and better for French and Italian. However, these differences are within ± 0.01 on the [0, 1] F1 score scale for all these languages except Swedish. For low-resource languages (bottom panel), this pattern replicates in many languages, including those with other scripts (e.g., Bulgarian, Greek, and Russian) and/or from other language families than the Romance and Germanic families (Czech, Estonian, Finnish, Hungarian, Romanian, and Slovenian). Comparatively large negative differences (≤ −0.015) exist only for Norwegian, Polish, Slovak, Turkish, and Ukrainian. However, it is worth emphasizing that some of these languages are only represented in one dataset of our benchmark (see Table 4), which limits our ability to generalize the language-specific patterns we find for these languages.

Figure 3.

Predicted language-specific F1 scores by language and type of MT model. Estimates based on regression reported in Table C18.