2.1 Prior surveys of document-level NMT

Several papers have registered the advances in the document-level field of MT, surveying, discussing, and reporting different methodologies and model architectures, with some simultaneously introducing new approaches to the problems. In 2019, Popescu-Belis (Reference Popescu-Belis2019) published a review paper describing work from 2017 to 2018 that made use of contextual information in NMT. The author gives a comprehensive overview of the rapid evolution of the NMT models and the first attempts to add context to those systems, dating back from statistical MT systems, to newer NMT models with context information. In the same year, (Kim, Tran, and Ney, Reference Kim, Tran, Ney, Popescu-Belis, Loáiciga, Hardmeier and Xiong2019, p. 24) experiment with evaluating a document-level model with test sets in order to quantify “when and why document-level context improves NMT,” providing insights into how and when context can be useful in NMT. The authors considered “document-level” to be one previous source sentence, and filtered words in the context, retaining only the “context words that are likely to be useful” (Kim, Tran, and Ney, Reference Kim, Tran, Ney, Popescu-Belis, Loáiciga, Hardmeier and Xiong2019, p. 26). Their findings demonstrated that, at that time, (i) most of the improvements lacked clear interpretation regarding context utilisation, (ii) minimal encoding was sufficient for the context modelling, (iii) and very long context did not offer substantial benefits for NMT.

Lopes et al. (Reference Lopes, Farajian, Bawden, Zhang, Martins, Martins, Moniz, Fumega, Martins, Batista, Coheur, Parra, Trancoso, Turchi, Bisazza, Moorkens, Guerberof, Nurminen, Marg and Forcada2020) provided a comparison of existing and new (at the time) document-level NMT solutions with both automatic and manual evaluations. The approaches they compared were concatenation (previous source and/or target sentence concatenated with the current), multi-source (previous two source sentences) and cache-based (all previous source and/or target sentences). For comparison, the authors also implemented one model based on the Star Transformer architecture (Guo et al., Reference Guo, Qiu, Liu, Shao, Xue, Zhang, Burstein, Doran and Solorio2019), newly proposed at that time, using the cache-based approach. They found that, at that stage, strong sentence-level baselines were still outperforming existing context-aware approaches in scenarios with larger datasets.

The survey by Maruf, Saleh, and Haffari (Reference Maruf, Saleh and Haffari2021) is a comprehensive analysis of the major work in the domain of document-level MT from the introduction of NMT to the date of publication. The survey encompasses research conducted within the frameworks of both RNN and Transformer, offering a comprehensive taxonomy, with different types of context span used. They also provide a review of the evaluation strategies for document-level NMT.

Finally, Jin et al. (Reference Jin, He, May, Ma, Bouamor, Pino and Bali2023) present a review of literature pointing out some key obstacles relating to discourse phenomena, context usage, model architectures, and evaluation that hinder progress in the domain. These include: (i) document-level corpora with sparse number of discourse phenomena, (ii) context being less helpful for tense and discourse markers, (iii) concatenation-based NMT systems not outperforming context-agnostic sentence-level Transformer baselines, (iv) meaningful improvement not being achieved by advanced model architectures, and (v) current evaluation metrics not adequately measuring quality. The authors propose a novel setting for document-level translation with a dataset of aligned paragraphs.

In summary: As can be seen, extensive work has been done in the past decade to understand how context influences the translation process, since it is essential for enhancing the quality and fluency of MT systems. In the rest of this survey, we will briefly discuss earlier work to provide background and then attempt to augment these existing surveys with additional recent work.

2.2 Intersentential context

We now consider the question of context as interpreted to mean intersentential context. Performing MT at the sentence level makes an implicit assumption that sentences are independent—that is, that the translation of one sentence does not depend on additional information outside of the sentence itself. This is of course trivially untrue; even a sentence as simple as “I see the bank.” contains a polysemous word (bank) whose translation into another language may depend on whether this particular sentence is intended to refer to a financial institution or the side of a river, for example. Nevertheless, as evidenced by the success of the single-sentence translation paradigm, many sentences can be translated well enough in isolation. For those sentences that do require additional context to translate, MT research often focuses on intersentential context (part of what Melby and Foster (Reference Melby and Foster2010) would call “co-text”): context that crosses sentential boundaries but that exists within some sort of document boundary.Footnote ^a

Incorporating intersentential context or document context has the potential to provide discourse information such as disambiguation of pronouns, deixis, ellipsis, as well as general cohesion. For example, there may be cases of anaphoric pronouns, where information about the person or thing mentioned earlier in the text can be used to disambiguate the translation of a pronoun in a later sentence. Other portions of the text may similarly be able to fill in the necessary blanks to resolve questions related to deixis or ellipsis. As for cohesion, this could be a matter of improving consistency of terminology (see also Section 2.4) or consistency in senses for polysemous terms, since a single discourse tends to make use of just one sense per polysemous word (Gale, Church, and Yarowsky, Reference Gale, Church and Yarowsky1992). Carpuat (Reference Carpuat, Agirre, Màrquez and Wicentowski2009) found that the concept of one sense per discourse also held in the case of translation, showing for phrase-based statistical MT that improving sense consistency (e.g., by incorporating document context) had the potential to improve translation quality, so we have reason to expect the same to be true in the case of NMT.

This incorporation of intersentential context can take several forms, including translation at the sentence level that incorporates information beyond the sentence being translated as well as performing translation of units larger than the sentence. Maruf et al. (Reference Maruf, Saleh and Haffari2021) provide a thorough survey of approaches to handling intersentential context in NMT, so in this section we will provide only a brief overview of common approaches.

Perhaps the most intuitive of these approaches, albeit one that comes with computer-related challenges, is to perform translation of entire paragraphs or documents at one time. That is, rather than translating one sentence at a time, having the unit of translation be the paragraph or the document (i.e., document-to-document translation). This comes with several challenges from both the data side and the implementation side. Nevertheless, Junczys-Dowmunt (Reference Junczys-Dowmunt, Bojar, Chatterjee, Federmann, Fishel, Graham, Haddow, Huck, Yepes, Koehn, Martins, Monz, Negri, Névéol, Neves, Post, Turchi and Verspoor2019) observed strong performance gains over sentence-level baselines by extending the length of the input to 1000 subword tokens (i.e., using paragraphs or documents as the unit of translation rather than individual sentences; Popel et al. (Reference Popel, Macháček, Auersperger, Bojar, Pecina, Bojar, Chatterjee, Federmann, Fishel, Graham, Haddow, Huck, Yepes, Koehn, Martins, Monz, Negri, Névéol, Neves, Post, Turchi and Verspoor2019) took a similar approach, by lengthening the input to 1,000 characters or approximately 15 sentences), adding special characters to indicate sentence and document boundaries, backtranslation, and creating synthetic documents for training from parallel data that did not contain document boundary information. This last piece highlights one of the challenges of document-level MT: the fact that many large sources of training data, having been prepared for use in sentence-level translation, do not contain document boundary information. In re-examining document-to-document translation, Sun et al. (Reference Sun, Wang, Zhou, Zhao, Huang, Chen, Li, Muresan, Nakov and Villavicencio2022) found that multi-resolution training, wherein they incorporate both full documents and shorter segments, can improve performance. Al Ghussin et al. (Reference Al Ghussin, Zhang, van Genabith, Vlachos and Augenstein2023) explore the possibility of automatically extracting paragraph-level data from Paracrawl, for use in training document-level MT.

Another common approach is to perform sentence-level translation, but incorporate intersentential context. This often consists of providing the system with information about several nearby sentences, from the source side, the target side, or both sides. While it is most common to use preceding intersentential context, some work has also explored the use of future context, particularly for the resolution of cataphoric pronouns (Wong, Maruf, and Haffari, Reference Wong, Maruf, Haffari, Jurafsky, Chai, Schluter and Tetreault2020). Tiedemann et al. (Reference Tiedemann, Scherrer, Webber, Popescu-Belis and Tiedemann2017) examined both extending the unit of translation to two sentences (similar to the approaches in the previous paragraph) as well as providing the preceding source sentence concatenated with the sentence of interest as the input to the model and training the model to produce a translation of the source sentence of interest only. In this way, they sidestepped the issue of modifying the translation model by modifying the data used for training and the processing of the data for decoding. Similarly, Rippeth et al. (Reference Rippeth, Carpuat, Duh and Post2023) seek to improve word sense disambiguation by prefixing input data with salient keywords from the surrounding context. Jean and Cho (Reference Jean and Cho2019) also avoided modifying the model architecture by proposing a novel regularisation term for the learning algorithm to encourage systems to focus on useful context. A number of architecture modifications have been proposed to incorporate intersentential context, such as additional encoders and attention (Jean et al., Reference Jean, Lauly, Firat and Cho2017), memory networks that incorporate both source and target intersentential context (Maruf and Haffari, Reference Maruf, Haffari, Gurevych and Miyao2018), multi-head hierarchical attention networks (Miculicich et al., Reference Miculicich, Ram, Pappas, Henderson, Riloff, Chiang, Hockenmaier and Tsujii2018), hierarchical sparse attention (Maruf, Martins, and Haffari, Reference Maruf, Martins, Haffari, Burstein, Doran and Solorio2019), query-guided capsule networks (Yang et al., Reference Yang, Zhang, Meng, Gu, Feng, Zhou, Inui, Jiang, Ng and Wan2019), summarising cross-sentence context with a hierarchy of recurrent neural networks and then using this for initialisation or incorporating it via gating (Wang et al., Reference Wang, Tu, Way, Liu, Palmer, Hwa and Riedel2017), and incorporating context information into Transformer models using context encoders that share parameters with the source encoder (Voita et al., Reference Voita, Serdyukov, Sennrich, Titov, Gurevych and Miyao2018).

Huo et al. (Reference Huo, Herold, Gao, Dahlmann, Khadivi, Ney, Barrault, Bojar, Bougares, Chatterjee, Costa-jussà, Federmann, Fishel, Fraser, Graham, Guzman, Haddow, Huck, Yepes, Koehn, Martins, Morishita, Monz, Nagata, Nakazawa and Negri2020) compare four different context-aware architectures, while Gete et al. (Reference Gete, Etchegoyhen, Labaka, Nurminen, Brenner, Koponen, Latomaa, Mikhailov, Schierl, Ranasinghe, Vanmassenhove, Vidal, Aranberri, Nunziatini, Escartín, Forcada, Popovic, Scarton and Moniz2023) compare two concatenation-based approaches on the task of pronoun translation. These represent just a taste of the many approaches that have been employed and analysed (see also, Agrawal, Turchi, and Negri, Reference Agrawal, Turchi, Negri, Pérez-Ortiz, Sánchez-Martínez, Esplà-Gomis, Popović, Rico, Martins, Van den Bogaert and Forcada2018; Kim et al., Reference Kim, Tran, Ney, Popescu-Belis, Loáiciga, Hardmeier and Xiong2019; Macé and Servan Reference Macé, Servan, Niehues, Cattoni, Stüker, Negri, Turchi, Ha, Salesky, Sanabria, Barrault, Specia and Federico2019; Li et al., Reference Li, Liu, Wang, Jiang, Xiao, Zhu, Liu, Li, Jurafsky, Chai, Schluter and Tetreault2020; Kang et al., Reference Kang, Zhao, Zhang, Zong, Webber, Cohn, He and Liu2020; Saunders, Stahlberg, and Byrne, Reference Saunders, Stahlberg, Byrne, Jurafsky, Chai, Schluter and Tetreault2020b; Yu et al., Reference Yu, Sartran, Stokowiec, Ling, Kong, Blunsom and Dyer2020; Mansimov, Melis, and Yu, Reference Mansimov, Melis, Yu, Braud, Hardmeier, Li, Louis, Strube and Zeldes2021, i.a.). Returning to the issue of training data that may not always have document boundaries available, Zheng et al. (Reference Zheng, Yue, Huang, Chen and Birch2021) seek to balance local and global context, with an architecture that is flexible enough to allow for documents of any length (including single sentences), enabling the models to train on and translate either single isolated sentences or sentences in document context.

Herold and Ney (Reference Herold, Ney, Strube, Braud, Hardmeier, Li, Loaiciga and Zeldes2023) address the issues of memory usage and translation performance when using large context by proposing a constrained attention approach, which reduces memory consumption while focusing on relevant portions of the context. They also consider the issue of evaluation, as we will discuss in Section 3: document-level models often show small improvements in terms of standard automatic metrics, but may exhibit performance boosts on specific tasks or targeted test sets, particularly those closely related to intersentential context.

In summary: Much of the work on MT and MT evaluation to date has focused on the sentence level. This means that the current tools for evaluation are predominantly designed to capture facets of quality that can be measured at the sentence level, and may not be able to capture improvements that are related to intersentential context. We discuss these issues of evaluation more in Sections 3.2 and 3.2. Nevertheless, as has been made clear by analyses like Toral et al. (Reference Toral, Castilho, Hu, Way, Bojar, Chatterjee, Federmann, Fishel, Graham, Haddow, Huck, Yepes, Koehn, Monz, Negri, Névéol, Neves, Post, Specia, Turchi and Verspoor2018) and Läubli et al. (Reference Läubli, Sennrich, Volk, Riloff, Chiang, Hockenmaier and Tsujii2018), there remains room for improvement over the current sentence-level paradigm of translation, some of which will likely be found through the incorporation of intersentential context. It remains to be seen what the best ways of incorporating such context are, that is whether it is sufficient to simply perform translation on units the size of whole documents, in what ways training or model architectures should be modified to incorporate more context, or whether other approaches will prevail. It is clear, however, that translation at the level of the single sentence, without intersentential context, differs at least some of the time in notable ways from optimal human references, including when it comes to issues of discourse and cohesion. We expect the best systems in the future—especially when those systems are ranked or compared by human annotators or automatic metrics in ways that take intersentential context into consideration—to incorporate intersentential context in some manner, as human translators do in their work.

2.3 World knowledge and external information

In the realm of the type of context that might be called non-text, as described in Melby and Foster (Reference Melby and Foster2010), we can consider ways in which real-world context is relevant to translation. This can include knowledge about the world, as might be found in a knowledge base, as well as knowledge about the goals of the translation task itself. While some of this information may be captured at the document level, at other times it may need to be provided in some other way to the system.

Regarding the latter, this can include how to handle desired levels of formality. Sennrich et al. (Reference Sennrich, Haddow, Birch, Knight, Nenkova and Rambow2016a) proposed the use of “side constraints” or special tokens inserted into the source sentence in order to control the formality of MT output, in their case specifically in terms of the formal/informal “you” (T-V) distinction. Similarly, Feely et al. (Reference Feely, Hasler, de Gispert, Nakazawa, Ding, Dabre, Kunchukuttan, Doi, Oda, Bojar, Parida, Goto and Mino2019) used source-side tags to control the generation of honorifics in English to Japanese translation. Niu et al. (Reference Niu, Rao, Carpuat, Bender, Derczynski and Isabelle2018) generalised beyond pronouns and honorifics to a more general sense of (in)formality in formality-sensitive MT, using similar side constraints as well as a multi-task approach that incorporated formality transfer. Niu and Carpuat (Reference Niu and Carpuat2020) subsequently proposed an approach involving on-the-fly synthetic example generation during training to improve formality translation.

This approach to tagging input sentences with special tokens has also been applied to controlling the output domain in multi-domain NMT systems (Kobus, Crego, and Senellart, Reference Kobus, Crego, Senellart, Mitkov and Angelova2017). Handling multiple domains in one MT system involves either providing the NMT system with information about the desired output domain or training it to learn to handle different input domains, such as by predicting their domain from the input sentence. Beyond the use of special tokens as side constraints, approaches to handling multiple domains and training systems to perform well in multi-domain settings include jointly learning to translate and discriminate between domains (Britz, Le, and Pryzant, Reference Britz, Le, Pryzant, Bojar, Buck, Chatterjee, Federmann, Graham, Haddow, Huck, Yepes, Koehn and Kreutzer2017), treating domain as a feature for factored NMT (Tars and Fishel, Reference Tars, Fishel, Pérez-Ortiz, Sánchez-Martínez, Esplà-Gomis, Popović, Rico, Martins, Van den Bogaert and Forcada2018), instance-based on-the-fly domain adaptation (Farajian et al., Reference Farajian, Turchi, Negri, Federico, Bojar, Buck, Chatterjee, Federmann, Graham, Haddow, Huck, Yepes, Koehn and Kreutzer2017; Li, Zhang, and Zong, Reference Li, Zhang, Zong, Calzolari, Choukri, Cieri, Declerck, Goggi, Hasida, Isahara, Maegaard, Mariani, Mazo, Moreno, Odijk, Piperidis and Tokunaga2018; Xu, Crego, and Senellart, Reference Xu, Crego, Senellart, Niehues, Cattoni, Stüker, Negri, Turchi, Ha, Salesky, Sanabria, Barrault, Specia and Federico2019), and domain-specific multi-head attentions (Jiang et al., Reference Jiang, Liang, Wang, Zhao, Jurafsky, Chai, Schluter and Tetreault2020). Pham et al. (Reference Pham, Crego and Yvon2021) propose a number of properties that should hold for high-quality multi-domain NMT systems, examine these various approaches, and conclude that there is still work to do on smoothly handling multi-domain NMT. Domain can be interpreted quite broadly, and the successful approaches to issues of domain and formality can also be applied to a range of other related and overlapping topics. This includes jointly translating and simplifying text (Agrawal and Carpuat, Reference Agrawal, Carpuat, Inui, Jiang, Ng and Wan2019) or controlling the reading level at which MT output is generated (Marchisio et al., Reference Marchisio, Guo, Lai, Koehn, Forcada, Way, Haddow and Sennrich2019). The concepts of domain and (intersentential) context have also been directly brought together in Stojanovski and Fraser (Reference Stojanovski, Fraser, Ben-David, Cohen, McDonald, Plank, Reichart, Rotman and Ziser2021), who note that providing additional intersentential context can help improve performance on new domains and can improve coherence and consistency in translation.

Most of these describe multi-domain NMT systems. However, NMT systems can also be built or adapted to specifically handle one particular domain. We can consider the concept of providing domain information to a system as a type of non-text or world knowledge that is being incorporated into the MT system. Chu et al. (Reference Chu, Dabre, Kurohashi, Barzilay and Kan2017) compared several approaches to domain adaptation and Chu and Wang (Reference Chu, Wang, Bender, Derczynski and Isabelle2018) surveyed approaches to domain adaptation for NMT. Even more recently, Saunders (Reference Saunders2022) surveyed both domain adaptation as well as multi-domain approaches in NMT.

While we focus primarily on topics related to “standard” text-to-text translation, we should also note that various other settings for translation may require specialised approaches; we limit our discussion to noting a few of them. In these cases, we are considering context in the sense of the use case for translation: the setting for which the particular system is designed, which may inform choices about the model or about desired qualities of the MT output. Translation for dubbing requires producing translations of similar lengths to the source (Lakew et al., Reference Lakew, Virkar, Mathur and Federico2022), while translation for subtitling may require handling additional formatting issues (Cherry et al., Reference Cherry, Arivazhagan, Padfield and Krikun2021), and simultaneous MT requires approaches to better handle potential variations in word order (Grissom II et al., Reference Grissom, He, Boyd-Graber, Morgan, Daumé, Moschitti, Pang and Daelemans2014) or incremental decoding approaches (Gu et al., Reference Gu, Neubig, Cho, Li, Lapata, Blunsom and Koller2017; Dalvi et al., Reference Dalvi, Durrani, Sajjad, Vogel, Walker, Ji and Stent2018). Similarly, NMT systems for use in computer-aided translation settings such as interactive translation prediction may use modified decoding or training approaches (Knowles and Koehn, Reference Knowles, Koehn, Green and Schwartz2016; Wuebker et al., Reference Wuebker, Green, DeNero, Hasan, Luong, Erk and Smith2016; Li et al., Reference Li, Liu, Huang, Shi, Zong, Xia, Li and Navigli2021). The external information provided to the MT system need not take the form of text. Saleh et al. (Reference Saleh, Berard, Calapodescu, Besacier, Birch, Finch, Hayashi, Konstas, Luong, Neubig, Oda and Sudoh2019) draw connections between natural language generation from structured data and document-level translation, and both Vincent et al. (Reference Vincent, Barrault, Scarton, Moniz, Macken, Rufener, Barrault, Costa-jussà, Declercq, Koponen, Kemp, Pilos, Forcada, Scarton, Van den Bogaert, Daems, Tezcan, Vanroy and Fonteyne2022) and Vincent et al. (Reference Vincent, Flynn, Scarton, Rogers, Boyd-Graber and Okazaki2023) seek to use metadata to inform translation. Another such example of external information that does not take the form of text is in multimodal translation, when an image and text are used as input for translation (Elliott, Frank, and Hasler, Reference Elliott, Frank and Hasler2015; Hitschler, Schamoni, and Riezler, Reference Hitschler, Schamoni, Riezler, Erk and Smith2016; Specia et al., Reference Specia, Frank, Sima’an, Elliott, Bojar, Buck, Chatterjee, Federmann, Guillou, Haddow, Huck, Yepes, Névéol, Neves, Pecina, Popel, Koehn, Monz, Negri, Post, Specia, Verspoor, Tiedemann and Turchi2016; Elliott et al., Reference Elliott, Frank, Barrault, Bougares, Specia, Bojar, Buck, Chatterjee, Federmann, Graham, Haddow, Huck, Yepes, Koehn and Kreutzer2017; Barrault et al., Reference Barrault, Bougares, Specia, Lala, Elliott, Frank, Bojar, Chatterjee, Federmann, Fishel, Graham, Haddow, Huck, Yepes, Koehn, Monz, Negri, Névéol, Neves, Post, Specia, Turchi and Verspoor2018, i.a.), or when both text and speech or some other signal are used. Translation from text or speech into sign languages also requires consideration of how to produce a signing avatar (McDonald et al., Reference McDonald, Wolfe, Efthimiou, Fontinea, Picron, Van Landuyt, Sioen, Braffort, Filhol, Ebling, Hanke, Krausneker and Shterionov2021), raises ethical questions around language data that contains video of individual signers (Bragg et al., Reference Bragg, Caselli, Hochgesang, Huenerfauth, Katz-Hernandez, Koller, Kushalnagar, Vogler and Ladner2021), and highlights the importance of language community involvement in technology development (De Meulder, Reference De Meulder and Shterionov2021), all aspects of a much broader world of context surrounding the task of translation.

In addition to the non-text sources of real-world context, there are other text-based sources that can incorporate world knowledge and external information into NMT systems. Linguistic knowledge, typically in the form of text, but potentially in other forms, may also be a source of knowledge to be incorporated into NMT systems. Particularly in low-resource settings, incorporating forms of linguistic knowledge or additional examples through data augmentation may be beneficial (Fadaee, Bisazza, and Monz, Reference Fadaee, Bisazza, Monz, Barzilay and Kan2017); we discuss the specific case of external lexicons in the following section. Linguistic knowledge that has already been made machine-readable through its inclusion in rule-based MT systems can be used in NMT systems (Torregrosa et al., Reference Torregrosa, Pasricha, Masoud, Chakravarthi, Alonso, Casas, Arcan, Forcada, Way, Tinsley, Shterionov, Rico and Gaspari2019). Incorporating monolingual data is also a way of providing additional linguistic information (i.e., improving language model capabilities) as well as a way of potentially providing the system with world knowledge (e.g., names of famous figures, news and events, etc.). A common way of incorporating monolingual data is by converting it into pseudo-parallel text through the process of backtranslation (Sennrich, Haddow, and Birch, Reference Sennrich, Haddow, Birch, Erk and Smith2016b), though other approaches have also been proposed. These include performing monolingual target language automatic post-editing at the document level to “repair” translations that were produced at the sentence level and improve their consistency (Voita, Sennrich, and Titov, Reference Voita, Sennrich, Titov, Inui, Jiang, Ng and Wan2019a), using target-side monolingual data for a particular domain to train a document-level language model and incorporating this into decoding for a sentence-level NMT model (Sugiyama and Yoshinaga, Reference Sugiyama, Yoshinaga, Toutanova, Rumshisky, Zettlemoyer, Hakkani-Tur, Beltagy, Bethard, Cotterell, Chakraborty and Zhou2021), and using multilingual modelling to transfer document-level translation information to language pairs without large corpora of document-level data (Zhang et al., Reference Zhang, Bapna, Johnson, Dabirmoghaddam, Arivazhagan, Firat, Muresan, Nakov and Villavicencio2022). For more discussion of monolingual data in NMT, see Burlot and Yvon (Reference Burlot, Yvon, Bojar, Chatterjee, Federmann, Fishel, Graham, Haddow, Huck, Yepes, Koehn, Monz, Negri, Névéol, Neves, Post, Specia, Turchi and Verspoor2018).

In summary: The external context in which translation occurs, as well as the intended audience of translation, are important factors in determining whether a given translation is appropriate. To this end, approaches to handling formality, domain, and specific use cases have been proposed. Additionally, particularly when the amount of available parallel text is limited, monolingual data can serve as useful training input for learning both linguistic features and incorporating world knowledge into NMT systems.

2.4 Terminology

Another source of external information for MT is the use of terminology. This can be viewed through the lens of context as what Melby and Foster (Reference Melby and Foster2010) would call rel-text, a text source of information outside of the particular document being translated. There is not necessarily a clean distinction between terminology and the types of world knowledge and external information that we discussed in Section 2.3; we discuss it in a separate section here due to both the large quantity of research in this area as well as the fact that the field has often treated it as a separate specific task. While the incorporation of terminology—in the form of source-target pairs of words or phrases—was relatively straightforward in the phrase-based statistical MT paradigm, it has been a topic of interest in NMT research, even resulting in a shared task at the Conference on Machine Translation in 2021 (Alam et al., Reference Alam, Kvapilíková, Anastasopoulos, Besacier, Dinu, Federico, Gallé, Jung, Koehn, Nikoulina, Barrault, Bojar, Bougares, Chatterjee, Costa-jussa, Federmann, Fishel, Fraser, Freitag, Graham, Grundkiewicz, Guzman, Haddow, Huck, Yepes, Koehn, Kocmi, Martins, Morishita and Monz2021). This topic and the various approaches to it all share two main connections to the topic of context and NMT. The first is that this represents a form of introducing additional context (rel-text) from the lexicon or terminology resource into the process of MT. The second is that this can contribute to within-document or even across-document consistency in translation, potentially improving document coherence. This second connection is one that ties closely to (human) evaluation of MT in context, which we discuss in Section 3.1.

There exist a number of motivations for wanting to incorporate terminological resources into MT, ranging from specific domains with highly technical vocabularies to client-specific preferred terms to low-resource settings where dictionary-like text may form a major component of the available bitext. Starting with the early days of NMT that used restricted vocabularies, one area of interest has been improving the translation of rare or out-of-vocabulary words using lexical resources (Jean et al., Reference Jean, Cho, Memisevic, Bengio, Zong and Strube2015; Luong et al., Reference Luong, Sutskever, Le, Vinyals, Zaremba, Zong and Strube2015, i.a.). Another reason for incorporating terminology into NMT is to improve consistency, for example of a particular term that appears repeatedly in a document or broadly across a client’s data. The types of terminological or lexical resources may vary based on the scenario, including but not limited to bilingual dictionary entries, term banks, or phrase tables extracted from corpora. Some are designed specifically with MT in mind as a use case, while others may need to be adapted to this (e.g., terminological resources designed for use by human translators). They may also vary in whether they contain only citation forms of words or fully inflected forms (and, in the latter case, whether they contain the full possible range of inflections or only a subset).

Yvon and Rauf (Reference Yvon and Rauf2020) provide a thorough overview of approaches to incorporate bilingual lexicons and other terminological resources into NMT. Some approaches require modifications at training time, while others only make modifications at inference time. The appropriateness of a given approach will depend on the use case and the available terminological resources. For example, if the terminology is expected to change over time, one may wish to select an approach that does not require a fixed terminology at training time. If the terminology only contains citation forms (e.g., roots or lemmas), care may need to be taken to ensure that the approach is flexible enough to support producing or recognising inflected surface forms.

Of the approaches that occur at training time, these can roughly be broken into two categories: those that rely on the fixed terminology (known in advance) for training and those that train for a specific behaviour but can incorporate flexible or changing terminological resources. A very simple approach is to treat a bilingual lexicon as simply more parallel text to use as training data. Kothur et al. (Reference Kothur, Knowles, Koehn, Birch, Finch, Luong, Neubig and Oda2018) fine-tune NMT systems using bilingual lexicons of novel vocabulary in order to improve translation consistency within individual documents for these novel terms, but warn of the risks of overfitting to such small sets of training data. Given a lexicon that is fixed in advance, another way of incorporating it is to combine translation probabilities from a lexicon with the NMT model’s probabilities (Arthur, Neubig, and Nakamura, Reference Arthur, Neubig, Nakamura, Su, Duh and Carreras2016) or to train a model to directly generate a target word based on a source word and combine this with the model’s scores (Nguyen and Chiang, Reference Nguyen, Chiang, Walker, Ji and Stent2018). These models would require modification or retraining in order to incorporate new entries in a terminology set.

There are approaches at training time that do allow for later changes to the terminology set, without retraining. These include copying mechanisms and placeholders. Post et al. (Reference Post, Ding, Martindale, Wu, Forcada, Way, Haddow and Sennrich2019) train models to incorporate special placeholders, which can then be replaced in postprocessing. Dinu et al. (Reference Dinu, Mathur, Federico, Al-Onaizan, Korhonen, Traum and Màrquez2019) use factors to train models to use custom terminology, relatively flexibly. Bergmanis and Pinnis (Reference Bergmanis, Pinnis, Merlo, Tiedemann and Tsarfaty2021b) expand on this, focusing on improving performance on morphological inflections. Song et al. (Reference Song, Zhang, Yu, Luo, Wang, Zhang, Burstein, Doran and Solorio2019) perform data augmentation in which the input is converted into “code-switched” data that includes desired target terminology, with the goal of encouraging the system to copy this terminology into the output. Ailem et al. (Reference Ailem, Liu, Qader, Zong, Xia, Li and Navigli2021) combine modification to the training data (in the form of tags to indicate when terminology should be used), token masking, and a weighted cross-entropy loss in order to incorporate terminology.

For cases where the use of terms from the lexicon is strictly required, these training-time approaches may not offer strong enough guarantees that the terms will be included in the output. Approaches like lexically constrained decoding, in which the beam search algorithm is modified to force the output to include certain tokens, offer stronger guarantees (Hokamp and Liu, Reference Hokamp, Liu, Barzilay and Kan2017; Post and Vilar, Reference Post, Vilar, Walker, Ji and Stent2018; Hu et al., Reference Hu, Khayrallah, Culkin, Xia, Chen, Post, Van Durme, Burstein, Doran and Solorio2019) at the cost of flexibility and occasional degradation of output quality. Hasler et al. (Reference Hasler, de Gispert, Iglesias, Byrne, Walker, Ji and Stent2018) present an approach to constrained decoding using finite-state machines, with improvements to the placement of the constraints in the resulting output. Susanto et al. (Reference Susanto, Chollampatt, Tan, Jurafsky, Chai, Schluter and Tetreault2020) propose an approach to lexically constrained decoding using Levenshtein transformers (Gu, Wang, and Zhao, Reference Gu, Wang, Zhao, Wallach, Larochelle, Beygelzimer, d’ Alché-Buc, Fox and Garnett2019). Inspired by the XML-style constraints in phrase-based statistical MT, Chatterjee et al. (Reference Chatterjee, Negri, Turchi, Federico, Specia, Blain, Bojar, Buck, Chatterjee, Federmann, Graham, Haddow, Huck, Yepes, Koehn and Kreutzer2017) propose guided decoding to enforce specified translations of text spans.

These training-time and inference-time approaches offer various tradeoffs; Exel et al. (Reference Exel, Buschbeck, Brandt, Doneva, Martins, Moniz, Fumega, Martins, Batista, Coheur, Parra, Trancoso, Turchi, Bisazza, Moorkens, Guerberof, Nurminen, Marg and Forcada2020) examine some of these by comparing an approach from Dinu et al. (Reference Dinu, Mathur, Federico, Al-Onaizan, Korhonen, Traum and Màrquez2019) to lexically constrained decoding. Bane et al. (Reference Bane, Zaretskaya, Miró, Uguet, Torres, Nurminen, Brenner, Koponen, Latomaa, Mikhailov, Schierl, Ranasinghe, Vanmassenhove, Vidal, Aranberri, Nunziatini, Escartín, Forcada, Popovic, Scarton and Moniz2023) compare three different approaches to incorporating terminology, but find that there is no clear winner; different approaches perform differently in different settings. Bergmanis and Pinnis (Reference Bergmanis, Pinnis, Barrault, Bojar, Bougares, Chatterjee, Costa-jussa, Federmann, Fishel, Fraser, Freitag, Graham, Grundkiewicz, Guzman, Haddow, Huck, Yepes, Koehn, Kocmi, Martins, Morishita and Monz2021a) highlight the importance of having high-quality terminology sources, noting that it is not sufficient that they be machine-readable term banks; the term banks used by professional translators often incorporate ambiguity that the skilled translator is expected to be able to handle, but either approaches to incorporating terminology need to improve to also handle this, term banks need to be produced specifically to work well for MT, or both of these need to be combined. Examining the same shared task as the previous work, Ballier et al. (Reference Ballier, Cho, Faye, Ke, Martikainen, Pecman, Wisniewski, Yunès, Zhu, Zimina-Poirot, Barrault, Bojar, Bougares, Chatterjee, Costa-jussa, Federmann, Fishel, Fraser, Freitag, Graham, Grundkiewicz, Guzman, Haddow, Huck, Yepes, Koehn, Kocmi, Martins, Morishita and Monz2021) also performed analysis of the provided term banks and showed a number of problematic areas. Focusing on one specific case of parliamentary text, Knowles et al. (Reference Knowles, Larkin, Tessier, Simard, Nurminen, Brenner, Koponen, Latomaa, Mikhailov, Schierl, Ranasinghe, Vanmassenhove, Vidal, Aranberri, Nunziatini, Escartín, Forcada, Popovic, Scarton and Moniz2023) discussed tradeoffs around the question of incorporating terminology.

In summary: While often considered as a separate topic, incorporating external terminology resources can be viewed as a problem of how to best incorporate context into NMT. There exist a number of effective approaches to incorporating lexicons and other terminological resources, though these come with various tradeoffs. Strong guarantees that exact phrases will appear in the output can lead to degraded translation quality or can lead to phrases being used in contexts where they are not actually appropriate. On the other hand, soft guarantees may still leave an undesired level of variation. Future work in this area will likely seek to balance these, as well as consider what types of terminology resources to use, or how to make better use of terminology resources that were initially designed for human translator use.

3.1 Context, human evaluation, and semi-automatic evaluation

With the improvement of translation quality in recent years, discriminating MT output from human translation has proven difficult with the current human evaluation methods (Läubli et al., Reference Läubli, Castilho, Neubig, Sennrich, Shen and Toral2020; Kocmi et al. Reference Kocmi, Avramidis, Bawden, Bojar, Dvorkovich, Federmann, Fishel, Freitag, Gowda, Grundkiewicz, Haddow, Koehn, Marie, Monz, Morishita, Murray, Nagata, Nakazawa, Popel, Popović, Shmatova, Koehn, Haddow, Kocmi and Monz2023), and it has become clear that new forms of best practices are required. Therefore, human evaluation of MT at the document level has attracted much attention as it allows for a more thorough examination of the output quality with intersentential context (Toral et al., Reference Toral, Castilho, Hu, Way, Bojar, Chatterjee, Federmann, Fishel, Graham, Haddow, Huck, Yepes, Koehn, Monz, Negri, Névéol, Neves, Post, Specia, Turchi and Verspoor2018; Läubli et al., Reference Läubli, Sennrich, Volk, Riloff, Chiang, Hockenmaier and Tsujii2018). Research has been carried out in order to find out how much context needs to be shown to translators (Castilho et al. Reference Castilho, Popović, Way, Calzolari, Béchet, Blache, Choukri, Cieri, Declerck, Goggi, Isahara, Maegaard, Mariani, Mazo, Moreno, Odijk and Piperidis2020), as well as different methodologies for adding context into human evaluation of MT (Castilho, Reference Castilho, Barrault, Bojar, Bougares, Chatterjee, Costa-jussà, Federmann, Fishel, Fraser, Graham, Guzman, Haddow, Huck, Yepes, Koehn, Martins, Morishita, Monz, Nagata, Nakazawa and Negri2020, Freitag et al. Reference Freitag, Foster, Grangier, Ratnakar, Tan and Macherey2021a; Freitag et al., Reference Freitag, Foster, Grangier, Ratnakar, Tan and Macherey2021a).

The findings from Castilho et al. (Reference Castilho, Popović, Way, Calzolari, Béchet, Blache, Choukri, Cieri, Declerck, Goggi, Isahara, Maegaard, Mariani, Mazo, Moreno, Odijk and Piperidis2020) revealed that over 33% of the tested sentences needed more context than just the sentence itself for effective translation or evaluation, and from those, 23% required more than two preceding sentences to be adequately assessed.Footnote ^b This is often described as the “context span” needed for disambiguation. Challenges impeding translation included ambiguity, terminology, and gender agreement. Equipped with these findings, the author conducted a series of experiments, testing different methodologies for adding context to human evaluation of MT.

Castilho et al. (Reference Castilho, Barrault, Bojar, Bougares, Chatterjee, Costa-jussà, Federmann, Fishel, Fraser, Graham, Guzman, Haddow, Huck, Yepes, Koehn, Martins, Morishita, Monz, Nagata, Nakazawa and Negri2020) looked into methodologies and inter-annotator agreement (IAA) between single-sentence and document-level evaluation setups. In the first study, translators were tasked with evaluating the MT output in terms of fluency, adequacy (using a Likert scale), ranking, and error annotation. Two evaluation setups were employed: (i) translators assigned a single score per isolated sentence, and (ii) translators assigned a single score per entire document. The results indicated that IAA scores for the document-level setup reached negative levels, and the satisfaction level of translators with this methodology was notably low. Despite this, the document-level setup successfully mitigated cases of misevaluation that occurred in the single-sentences setup (e.g., for sentences where there is an ambiguity about translation when the sentence is shown in isolation but not when intersentential context is provided).

Building on these results, Castilho (Reference Castilho, Belz, Agarwal, Graham, Reiter and Shimorina2021) modified the document-level setup and repeated the experiment with a larger group of translators. The study compared IAA in the evaluation of: (i) random isolated single sentences, (ii) individual sentences evaluated with access to the full document, and (iii) full documents. The findings demonstrated that a methodology where translators assess individual sentences within the context of a document (ii) achieved a satisfactory level of IAA compared to the random single-sentence methodology (i). Conversely, a methodology where translators assigned a single score per document (iii) showed a significantly lower level of IAA. The author illustrated that the approach of assigning one score per sentence in context effectively avoided misevaluation cases, which are prevalent in random-sentence-based evaluation setups. Moreover, she argued that the higher IAA in the random single-sentence setup might be attributed to raters accepting translations when adequacy is ambiguous but the translation is potentially correct, particularly if it is fluent. Consequently, the author recommended avoiding the single random sentence evaluation method, emphasising the heightened risk of misevaluation, particularly when assessing the quality of NMT systems due to their improved fluency level.

Still with regard to context span, Castilho et al. (Reference Castilho, Popović, Way, Calzolari, Béchet, Blache, Choukri, Cieri, Declerck, Goggi, Isahara, Maegaard, Mariani, Mazo, Moreno, Odijk and Piperidis2020) found that the different issues reported in the DELA corpus (Castilho et al. Reference Castilho, Cavalheiro Camargo, Menezes, Way, Barrault, Bojar, Bougares, Chatterjee, Costa-jussa, Federmann, Fishel, Fraser, Freitag, Graham, Grundkiewicz, Guzman, Haddow, Huck, Yepes, Koehn, Kocmi, Martins, Morishita and Monz2021) required differently sized context spans to be solved. Interestingly, the domains of the data also played a major role in determining the required size of the context span. For example, the subtitle domain (TED talks) required the longest context span to solve the issue of grammatical number.

Finally, when testing shorter context spans that contain the issues from the DELA corpus, Castilho et al. (Reference Castilho, Mallon, Meister, Yue, Nurminen, Brenner, Koponen, Latomaa, Mikhailov, Schierl, Ranasinghe, Vanmassenhove, Vidal, Aranberri, Nunziatini, Escartín, Forcada, Popovic, Scarton and Moniz2023) found that the position of the context span (before or after the source sentence) did not seem to affect the results much, although for lexical ambiguity, gender and number the most correct translations happen when the context is positioned before the sentence being translated. Interestingly, single sentences that contained the cue for the solution of the issues were not always enough for the systems to translate them correctly.

In 2019, the Conference for Machine Translation (WMT) also ventured into document-level human assessment for the news domain (Barrault et al., Reference Barrault, Bojar, Costa-jussà, Federmann, Fishel, Graham, Haddow, Huck, Koehn, Malmasi, Monz, Müller, Pal, Post, Zampieri, Bojar, Chatterjee, Federmann, Fishel, Graham, Haddow, Huck, Yepes, Koehn, Martins, Monz, Negri, Névéol, Neves, Post, Turchi and Verspoor2019). Employing the direct assessment (DA) methodology (Graham et al., Reference Graham, Baldwin, Dowling, Eskevich, Lynn, Tounsi, Matsumoto and Prasad2016), crowdworkers were tasked with assigning scores (ranging from 0 to 100) to individual sentences. Raters were instructed to evaluate (i) randomly selected segments, (ii) consecutive segments in their original order, and (iii) entire texts. Subsequently, in the following year, WMT20 adopted a modified approach by expanding the contextual span to encompass full documents. Raters were required to evaluate specific segments while having access to the complete document and to assess the overall translation quality of the content (Barrault et al., Reference Barrault, Biesialska, Bojar, Costa-jussà, Federmann, Graham, Grundkiewicz, Haddow, Huck, Joanis, Kocmi, Koehn, Lo, Ljubešić, Monz, Morishita, Nagata, Nakazawa, Pal, Post, Zampieri, Barrault, Bojar, Bougares, Chatterjee, Costa-jussà, Federmann, Fishel, Fraser, Graham, Guzman, Haddow, Huck, Yepes, Koehn, Martins, Morishita, Monz, Nagata, Nakazawa and Negri2020). Since 2022, the human evaluation is performed as a source-based (“bilingual”) DA + SQM (Scalar Quality Metrics) of individual segments in document context. Annotators are presented with the entire translated document snippet (typically about 10 sentences) randomly selected from competing systems with additional static contexts, and asked to rate the translation of individual segments and then the entire snippet on sliding scales between 0 and 100. However, the slider is presented to the annotator with seven labelled tick marks, thus incorporating the SQM component (Kocmi et al., Reference Kocmi, Matsushita, Federmann, Koehn, Barrault, Bojar, Bougares, Chatterjee, Costa-jussà, Federmann, Fishel, Fraser, Freitag, Graham, Grundkiewicz, Guzman, Haddow, Huck, Jimeno Yepes, Kocmi, Martins, Morishita, Monz, Nagata, Nakazawa, Negri, Névéol, Neves, Popel, Turchi and Zampieri2022).

The Metrics Shared Task, running since 2008, has compared a number of evaluation metrics submitted by the community. Until 2020, the Metrics Shared Task used the official human scores from the WMT News Translation Task to evaluate the submitted metrics. From 2021 onward (Freitag et al. Reference Freitag, Rei, Mathur, Lo, Stewart, Foster, Lavie, Bojar, Barrault, Bojar, Bougares, Chatterjee, Costa-jussa, Federmann, Fishel, Fraser, Freitag, Graham, Grundkiewicz, Guzman, Haddow, Huck, Yepes, Koehn, Kocmi, Martins, Morishita and Monz2021b), the organisers have also collected their own human evaluation for three language pairs from professional translators via Multidimensional Quality Metrics (MQM; Lommel, Uszkoreit, and Burchardt, Reference Lommel, Uszkoreit and Burchardt2014), a framework for identifying errors in translations within an error typology, with severity rankings (Freitag et al., Reference Freitag, Mathur, Lo, Avramidis, Rei, Thompson, Kocmi, Blain, Deutsch, Stewart, Zerva, Castilho, Lavie, Foster, Koehn, Haddow, Kocmi and Monz2023). Annotators are given access to the full document context when performing these MQM annotations.

Focusing on the broader definition of context, Licht et al. (Reference Licht, Gao, Lam, Guzman, Diab, Koehn, Duh and Guzmán2022) proposed XSTS, a Crosslingual Semantic Textual Similarity scoring approach for human evaluation of machine translation. They propose it in the setting of low-resource translation, where the main goal may be on ensuring that translations are adequate, with fluency potentially a secondary concern. Additionally, it ties to other questions of the context in which translation is used, such as social media text, where it may be more appropriate to score some level of faithfulness (i.e., not penalising grammatical disfluencies or linguistic variation that appears naturally in the source) rather than standard measures of fluency.

Finally, another prevalent approach to evaluating translation quality at the document level involves the use of test suites, as they gauge the model’s proficiency in translating specific discourse-level phenomena including but not limited to anaphora, deixis, ellipsis, lexical ambiguity and consistency (Rios Gonzales, Mascarell, and Sennrich, Reference Rios Gonzales, Mascarell, Sennrich, Bojar, Buck, Chatterjee, Federmann, Graham, Haddow, Huck, Yepes, Koehn and Kreutzer2017; Bawden et al., Reference Bawden, Sennrich, Birch, Haddow, Walker, Ji and Stent2018; Guillou et al., Reference Guillou, Hardmeier, Lapshinova-Koltunski, Loáiciga, Bojar, Chatterjee, Federmann, Fishel, Graham, Haddow, Huck, Yepes, Koehn, Monz, Negri, Névéol, Neves, Post, Specia, Turchi and Verspoor2018; Müller et al., Reference Müller, Rios, Voita, Sennrich, Bojar, Chatterjee, Federmann, Fishel, Graham, Haddow, Huck, Yepes, Koehn, Monz, Negri, Névéol, Neves, Post, Specia, Turchi and Verspoor2018; Voita, Sennrich, and Titov, Reference Voita, Sennrich, Titov, Korhonen, Traum and Màrquez2019b; Cai and Xiong, Reference Cai, Xiong, Liu, Xiong, Ge and Zhang2020, i.a.). Some of these also focus specifically on issues related to bias, such as the translation of gendered pronouns, a topic at the intersection of context (due to anaphora resolution, for example) and bias (Saunders, Sallis, and Byrne, Reference Saunders, Sallis, Byrne, Costa-jussà, Hardmeier, Radford and Webster2020a). Castilho et al. (Reference Castilho, Mallon, Meister, Yue, Nurminen, Brenner, Koponen, Latomaa, Mikhailov, Schierl, Ranasinghe, Vanmassenhove, Vidal, Aranberri, Nunziatini, Escartín, Forcada, Popovic, Scarton and Moniz2023) used a test suite covering a number of context-related issues, including grammatical gender, to test different NMT systems and a ChatGPTFootnote ^c model. The findings indicated that all systems encountered difficulties in accurately translating grammatical gender, even when provided with contextual information. Additionally, the authors observed a consistent misidentification of the speaker’s gender by the GPT model when prompted to determine the gender in sentences without explicit speaker indications, particularly when specific adjectives were employed.

While test suites fall somewhere between the categories of human evaluation and automatic evaluation, we discuss them in this section because of the large amount of human labour typically involved in creating them, though in their use, they may either involve human annotation or automatic scoring. These test suites can incorporate both accurate and inaccurate translations corresponding to particular phenomena, thereby enabling the assessment of the model’s accuracy in identifying the correct translation. Nonetheless, the availability of test suites delineated explicitly at the document level remains relatively limited (Vojtěchová et al., Reference Vojtěchová, Novák, Klouček, Bojar, Bojar, Chatterjee, Federmann, Fishel, Graham, Haddow, Huck, Yepes, Koehn, Martins, Monz, Negri, Névéol, Neves, Post, Turchi and Verspoor2019; Rysová et al., Reference Rysová, Rysová, Musil, Poláková, Bojar, Bojar, Chatterjee, Federmann, Fishel, Graham, Haddow, Huck, Yepes, Koehn, Martins, Monz, Negri, Névéol, Neves, Post, Turchi and Verspoor2019; Castilho et al., Reference Castilho, Cavalheiro Camargo, Menezes, Way, Barrault, Bojar, Bougares, Chatterjee, Costa-jussa, Federmann, Fishel, Fraser, Freitag, Graham, Grundkiewicz, Guzman, Haddow, Huck, Yepes, Koehn, Kocmi, Martins, Morishita and Monz2021). Post and Junczys-Dowmunt (Reference Post and Junczys-Dowmunt2023) propose generative variants of existing contrastive test sets that are better able to discriminate different levels of quality for document-level systems. They have stated the importance of using “discourse-dense” datasets to evaluate document systems as they have shown that “the gap between translating without and with context is much larger on the discourse-dense subset” (Post and Junczys-Dowmunt, Reference Post and Junczys-Dowmunt2023). Similarly focusing on “discourse-dense” data, Wicks and Post (Reference Wicks, Post, Koehn, Haddow, Kocmi and Monz2023) propose an approach to building test sets that specifically focus on sentences that require additional intersentential context in order to be correctly translated. These test sets could then be used for both human and automatic evaluations that target discourse and document-level phenomena. Fernandes et al. (Reference Fernandes, Yin, Liu, Martins, Neubig, Rogers, Boyd-Graber and Okazaki2023b) release benchmark data and taggers for 14 language pairs to identify phenomena that require discourse and intersentential context to disambiguate, enabling evaluation of these phenomena. They also modify Conditional Cross-Mutual Information—which they had previously used to identify the extent to which context is used by context-aware models (Fernandes et al., Reference Fernandes, Yin, Neubig, Martins, Zong, Xia, Li and Navigli2021)—to a pointwise version for identifying particular words that are strongly context-dependent.

Another area where context and evaluation meet—when we consider a broad definition of context—is the topic of situated or task-specific evaluation of machine translation. We briefly describe a few examples of this here. Zouhar et al. (Reference Zouhar, Popel, Bojar and Tamchyna2021) evaluate NMT output by examining post-editing time; since post-editing is a real use case for translation, examining the time required to post-edit a translation provides a task-specific view of the quality of translation in the context in which it may be used (e.g., by a language service provider). Where the use case of interest is user engagement or understanding, NMT and human translation can be extrinsically compared by measuring user behaviour in A/B tests (Kovacs and DeNero, Reference Kovacs, DeNero, Duh and Guzmán2022). Similarly, Schmidtke (Reference Schmidtke2016) used customer feedback in the format of asking whether users of help pages had found the information they needed in order to measure MT success and allow for improvements to the translations. In an in-vivo study, Mehandru et al. (Reference Mehandru, Agrawal, Xiao, Gao, Khoong, Carpuat, Salehi, Bouamor, Pino and Bali2023) examined NMT quality estimation and backtranslation as tools for helping physicians using NMT to detect clinically harmful translations in discharge instructions. Roelofsen et al. (Reference Roelofsen, Esselink, Mende-Gillings, de Meulder, Sijm, Smeijers and Shterionov2021) provide methodological recommendations for performing evaluations of avatars for text-to-sign language translation, with a focus on two aspects of context: the community of potential users of the technology and the format in which the evaluation is performed (online).

In summary: As we explore the complexities associated with the integration of context into MT systems, the landscape of human evaluation emerges as an essential area requiring rigorous examination. Although human scores produced following current best practices have traditionally served as fundamental criteria for assessing MT performance, the exclusive dependence on these segment-level scores may not provide a sufficiently comprehensive framework to address the intricate challenges introduced by context-aware translations.

We have seen that current best practices often gravitate towards expert assessments of adequacy and fluency, and comparative analyses, with different methodologies. Moreover, the best methodologies to run context-aware human evaluations are still to be identified. The best practices of the future will need to incorporate context, in order to remain a suitable gold standard for evaluating context-aware capabilities of MT models.

The future direction of human evaluation within the context-aware MT domain is expected to present both challenges and transformative opportunities. This evolution is expected to advance improvements in translation and is also likely to have overlap and intersections with questions about bias and ethics.

3.2 Context and automatic evaluation

As mentioned previously, standard automatic evaluation metrics fail to evaluate a number of important features in the quality of longer texts since they do not seek out or measure specific discourse phenomena in the translation (Maruf et al., Reference Maruf, Saleh and Haffari2021). Therefore, developing “more robust, automatic, and interpretable document-level translation metrics” is essential (Jin et al., Reference Jin, He, May, Ma, Bouamor, Pino and Bali2023, p.15253).

Post and Junczys-Dowmunt (Reference Post and Junczys-Dowmunt2023) have identified three impediments to moving the MT field to context-aware translation, one of them being the current state of MT evaluation metrics. According to the authors, “document-level phenomena are rare enough that gains there are unlikely to be reflected with current test-set level metrics, leading to a perception, perhaps, of diminishing returns, and to the idea that the effort is not worthwhile when all costs are considered”—they argue that without an appropriate way of measuring improvements, we can’t expect to see or motivate improvement. While the human evaluation improvements we discussed in the prior section are important, automatic metrics are used in the day-to-day development and rapid iteration of MT models, so it is vital to have appropriate ones available.

Efforts have been made to extend traditional automatic MT metrics to documents by merging the sentences within a document into a single expanded sentence (concatenation) and then applying the traditional metric to evaluate it (Wong and Kit, Reference Wong, Kit, Tsujii, Henderson and Paşca2012; Gong, Zhang, and Zhou, Reference Gong, Zhang, Zhou, Webber, Carpuat, Popescu-Belis and Hardmeier2015; Xiong et al., Reference Xiong, He, Wu and Wang2019; Liu et al., Reference Liu, Gu, Goyal, Li, Edunov, Ghazvininejad, Lewis and Zettlemoyer2020; Saunders, Stahlberg, and Byrne, Reference Saunders, Stahlberg, Byrne, Jurafsky, Chai, Schluter and Tetreault2020b, i.a.). However, “this type of evaluation is still limited as it does not consider discourse aspects of the text” (Maruf et al., Reference Maruf, Saleh and Haffari2021, p. 22). In particular, this approach does not directly address important issues like consistency in terminology, anaphora and other discourse features, and overall document coherence. Some of them may be handled indirectly, for example, if the reference text being evaluated against is consistent in its terminology use and the MT output is not, the MT output may be penalised. However, any such penalty may not be directly observable or actionable as being context-related.

Several automatic metrics specifically designed for document-level MT have been introduced in the past years. Jiang et al. (Reference Jiang, Liu, Ma, Zhang, Yang, Huang, Sennrich, Cotterell, Sachan, Zhou, Carpuat, de Marneffe and Meza Ruiz2022) introduced BlonDe, a document-level automatic metric that calculates the similarity-based F1 measure of discourse-related spans across categories, taking discourse coherence into consideration. The authors compare BlonDe with 11 other metrics, including standard sentence-level metrics, document-level metrics,Footnote ^d and embedding-based metrics, evaluating them all in terms of their Pearson correlation with human assessment. The human assessment of the metric was conducted with both a sentence-level evaluation, where isolated sentences are shown, and a document-level evaluation where entire documents are shown and raters evaluate the overall quality of sequential blocks of sentences. Results showed that BlonDe is better than the other metrics at distinguishing between context-aware and context-agnostic MT systems.

The method proposed by Vernikos et al. (Reference Vernikos, Thompson, Mathur, Federico, Koehn, Barrault, Bojar, Bougares, Chatterjee, Costa-jussà, Federmann, Fishel, Fraser, Freitag, Graham, Grundkiewicz, Guzman, Haddow, Huck, Jimeno Yepes, Kocmi, Martins, Morishita, Monz, Nagata, Nakazawa, Negri, Névéol, Neves, Popel, Turchi and Zampieri2022) extends pre-trained MT metrics to integrate context by constructing sentence representations from context. The approach is applied to four widely used pre-trained metrics: BERTScore (Zhang et al., Reference Zhang, Kishore, Wu, Weinberger and Artzi2020), Prism (Thompson and Post, Reference Thompson, Post, Webber, Cohn, He and Liu2020), COMET (Rei et al. Reference Rei, Stewart, Farinha, Lavie, Webber, Cohn, He and Liu2020), and the reference-free metric COMET-QE (Rei et al., Reference Rei, Stewart, Farinha, Lavie, Webber, Cohn, He and Liu2020). Evaluation involved testing the system-level correlation with human judgments using MQM judgments from the WMT21 metrics task (Freitag et al., Reference Freitag, Rei, Mathur, Lo, Stewart, Foster, Lavie, Bojar, Barrault, Bojar, Bougares, Chatterjee, Costa-jussa, Federmann, Fishel, Fraser, Freitag, Graham, Grundkiewicz, Guzman, Haddow, Huck, Yepes, Koehn, Kocmi, Martins, Morishita and Monz2021b), along with test suites. Comparisons were made with the sentence-level versions of each metric and the BlonDe metric (Jiang et al., Reference Jiang, Liu, Ma, Zhang, Yang, Huang, Sennrich, Cotterell, Sachan, Zhou, Carpuat, de Marneffe and Meza Ruiz2022). The results demonstrated that incorporating document-level context in pre-trained metrics enhances correlation with human judgments. Additionally, the study revealed that BlonDe shows lower performance compared to both pre-trained metrics and the proposed document-level extensions. However, it is worth noting that although document context is used, this metric, now commonly called Doc-COMET, still scores single sentences one at a time.

Zhao et al. (Reference Zhao, Strube, Eger, Vlachos and Augenstein2023) integrate discourse and BERT representations to introduce DiscoScore, a discourse metric, allowing for customisation with nouns or semantic entities. The metric was compared to 16 standard and discourse metrics using various test sets and human judgments from MQM. The study reveals that DiscoScore shows robust system-level correlation with human ratings, surpassing the current state-of-the-art BARTScoreFootnote ^e in coherence, factual consistency, and other aspects, with an average improvement of over 10 correlation points. However, while it performs well at the system level over whole test sets, when it is evaluated at finer-grained levels such as the document, DiscoScore does not outperform BARTScore.

Raunak et al. (Reference Raunak, Kocmi, Post, Koehn, Haddow, Kocmi and Monz2023a) used two versions of the COMET metric, COMET20-QE (Rei et al., Reference Rei, Stewart, Farinha, Lavie, Webber, Cohn, He and Liu2020) and COMETKiwi (Rei et al., Reference Rei, Treviso, Guerreiro, Zerva, Farinha, Maroti, C. de Souza, Glushkova, Alves, Coheur, Lavie, Martins, Koehn, Barrault, Bojar, Bougares, Chatterjee, Costa-jussà, Federmann, Fishel, Fraser, Freitag, Graham, Grundkiewicz, Guzman, Haddow, Huck, Jimeno Yepes, Kocmi, Martins, Morishita, Monz, Nagata, Nakazawa, Negri, Névéol, Neves, Popel, Turchi and Zampieri2022), to evaluate the performance of a simpler approach: to concatenate sentences in both the source and hypothesis. The authors present SLIDE (SLIding Document Evaluator), a document-level metric which operates by processing blocks of sentences through a sliding window across each document in the test set, with each chunk being input into an unaltered, readily available quality estimation model. The authors compared the results with Doc-COMET (Vernikos et al., Reference Vernikos, Thompson, Mathur, Federico, Koehn, Barrault, Bojar, Bougares, Chatterjee, Costa-jussà, Federmann, Fishel, Fraser, Freitag, Graham, Grundkiewicz, Guzman, Haddow, Huck, Jimeno Yepes, Kocmi, Martins, Morishita, Monz, Nagata, Nakazawa, Negri, Névéol, Neves, Popel, Turchi and Zampieri2022) on the ContraPro dataset (Müller et al., Reference Müller, Rios, Voita, Sennrich, Bojar, Chatterjee, Federmann, Fishel, Graham, Haddow, Huck, Yepes, Koehn, Monz, Negri, Névéol, Neves, Post, Specia, Turchi and Verspoor2018). They also checked pairwise system ranking accuracy against the WMT22-MQM annotations (Freitag et al., Reference Freitag, Rei, Mathur, Lo, Stewart, Avramidis, Kocmi, Foster, Lavie, Martins, Koehn, Barrault, Bojar, Bougares, Chatterjee, Costa-jussà, Federmann, Fishel, Fraser, Freitag, Graham, Grundkiewicz, Guzman, Haddow, Huck, Jimeno Yepes, Kocmi, Martins, Morishita, Monz, Nagata, Nakazawa, Negri, Névéol, Neves, Popel, Turchi and Zampieri2022). Results showed that SLIDE achieved higher pairwise system ranking accuracy compared to its sentence-level baseline, and in certain instances, it even closed the gap with reference-based metrics. This highlights the potential of using quality estimation metrics alongside or instead of reference-based ones.

The evaluation of paragraphs was also a focus of study. Deutsch et al. (Reference Deutsch, Juraska, Finkelstein, Freitag, Koehn, Haddow, Kocmi and Monz2023) examined the possibility of using neural metrics, originally trained at the sentence level, for the evaluation of paragraphs. The evaluation of the metrics is calculated via a meta-evaluation with pairwise accuracy correlations from WMT22 MQM scores. Results show that their approach trained on paragraph-level data is as effective as sentence-level metrics, but does not necessarily out-perform them. Moreover, is important to notice that their evaluation was limited to reference-based metrics.

In summary: As can be seen, the landscape of automatic evaluation for MT has witnessed notable advancements with the introduction of various metrics designed for document and paragraph-level assessments in the past few years. Efforts to extend traditional sentence-level metrics to documents have been made, but they often fall short in capturing discourse aspects of the text (Maruf et al., Reference Maruf, Saleh and Haffari2021). SLIDE, a document-level metric, leverages source context effectively, suggesting its potential to provide information akin to human references. Additionally, studies on evaluating paragraphs using neural metrics highlight comparable effectiveness but underscore the need for further exploration, particularly beyond reference-based metrics.

Recent introductions showcase promising performance at the system level. However, challenges persist, as some metrics, including BlonDe, exhibit lower efficacy compared to pre-trained metrics or document-level extensions.

Moreover, what the community still needs to understand is the limitation of each of these proposed approaches. There is also the need to refine existing metrics, address limitations in capturing nuanced linguistic phenomena, and explore more comprehensive evaluation strategies. All of these are critical for advancing the field of automatic MT evaluation. Future research should aim to develop metrics that align more closely with human judgments across diverse linguistic contexts and encompass a broader understanding of document- and paragraph-level translations. Another fruitful area of research may be in explainable metrics, which could provide more information to system developers about what areas of translation quality still need improvement.

4.1 LLMs for translation

The most recent edition of the Conference on Machine Translation marked the first time that LLMs were used directly as submissions in the shared task on General Machine Translation (Kocmi et al., Reference Kocmi, Avramidis, Bawden, Bojar, Dvorkovich, Federmann, Fishel, Freitag, Gowda, Grundkiewicz, Haddow, Koehn, Marie, Monz, Morishita, Murray, Nagata, Nakazawa, Popel, Popović, Shmatova, Koehn, Haddow, Kocmi and Monz2023). That task found that GPT-4Footnote ^f performed in the top cluster for systems translating into English, while it dropped somewhat in the out-of-English direction.

While there has been wide interest in examining LLM performance on a wide range of tasks, there is a particular reason that LLM performance on MT tasks is closely tied to this work on context: the use of LLMs that permit very large intersentential context windows. Pawar et al. (Reference Pawar, Tonmoy, Zaman, Jain, Chadha and Das2024) provide a survey of techniques used for increasing the context span that is used by LLMs. Given our discussion of the role of intersentential or document context in MT, it is clear that the approaches from LLMs provide an opportunity to explore these questions. This can be explored both from the perspective of using LLMs for translation, as well as when or how such approaches could be incorporated into MT systems. A number of works have begun to examine how LLMs can be used as translation systems, examining their performance either at the sentence level or beyond. Other work has touched on other types of context examined in this survey.

Examining ChatGPT’s performance as an MT system across a wide range of languages and comparing against other LLMs and dedicated MT systems, Robinson et al. (Reference Robinson, Ogayo, Mortensen, Neubig, Koehn, Haddow, Kocmi and Monz2023) observe that LLMs exhibit competitive behaviour on some high-resource languages, but underperform dedicated MT systems for the majority of low-resource languages. These benchmarks are intended to provide end users with an understanding of the level of quality they might expect for translation into their languages using LLMs. The paper finds that the number of existing Wikipedia pages in a given language is a strong predictor of LLM performance as a translation system for that language. Manakhimova et al. (Reference Manakhimova, Avramidis, Macketanz, Lapshinova-Koltunski, Bagdasarov, Möller, Koehn, Haddow, Kocmi and Monz2023) examine the performance of GPT-4 on fine-grained linguistic nuances, explored through the use of test suites. They observe performance competitive with dedicated MT systems, though they also note that there are still areas with room for improvement.

In their submission to the shared task, the Lan-Bridge team explored different natural language prompts for using GPT-3.5 for translation between Chinese and English (Wu and Hu, Reference Wu, Hu, Koehn, Haddow, Kocmi and Monz2023). These included prompts that focused on sentence-by-sentence translation, as well as ones that asked the model to attend to intersentential context as well, concluding that the latter improved translation quality according to automatic metrics. Wang et al. (Reference Wang, Lyu, Ji, Zhang, Yu, Shi, Tu, Bouamor, Pino and Bali2023) also examine a variety of prompts for document-level translation with LLMs, comparing several MT models and testing their performance on the discourse-focused test suite from Voita et al. (Reference Voita, Sennrich, Titov, Korhonen, Traum and Màrquez2019b), finding recent performance improvements on discourse in document-level translation with GPT-4. Karpinska and Iyyer (Reference Karpinska, Iyyer, Koehn, Haddow, Kocmi and Monz2023) consider the task of literary translation, prompting GPT-3.5 to perform sentence-level translation, sentence-level translation with context, and paragraph-level translation. Human evaluation finds the best performance with paragraph-level translation, but still notes critical errors even in these better translations. Also focusing in the literature domain, Thai et al. (Reference Thai, Karpinska, Krishna, Ray, Inghilleri, Wieting, Iyyer, Goldberg, Kozareva and Zhang2022) explored using LLMs to perform post-editing in order to mitigate discourse errors, after finding a large gap in quality between MT and human literary translation in human evaluations. Much of the work on LLMs has focused on prompting and few-shot learning (Briakou, Cherry, and Foster, Reference Briakou, Cherry, Foster, Rogers, Boyd-Graber and Okazaki2023; Raunak, Menezes, and Awadalla, Reference Raunak, Menezes, Awadalla, Bouamor, Pino and Bali2023b, i.a.), but in addition to these, Zhang et al. (Reference Zhang, Rajabi, Duh, Koehn, Koehn, Haddow, Kocmi and Monz2023) explore finetuning of LLMs for translation, finding performance improvements even when finetuning only a very small fraction of the model’s parameters. Iyer et al. (Reference Iyer, Chen, Birch, Koehn, Haddow, Kocmi and Monz2023) consider the translation of ambiguous words with LLMs, including few-shot (or in-context) learning, where additional examples are fed to the model in order for it to use that contextual information for disambiguation.

On the topic of formality control discussed in Section 2.3, Marrese-Taylor et al. (Reference Marrese-Taylor, Wang, Matsuo, Koehn, Haddow, Kocmi and Monz2023) explore prompting LLMs to handle formality control for English to Japanese translation, finding it to be a viable approach. Touching on the topics discussed in Section 2.4, several papers have considered how to use LLMs to incorporate terminology into translation. Moslem et al. (Reference Moslem, Romani, Molaei, Kelleher, Haque, Way, Koehn, Haddow, Kocmi and Monz2023) explore ways of using LLMs for improving terminology translation, both by using the LLM to produce synthetic bitext with the terminology, as well as by using the LLM to postedit MT output to incorporate desired terminology. On the same topic, Bogoychev and Chen (Reference Bogoychev, Chen, Koehn, Haddow, Kocmi and Monz2023) prompt LLMs to revise existing translations so that they incorporate terminology constraints.

In general, the papers discussed so far have looked at evaluating or improving performance of LLMs as translation systems. Petrick et al. (Reference Petrick, Herold, Petrushkov, Khadivi, Ney, Koehn, Haddow, Kocmi and Monz2023) explore ways of fusing document-level language models with NMT systems, including LLMs. This provides an example of ways we might see knowledge from LLMs incorporated into dedicated MT systems, an avenue that could be explored in parallel to that of treating LLMs as MT systems.

In summary: The topic of LLMs has seen growing interest in the field of MT. This initial interest has often focused on evaluating LLMs as translation systems, exploring how to modify prompts or provide additional context to improve translation, as well as ways to combine them with MT systems. We expect that this is an area that will continue to see growth and exploration in the immediate future, including improved understanding of suitable prompts and the data already incorporated in LLMs (Briakou et al., Reference Briakou, Cherry, Foster, Rogers, Boyd-Graber and Okazaki2023), methods of finetuning, or methods of combining LLMs and MT systems. As with all work on LLMs, a challenge for the research field will be how to ensure reproducibility and fair evaluation, given a landscape of proprietary models where full information about training data may not be available.

4.2 LLMs for evaluation

As previously discussed, we have seen a growing interest in the application of LLMs to the domain of automatic translation, among many other avenues of exploration. Beyond translation, researchers have explored the potential of LLMs to assess the quality of automatic translations as well. An example is the work by Kocmi and Federmann (Reference Kocmi, Federmann, Nurminen, Brenner, Koponen, Latomaa, Mikhailov, Schierl, Ranasinghe, Vanmassenhove, Vidal, Aranberri, Nunziatini, Escartín, Forcada, Popovic, Scarton and Moniz2023b) who introduced GEMBA, a GPT-based metric designed for translation evaluation. The authors compared the metric in two forms—with a reference translation and without—against COMET and BLEURT scores, and compared the results of the MQM-based human labels from the WMT22 Metrics Shared task. Results on the system level show that the metric achieved state-of-the-art performance even outperforming traditional metrics and human evaluations in some cases. The authors state that this is a great step towards document-level evaluation as these models have the ability to use much larger context windows.

Another metric leveraging LLMs is the one introduced by Fernandes et al. (Reference Fernandes, Deutsch, Finkelstein, Riley, Martins, Neubig, Garg, Clark, Freitag, Firat, Koehn, Haddow, Kocmi and Monz2023a). Their work investigates the capabilities of these models to perform automatic MQM-style error annotation and score prediction. The authors introduce AutoMQM, a prompt technique for evaluating translation quality that instructs the model to identify and categorise errors rather than asking it to assign a numerical score. Their results show that prompting LLMs to predict a quality score does not improve performance over the trained standard automatic metrics. Moreover, the authors claim that without fine-tuning, AutoMQM can provide interpretable results through error spans that correlate well with human annotations. GEMBA-MQM (Kocmi and Federmann, Reference Kocmi, Federmann, Koehn, Haddow, Kocmi and Monz2023a), a follow-up work to Kocmi and Federmann (Reference Kocmi, Federmann, Nurminen, Brenner, Koponen, Latomaa, Mikhailov, Schierl, Ranasinghe, Vanmassenhove, Vidal, Aranberri, Nunziatini, Escartín, Forcada, Popovic, Scarton and Moniz2023b), also aims to use LLMs to identify MQM-style error spans, though Lo, Larkin & Knowles (Reference Lo, Larkin, Knowles, Koehn, Haddow, Kocmi and Monz2023) note that it may struggle to distinguish between lower-quality systems (which are distinguishable by other existing metrics).

In the area of human judgements on question-answering systems, but potentially applicable to NMT evaluation, Wadhwa et al. (Reference Wadhwa, Chen, Li and Durrett2023) use LLMs to rescale human judgments by analysing human explanations of the scores they assigned. The methodology involves inputting both the label and its corresponding explanation provided by annotators into the LLM, generating a score on a scale from 0 to 100 based on a rubric. The findings demonstrate a strong correlation with expert annotations.

Finally, in their recent work, Huang et al. (Reference Huang, Zhang, Geng, Du, Chen and Huang2024) investigate how LLMs leverage source and reference information in evaluating translations, by employing coarse-grained and fine-grained prompts. The authors instruct both open and closed LLMs to predict coarse-grained quality scores like GEMBA, but given different information such as sources and references. Furthermore, the authors adopt the AutoMQM prompt template for fine-grained error detection. The results highlight that reference information enhances system-level accuracy and segment-level correlations. Interestingly, the utilisation of source information is at times counterproductive, suggesting limitations in cross-lingual capability when employing LLMs for evaluating translated sentences.

In summary: As can be seen, the rise of LLMs in the field of MT evaluation shows substantial promise and brings intriguing challenges. These new developments with the capacity of handling more context can reshape the way we evaluate translations. However, as we dive in this emerging area, it becomes clear that there are still aspects that need refinement. The ongoing pursuit of best strategies to use LLMs in MT evaluation will likely be an area of continuing interest. Moreover, the way we test these metrics should be strict and well-defined, with human evaluation playing a part in the process. These metrics should also be evaluated on a wide range of levels of system quality, as well as across diverse language pairs to highlight potential weaknesses in both the approach to evaluation (Lo et al., Reference Lo, Larkin, Knowles, Koehn, Haddow, Kocmi and Monz2023) and the underlying model’s language-specific capabilities (see, e.g., Robinson et al., Reference Robinson, Ogayo, Mortensen, Neubig, Koehn, Haddow, Kocmi and Monz2023). The road ahead involves not only addressing the current limitations but also pushing the boundaries, guiding the field toward a more comprehensive and context-aware era in MT evaluation.