2.1 Before 2010: fewer data, less misspellings

Before the explosion of user-generated data on the Internet, the vast majority of content available on the web (static web pages, journal articles, etc.) was characterised by the fact that the content was moderately well curated. As a result, the amount of data was relatively limited, and the available data contained few misspellings. For this reason, automated text analysis technologies were rarely concerned with the presence of misspellings, if at all. The study of misspellings was confined to the development of automatic correction tools that aid users in producing misspelling-free texts by, for example, correcting typos or applying OCR-produced errors.

Arguably, the first works on misspellings were those by Blair (Reference Blair1960) and Damerau (Reference Damerau1964), which proposed the earliest dictionary-based methods for spelling correction. In this survey, we do not focus on spelling correction per se (we refer the interested reader to Bryant et al. Reference Bryant, Yuan, Qorib, Cao, Ng and Briscoe2023; Hládek et al. Reference Hládek, Staš and Pleva2020; Wang et al. Reference Wang, Wang, Dang, Liu and Liu2021b), but rather on NLP systems designed to be resilient to misspellings. In this context, Vinciarelli (Reference Vinciarelli2005) stands out as one of the pioneering studies, specifically addressing errors introduced by OCR technologies.

Some studies seemed to indicate that the problem of misspellings was not paramount for text classification technologies, at least when these concern the classification by topic of (curated) text documents (Agarwal et al. Reference Agarwal, Godbole, Punjani and Roy2007). The situation differed somewhat when shifting to other, less curated sources, such as emails, blogs, forums and SMS data, or when analysing the output generated by automatic speech recognition engines from call centres. The problem attracted little attention from the research community at the time, and it was not until 2007 that a dedicated workshop, called Analysis for Noisy Unstructured Text Data (AND), emerged and renewed interest in the field (see also Section 7.4).

To the best of our knowledge, the only survey on NLP systems robust to noise was published in 2009 (Subramaniam et al. Reference Subramaniam, Roy, Faruquie and Negi2009). This survey primarily focused on handling noise in OCR scans, blogs, call centre transcriptions and similar sources.

2.2 After 2010: the rise of social networks and deep learning

Since 2010, user-generated content has become increasingly pervasive, mainly due to the revolution of social networks. At the same time, deep learning technologies have taken the world by storm (Krizhevsky et al. Reference Krizhevsky, Sutskever and Hinton2012), not only due to the increase in performance they show off in most NLP tasks (Collobert et al. Reference Collobert, Weston, Bottou, Karlen, Kavukcuoglu and Kuksa2011), but also because of their potential to eliminate the need for manual feature engineering; instead, the neural network itself learns to represent the input. This raises questions about the necessity of a pre-processing step for correcting misspellings beforehand.

The increasing prevalence of misspelt data and the proliferation of NLP technologies have inspired numerous studies analysing the impact of misspellings on state-of-the-art models (Section 4), as well as papers proposing systems that are resilient to misspellings (Section 5).

The study of misspellings in NLP presents significant benefits. The most apparent advantage is the enhancement of performance in any NLP tool, but not only. Systems that are resilient to misspellings are also safer. For reasons discussed later, some misspellings are intentional, designed to evade the scrutiny of content moderation tools or spam filters. Ultimately, the study of misspelling resilience aims to deepen our understanding of language (further discussions are offered in Section 10).

This increased interest in the subject was partially fostered by the work of Belinkov and Bisk (Reference Belinkov and Bisk2018); Heigold et al. (Reference Heigold, Varanasi, Neumann and van Genabith2018); Edizel et al. (Reference Edizel, Piktus, Bojanowski, Ferreira, Grave and Silvestri2019), who showed the performance of different models decrease noticeably in the presence of misspellings. This renewed momentum has led to the appearance of dedicated workshops devoted to studying the phenomenon from the point of view of user-generated content (such as the WNUT workshop seriesFootnote ² ) as well as from the point of view of machine translation (such as the WMT workshop/conference seriesFootnote ³ ); more information about dedicated tasks and datasets can be found in Section 7.4.

Between 2017 and 2022, neural machine translation emerged as the most prolific field in the study of misspellings, closely followed by sentiment analysis.

2.3 After 2020: the advent of LLMs

In recent years, the trend has shifted markedly: whereas earlier research focused heavily on developing methods to combat misspellings in downstream tasks (with a disproportionate emphasis on machine translation), the field has now moved toward a growing number of evaluation papers that analyse the ability of LLMs to handle misspellings.

The advent of LLMs around 2020–2021 has dramatically reshaped the NLP and AI landscape, including research on misspelling resilience. Due to their high computational costs, LLMs are not easily amenable to experimentation with new training methods aimed at improving robustness to misspellings. Moreover, some language models (especially commercial ones) already exhibit strong resistance to misspellings (Moffett and Dhingra Reference Moffett and Dhingra2025). Nevertheless, there is substantial interest in understanding how LLMs handle misspellings in both downstream tasks and more general settings, as evidenced by the relatively high number of papers published on the topic between 2024 and 2025 (see, e.g., Pan et al. Reference Pan, Leng and Xiong2024; Wang et al. Reference Wang, Hu, Hou, Chen, Zheng, Wang, Yang, Ye, Huang, Geng, Jiao, Zhang and Xie2024, Reference Wang, Gu, Wei, Gao, Song and Chen2025; Zhang et al. Reference Zhang, Hao, Li, Zhang and Zhao2024).

Figure 1 shows the publication trends with respect to all NLP papers related to misspellings.

Figure 1.

Publication trends in NLP papers on misspellings (2004–2025).

3.1 Under the lens of linguistics

In linguistics, there are primarily three fundamental viewpoints for models dealing with misspellings, which we cover in this section. The first is that of general linguistics (Section 3.1.1), where some authors have attempted to define and categorise various types of misspellings. The second one is that of sociolinguistics (Section 3.1.2), in which authors analyse misspellings from a social perspective. The last one originates from psycholinguistics (Section 3.1.3), which rather focuses on cognitively relevant aspects of the problem.

3.2 Under the lens of NLP

In the field of NLP, there is no single, clear-cut definition of misspelling. Indeed, the same type of problem (morphological error) is often expressed with different words, such as noise, typo, and spelling mistake.

The most common of these, along with misspelling, is noise, which is defined as any non-standard spelling variation (Nguyen and Grieve Reference Nguyen and Grieve2020). While this definition of noise may seem appropriate, any attempt to provide a universal definition of misspellings would appear contrived and, above all, imprecise.

To tackle this issue and establish clear boundaries around the concept of misspelling, NLP researchers have proposed various categorisations, which draw from different points of view: For example, from the perspective of the word surface, from the point of view of the user who generated them, or pointing up to the methods used to generate misspelt datasets in an experimental setting. The types of misspellings can be fine-grained, where multiple categories of misspellings are defined in detail, or less fine-grained, where fewer, more representative categories are selected. This lack of uniformity, among other things, complicates the search for relevant papers on the subject (which has indeed represented one of the significant challenges we faced when developing this survey).

In this section, we cover some of the main categorisations of misspellings proposed in the literature. In doing so, we note that the problem of misspelling can be approached from diverse viewpoints and thus there are multiple perspectives on this matter. For example, Heigold et al. Reference Heigold, Varanasi, Neumann and van Genabith(2018) have established three categories based on the word surface:

• • character swaps: nice $\rightarrow$ ncie, the position of two subsequent characters is exchanged;

• • word scrambling: absolute $\rightarrow$ alusobte, the order of the characters is permuted with the exception of the first and last one;

• • character flipping: nice $\rightarrow$ nite, one character is replaced by another.

Belinkov and Bisk (Reference Belinkov and Bisk2018) proposed an alternative classification of misspellings into two main categories based on the dataset generation method:

• • Natural misspellings: real misspellings that occur in real-world data spontaneously;

• • Synthetic misspellings: misspellings artificially generated by means of procedural rules.

Note that the differentiation between natural and synthetic misspellings does not establish a clear-cut boundary, as any misspelling generated synthetically could plausibly occur naturally. That is to say, the difference between natural and synthetic misspellings is extrinsic to the word surface form, and regards the mode of generation (spontaneous vs. procedural) while both represent (or mimic) the same underlying phenomenon. Indeed, this distinction is functional to experimental setups and was originally conceived with dataset generation in mind. Synthetic misspellings are more widely used due to the low frequency of natural misspellings, which hinders the collection of large, diverse corpora.Footnote ⁴

What Belinkov and Bisk (Reference Belinkov and Bisk2018) referred to as natural misspellings actually involves lexical lists that provide context for the misspellings, which can be exploited to artificially introduce natural-sounding misspellings into otherwise correct sentences. In such cases, we will refer to a third category of hybrid misspelling, and reserve the term natural misspelling for real-world misspellings found in actual data. Further aspects related to dataset generation will be discussed in Section 7.4.

Several other researchers, including van der Goot et al. (Reference van der Goot, van Noord and van Noord2018) and Nguyen and Grieve (Reference Nguyen and Grieve2020), have emphasised the user’s intention rather than focusing solely on the surface-level characteristics of misspelt words. They propose to distinguish between intentional and unintentional misspellings. For instance, Nguyen and Grieve (Reference Nguyen and Grieve2020) observed that intentional misspellings, such as lengthening a word, are often used to add emphasis to an opinionated statement. An example of this could be the use of the interjection wow in a context where the user wants to emphasise their surprise, as in wooooooooooow. Furthermore, the categorisation of misspellings can be more or less fine-grained; in this respect, van der Goot et al. (Reference van der Goot, van Noord and van Noord2018) have proposed as many as 14 different categories of misspellings, while Heigold et al. (Reference Heigold, Varanasi, Neumann and van Genabith2018) have proposed only 3.

It is thus important to bear in mind that the variability in the terminologies and approaches to this topic—along with the lack of a universal definition for this type of problem—represents one of the greatest challenges faced by NLP researchers. As for our survey, we found it particularly useful to list the types of misspellings for dataset generation (see Section 7.4).

4.1 The harm of synthetic misspellings

Synthetic misspellings are the most commonly employed type of misspelling in the related literature, likely due to the ease with which they can be artificially generated, making them convenient for testing specific approaches without the need for large datasets of naturally occurring errors.

In this section, we review related studies, dividing them into two groups: those conducted before the transformer era (Section 4.1.1) and those focusing on quantifying the impact of misspellings on BERT and other transformer-based models (Section 4.1.2).

4.2 The harm of natural misspellings

Naturally occurring misspellings are invaluable resources for testing NLP applications in real-world settings (Baldwin et al. Reference Baldwin, Cook, Lui, MacKinlay and Wang2013; Belinkov and Bisk Reference Belinkov and Bisk2018). However, they are rarely employed in practice since collecting natural misspellings is anything but a simple task. With a lack of consensus on what precisely a misspelling is, some authors have considered as “natural misspellings” phenomena like the errors generated by second language learners (Náplava et al. Reference Náplava, Popel, Straka and Straková2021) or by OCR scans (Vinciarelli Reference Vinciarelli2005). Having said this, natural misspellings may include all the types of misspellings listed in Section 7.4.2, as long as they are user-generated.

In this section, we review works that try to characterise the presence of natural misspellings (Section 4.2.1) and other studies that test the resiliency of different models to natural misspellings (Section 4.2.2).

4.3 The harm of hybrid misspellings

As recalled from Section 3.2, aside from the synthetic and natural misspellings, there is a third type of misspelling called hybrid that refers to real misspellings that have been artificially injected in different contexts for evaluation purposes (more on this in Section 7.4.3). To our knowledge, the only published record that employs hybrid misspellings to quantify the performance impact on systems that do not handle them is that of Belinkov and Bisk (Reference Belinkov and Bisk2018). In their study, words from error correction databases (such as Wikipedia edits and second language learner corrections) were injected into the IWSLT 2016 machine translation dataset. While hybrid misspellings had less of an impact on machine translation tasks compared to synthetic misspellings, Belinkov and Bisk (Reference Belinkov and Bisk2018) noted that hybrid and natural misspellings are more challenging to evaluate in a rigorous manner.

6.1 Quantifying multilingual coverage in misspelling research

In order to provide a rough estimate of language coverage in the field, we rely on the 42 papers surveyed in this work as a sample that we hope is reasonably representative, while still acknowledging that it may not fully reflect the actual distribution.

As discussed in Section 7, many of the reviewed studies on robustness to misspellings are machine translation studies, which inherently involve multiple languages. However, outside the context of machine translation, multilingual representation remains limited, with non-Western languages being particularly underrepresented.

Among the 42 papers summarised in our methods section, 28 include languages other than English. Nevertheless, only three of these (Malykh et al. Reference Malykh, Logacheva and Khakhulin2018; Namysl et al. Reference Namysl, Behnke and Köhler2020; Zhou et al. Reference Zhou, Zhang, Jin, Peng, Xiao and Cao2020) are not focused on machine translation. The most frequently studied languages beyond English are German (12) and French (11), followed by Czech (5). Other represented languages include Arabic (2), Turkish (2), Spanish (2), Italian (2) and one instance each of Vietnamese, Polish, Dutch, Dagur, Alsatian, Nazrabi, Moldovan, Romanian, Hinglish, Benglish, Japanese, Slovenian and Portuguese.

These data underscore a notable imbalance: while some work does address languages other than English, the vast majority of the world’s languages remain largely underrepresented in misspelling-related research.

6.2 Low-resource languages and cross-lingual approaches

Given that there are more than 7,000 living languages in the world, and that modern approaches—especially neural ones—are both computationally expensive and data-intensive, devising effective solutions for low-resource languages can appear daunting. In this context, cross-lingual approaches (i.e., methods that transfer knowledge from high-resource languages to low-resource ones) represent a particularly promising way out of this problem, if not the only viable one. Most cross-lingual approaches consider the source (typically a resource-rich language on which training is performed) and the target (typically a resource-scarce language on which the model is to be deployed) to belong to the same language family.

One of the cross-lingual approaches that has shown promising results in the context of spelling correction is that of Riabi et al. (Reference Riabi, Sagot and Seddah2021), who trained CharacterBERT using approximately 99,000 sentences in NArabizi (a North African colloquial dialect written in the Latin script). Their results demonstrate that this approach yields competitive performance on the NArabizi Treebank, a testbed for noisy textual inputs, and achieves results comparable to those of models trained on much larger datasets.

Similarly, Aepli and Sennrich Reference Aepli and Sennrich2022) show that introducing character-level misspellings in high-resource source data helps improve performance on part-of-speech (POS) tagging tasks in closely related target languages: Their work focuses on language pairs from closely related branches, including Finnic, Germanic variants, and Western Romance languages.

Bernhard and Dolińska (Reference Bernhard and Dolińska2025) explore POS tagging robustness to misspellings in two low-resource languages, Dagur (a Mongolic language spoken by approximately 130,000 people in northern China) and Alsatian (a Germanic language from northern Europe). They fine-tune neural models on related languages and apply noise-reduction strategies, showing that the proximity between the languages has an impact on zero-shot cross-lingual performance.

6.3 Misspellings and robustness in L1 learner English

Another relevant concept in this context is learner English, that is, English written by speakers whose first language (L1) is not English. Although the language in question remains English, learner English embodies the linguistic influence of diverse cultural and linguistic backgrounds. Nevertheless, this perspective remains largely underrepresented in the literature on misspellings and robustness, despite its ubiquity and practical relevance in many real-world NLP applications.

Mizumoto and Nagata (Reference Mizumoto and Nagata2017) demonstrate that applying a spell checker (i.e., a Double-Step method, see Section 5.3) consistently improves POS tagging accuracy on texts written by native Japanese speakers. Likewise, Miaschi et al. (Reference Miaschi, Brunato, Dell’Orletta and Venturi2022) investigate the impact of L1 learner errors using BERT, focusing on the CItA corpus (a collection of essays written by Italian L1 learners). Their experiments show that BERT’s performance on sentence similarity tasks is variably affected depending on the error category, with misspellings having a more detrimental impact than other types of linguistic errors.

6.4 Native language identification: misspellings as an ally

A different, but closely related task is Native Language Identification (NLI), the task of determining a writer’s first language (L1) based on their writing in a second language (typically English) Goswami et al. (Reference Goswami, Thilagan, North, Malmasi and Zampieri2024).

In the context of NLI, misspellings are not merely noise but can serve as informative features reflecting transfer effects from the writer’s native language. These orthographic errors often carry systematic patterns that NLP models can exploit to improve identification accuracy, making misspellings a valuable signal rather than a problem to solve in this specific task.

To the best of our knowledge, the first work to explicitly exploit the presence of misspellings in NLI was conducted by Koppel et al. (Reference Koppel, Schler and Zigdon2005), who used a multiclass SVM model that incorporated spelling errors alongside other stylistic and structural features. Later on, Brooke and Hirst (Reference Brooke and Hirst2012) extended the approach using a larger dataset and cross-corpus evaluation, focusing on lexical features and domain adaptation techniques, though not directly modelling misspellings.

A renewed emphasis on spelling-based features arose with Chen et al. (Reference Chen, Strapparava and Nastase2017a), who showed, using the TOEFL11 dataset, that misspellings alone could be very informative when codified as features. Markov et al. (Reference Markov, Chen, Strapparava and Sidorov2017) integrated such features in the CIC-FBK system for the NLI Shared Task, combining them with other features to further improve performance. Building on this idea, Markov et al. (Reference Markov, Nastase and Strapparava2019) introduced orthographic features like misspelt cognates and L2-ed words-terms from the native language adapted into English orthography, while Markov et al. (Reference Markov, Nastase and Strapparava2022) further confirmed the utility of these cues across multilingual learner corpora.

6.5 Spelling reforms

Over the years, official spelling reforms have been implemented in many languages. A spelling reform is a change in normative orthography, typically introduced by state language authorities through top-down political or institutional action. For example, the French Conseil supérieur de la langue française proposed a spelling reform in 1990 that affected around 2,000 words and sparked considerable public debate (Humphries Reference Humphries2019).Footnote ⁹ A similar case is the Dutch spelling reform of 2005, which also provoked widespread discussion and resistance (Nunn and Neijt Reference Nunn and Neijt2007).

Other notable examples include the German orthographic reform of 1996, which introduced systematic changes (e.g., replacing “ß” with “ss” in certain cases), and various proposals for simplified spelling in English, such as those of the so-called Simplified Spelling Board in the early 20th century (e.g., “nite” for “night”, “tho” for “though”), which, although not officially adopted, have influenced informal usage. In Spanish, the Royal Spanish Academy (Real Academia Española—RAE) has periodically introduced adjustments to spelling conventions, such as eliminating diacritical marks (e.g., in “solo”) and modifying treatment of foreign words (e.g., the English term “whisky” is replaced by “güisqui”).

Although not usually categorised as misspellings, these reforms can introduce variation and ambiguity in corpora, especially during transitional periods. NLP models trained on pre- or post-reform data may misclassify reformed spellings as errors or unknown words. In this context, spelling reforms represent a socio-linguistic phenomenon that can significantly affect downstream NLP tasks and serve as an additional example of the challenges systems resilient to misspellings have to cope with.

7.1 Main tasks

The main tasks in which the phenomenon of misspellings has been more thoroughly investigated are listed below:

• • Machine translation (MT) is a supervised learning task that involves producing a text in a target language that is a translation equivalent of a text written in a different source language. With most modern MT systems relying on neural networks, the field is nowadays broadly referred to as Neural Machine Translation (NMT). Undoubtedly, NMT is the field in which more methods for misspellings have been applied. One possible reason why this area has attracted a lot of attention may have to do with the appearance of two influential papers by Belinkov and Bisk (Reference Belinkov and Bisk2018) and Heigold et al. (Reference Heigold, Varanasi, Neumann and van Genabith2018) that brought the importance of misspellings in MT to the fore. Most methods to counter misspellings in NMT rely on data augmentation (Section 5.1); see, for example, (Alam and Anastasopoulos 2020; Belinkov and Bisk Reference Belinkov and Bisk2018; Cheng et al. Reference Cheng, Jiang and Macherey2019; Heigold et al. Reference Heigold, Varanasi, Neumann and van Genabith2018; Karpukhin et al. Reference Karpukhin, Levy, Eisenstein and Ghazvininejad2019; Li et al. Reference Li, Rei and Specia2021; Park et al. Reference Park, Sung, Lee and Kang2020; Passban et al. Reference Passban, Saladi and Liu2021; Salesky et al. Reference Salesky, Etter and Post2021; Wang et al. Reference Wang, Zhang and Xing2020; Vaibhav et al. Reference Vaibhav Singh and Neubig2019; Zheng et al. Reference Zheng, Liu, Ma, Zheng and Huang2019).

• • Text classification (TC) is the supervised learning task of assigning class labels to unseen documents (Sebastiani Reference Sebastiani2002). While the class labels may virtually represent anything (e.g., from types of news to opinion stance to characteristics of an author), the most important applications of TC for the concerns of this survey include spam filtering and content moderation (Kurita et al. Reference Kurita, Belova and Anastasopoulos2019). The use of misspellings in such contexts might be deliberate and malicious, targeting specific relevant words so that they become unrecognisable for an automatic detector (thus eluding any ban), but easily recognisable for the final recipient of the message. Some relevant approaches focusing in TC include (Chen et al. Reference Chen, Varoquaux and Suchanek2022; Kurita et al. Reference Kurita, Belova and Anastasopoulos2019; Li et al. Reference Li, Cohn and Baldwin2016; Li et al. Reference Li, Ji, Du, Li and Wang2019a; Sankar et al. Reference Sankar, Ravi and Kozareva2021).

• • Named entity recognition (NER) (Sang and De Meulder Reference Sang and De Meulder2003) is the task of identifying and categorising (potentially multi-word) expressions referring to entities such as names, locations, and organisations, in text (e.g., White House). Namysl et al. (Reference Namysl, Behnke and Köhler2020) observed that humans can resolve NER even in the presence of corrupted inputs and studied automatic ways for developing NER systems resilient to misspellings. Some techniques explored for NER include data augmentation (Namysl et al. Reference Namysl, Behnke and Köhler2020; Reference Zhou, Zhang, Jin, Peng, Xiao and CaoZhou et al. 2020) and contrastive learning (Chen et al. Reference Chen, Varoquaux and Suchanek2022).

• • Native language identification: Given a speaker of two languages, the goal of Native Language Identification is to discover the L1 language of the author of a given text written in an L2 language Goswami et al. (Reference Goswami, Thilagan, North, Malmasi and Zampieri2024). In this case, misspellings exhibit patterns that are often strongly correlated with the mother tongue, thus becoming allies to discover the L1 language (see also Section 6.4).

• • Other downstream tasks tested in the field of misspellings include:

  1. – Part-Of-Speech tagging (PoST) is the task of labelling words with their correct morphosyntactic part-of-speech (e.g., verb, noun, preposition, etc.).

  2. – Paraphrase detection is the task of determining whether two given sentences have the same meaning or not.

  3. – Identification of textual entailment is the task of determining whether there is entailment, contradiction, or no relation between two given sentences.

  4. – Question answering: is the task of providing natural language answers to natural language questions.

  Influential papers that tested some of these tasks are: Doval et al. (Reference Doval, Vilares and Gómez-Rodríguez2020); Jones et al. (Reference Jones, Jia, Raghunathan and Liang2020); Malykh et al. (Reference Malykh, Logacheva and Khakhulin2018); Sidiropoulos and Kanoulas (Reference Sidiropoulos and Kanoulas2022).

• • Intrinsic tasks: misspellings have also been tested by means of problems carefully designed to probe the capability of the system to cope with one specific language-related ability. Some of these include:

  1. – Semantic word similarity is the task of identifying semantic relationships between words like cat and dog, or tall and short (Lenci et al. Reference Lenci, Sahlgren, Jeuniaux, Gyllensten and Miliani2021).

  2. – Analogy completion, the task of inferring which, among a closed set of options, is the more likely missing word from an incomplete analogy like Colosseo stands to Rome as the Buckingham Palace stands to $\ldots ?$ (Lenci et al. Reference Lenci, Sahlgren, Jeuniaux, Gyllensten and Miliani2021).

  3. – Outlier detection, the task of identifying the word from a set of candidates that bears less semantic similarity to the rest of the terms in that set (Camacho-Collados and Navigli Reference Camacho-Collados and Navigli2016).

  Methods that have been assessed in intrinsic tasks include those by Chen et al. (Reference Chen, Varoquaux and Suchanek2022); Doval et al. (Reference Doval, Vilares and Gómez-Rodríguez2020); Edizel et al. (Reference Edizel, Piktus, Bojanowski, Ferreira, Grave and Silvestri2019); Pruthi et al. (Reference Pruthi, Dhingra and Lipton2019); Sperduti et al. Reference Sperduti, Moreo and Sebastiani(2021).

7.2 Evaluation metrics

The most straightforward way to measure the robustness of a system to the presence of misspellings (and the way most papers have indeed adhered to) comes down to simply confronting the performance a model scores with and without noisy inputs, given a standard evaluation measure for a specific task.

More formally, let $m\in \mathscr{M}$ be a generic inference model $m\,:\, \mathcal{X} \rightarrow \mathcal{Y}$ issuing predictions $\hat {y}\in \mathcal{Y}$ on textual inputs $x\in \mathcal{X}$ , that has been trained to perform any given downstream task (classification, translation, etc.), and let $e\,:\, \mathscr{M} \times (\mathcal{X} \times \mathcal{Y})^n \rightarrow \mathbb{R}$ be our evaluation measure ( $F_1$ score, BLEU score, etc.) of choice, that is any scoring function that takes as input a model and a (labelled) test set, and computes a value reflecting the empirical goodness of $m$ . The degradation in performance due to the presence of misspellings can generally be estimated as the difference in performance:

\begin{equation*} e\left(m,\{(x_i,y_i)\}_{i=1}^n\right) - e\left(m,\{\left(\tilde {x}_i,y_i\right)\}_{i=1}^n\right)\end{equation*}

where $\tilde {x}_i$ is a misspelt variant of the (clean) input $x_i$ .

Such an evaluation strategy is generic enough to apply to virtually any supervised task, and therefore has nothing specific to do with any particular evaluation metric. Typical evaluation measures used in the tasks discussed in Section 7.1 include, among others, BLEU (Papineni et al. Reference Papineni, Roukos, Ward and Zhu2002) and METEOR (Banerjee and Lavie Reference Banerjee and Lavie2005) for machine translation; precision, recall, and $F_\beta$ score for text classification and named entity recognition (van Rijsbergen Reference van Rijsbergen1979); Pearson correlation and Spearman correlation for intrinsic tasks (Doval et al. Reference Doval, Vilares and Gómez-Rodríguez2020; Lenci et al. Reference Lenci, Sahlgren, Jeuniaux, Gyllensten and Miliani2021). An in-depth survey of these and other specific evaluation metrics is out of the scope of this article.

A few metrics have been proposed by Anastasopoulos (Reference Anastasopoulos2019), which are particularly suitable for measuring the robustness of misspellings in the context of machine translation. These are based on the observation that any perfectly robust-to-noise MT system would produce the exact same output for the clean and erroneous versions of the same input sentence. The intuition is sketched as follows: let $m^*$ be a perfect MT system; then $m^*(x)$ should produce the same (correct) prediction $y^*$ as $m^*(\tilde {x})$ , with $\tilde {x}$ a noisy variant of $x$ . Such a perfect model is typically unavailable, but we might have a reasonably good model $m$ instead. System robustness is therefore estimated by computing the extent to which $m(x)$ produces outputs similar to $m(\tilde {x})$ , even though such predictions might not be perfect. In light of this, Anastasopoulos (Reference Anastasopoulos2019) proposes the Robustness Percentage (RB) as:

(1) \begin{equation} \mathrm{RB}=100 \times \frac {|\{(x,\tilde {x})\,:\, m(x)=m(\tilde {x})\}_{(x,\tilde {x})\in D}|}{|D|} \end{equation}

where $D$ is a dataset of pairs of correct and noisy inputs.

In the same paper, Anastasopoulos (Reference Anastasopoulos2019) propose the Target-Source Noise Ratio (NR), a more fine-grained evaluation measure that also accounts for the distance between $x$ and $\tilde {x}$ , given that small differences would count just as much as large differences for RB. Instead, NR tries to factor out this distance $d$ , which is computed using a surrogate evaluation metric like BLEU or METEOR, for example. NR is defined as:

(2) \begin{equation} \mathrm{NR}(m,x,\tilde {x})=\frac {d(m(x),m(\tilde {x}))}{d(x,\tilde {x})} \end{equation}

Anastasopoulos (Reference Anastasopoulos2019) suggest reporting the mean NR across all pairs $x$ and $\tilde {x}$ contained in a dataset.

Michel et al. (Reference Michel, Li, Neubig and Pino2019) propose a metric for evaluating adversarial attacks that requires access to the correct translation $y^*$ . The metric requires a similarity function $s$ and is computed for each pair of inputs $x$ and $\tilde {x}$ as follows:

(3) \begin{equation} A(m,x,\tilde {x},y^*) = s(x,\tilde {x})+\frac {s(m(x), y^*)-s(m(\tilde {x}), y^*)}{s(m(x), y^*)} \end{equation}

the adversarial attack is considered to be successful whenever $A(m,x,\tilde {x},y^*)\gt 1$ ; the metric therefore computes the fraction of successful cases against all cases in a dataset.

7.3 Conferences and workshops

The most important venues, including workshops, conferences, and shared tasks, that have been devoted to discussing the problem of misspellings in NLP include:

• • Workshop on Analytics for Noisy Unstructured Text Data (AND): This workshop had five editions run from 2007 to 2011. The main objective of AND was to gather papers discussing techniques for dealing with noisy inputs. The notion of noisy inputs encompasses misspellings, but also grammatical error correction, text normalisation, spelling correction, and any other form of noise affecting textual data as those generated through speech recognition systems or OCR. The first workshop was co-located in the 2007 edition of the Joint Conference of Artificial Intelligence (IJCAI), although no proceedings seem to be available online. The second edition was co-located at the SIGIR conference in the next year (Lopresti et al. Reference Lopresti, Roy, Schulz and Subramaniam2008), followed by a third edition co-located in the International Conference On Document Analysis and Recognition (ICDAR) (Lopresti et al. Reference Lopresti, Roy, Schulz and Subramaniam2009), and a fourth edition co-located in the International Conference on Information and Knowledge Management (CIKM) (Basili et al. Reference Basili, Lopresti, Ringlstetter, Roy, Schulz and Subramaniam2010). The workshop then evolved as a Joint Workshop on Multilingual OCR and Analytics for Noisy Unstructured Text Data (Dey et al. Reference Dey, Govindaraju, Lopresti, Natarajan, Ringlstetter and Roy2011) and was, to the best of our knowledge, discontinued after that.

• • Robustness task at the World Machine Translation (WMT) Conference: This task was first proposed in 2019 (Li et al. Reference Li, Michel, Anastasopoulos, Belinkov, Durrani, Firat, Koehn, Neubig, Pino and Sajjad2019b) and was later followed by a new edition in 2020 (Specia et al. Reference Specia, Li, Pino, Chaudhary, Guzmán, Neubig, Durrani, Belinkov, Koehn, Sajjad, Michel and Li2020). To the best of our knowledge, this is the only shared task specifically devoted to testing the MT systems’ resiliency to misspellings. In both editions, the shared tasks focused on the same language pairs: English-French and English-Japanese. In the first edition, the test set was constructed by applying the MTNT protocol (Michel and Neubig Reference Michel and Neubig2018) to data gathered from Reddit, while the previously existing datasets (WMT15 for the English-French pair, and KFTT (Neubig Reference Neubig2011), JESC (Pryzant et al. Reference Pryzant, Chung, Jurafsky and Britz2018), and TED Talks (Cettolo et al. Reference Cettolo, Federico, Specia and Way2012) for the English-Japanese pair) were employed as the training set. The systems were evaluated by professional translators as well as in terms of the BLEU score. In the second edition, the news dataset WMT20 was employed as the training set, while the test sets consisted of multiple sources, including Wikipedia and Reddit comments, among others.

• • Workshop on Noisy and User-Generated Text (W-NUT): Footnote ¹⁰ This is an ongoing workshop series that started in 2015, has been held every year (except 2023), with the last edition co-located at the NAACL 2025; the proceedings of all editions are published in the ACL Anthology. The workshop focuses on noise-generated content in social networks and is not exclusively devoted to misspellings. The workshop gathers papers dealing with tasks as disparate as geolocalisation prediction, global and regional trend detection and event extraction, fairness and biases in NLP models, etc.

7.4 Datasets

In this section, we turn to describe the most important types of datasets that have been used for training and evaluation of systems dealing with misspellings in literature, with particular emphasis on the techniques that have been employed for generating them. Since misspellings are relatively infrequent in real texts (with varying levels of prevalence that depend on the medium), the aim of these techniques is to guarantee a relatively high number of misspellings in the corpus, somewhat akin to oversampling strategies often used in extremely imbalanced supervised scenarios. However, as recalled from Section 3.2, the distinction between natural and synthetic misspellings is extrinsic to their surface form and lies in their mode of generation. Since the latter are created procedurally, they are easier to produce and therefore dominate the landscape of currently available datasets.

Broadly speaking, these techniques can be grouped as belonging to the following categories:

• • Natural misspellings (Section 7.4.1): techniques that collect real misspellings from textual data, that is errors that occur spontaneously in user-generated (e.g., in social media, emails) or technologically-generated contexts (e.g., optical character recognition, automatic transcription).

• • Artificial Misspellings (Section 7.4.2): techniques for generating synthetic misspellings out of the original (clean) words from the texts in a dataset.

• • Hybrid approach (Section 7.4.3): consists of using error correction databases (i.e.,, databases in which real misspellings have been labelled with the correct word) to inject misspellings in clean texts. The approach is called hybrid since the misspellings being injected are real, but the injection itself is artificial.

7.4.2 Datasets of artificial misspellings

Since misspellings affect written natural language in general, they potentially harm any textual application one could think of. For this reason, when it comes to measuring the impact that misspellings cause in any downstream task, or when training models that ought to be robust to them, it is customary to simply take standard datasets routinely used for these downstream tasks and produce variants of them that contain misspelt entries; this is the approach followed by, for example Belinkov and Bisk (Reference Belinkov and Bisk2018); Heigold et al. (Reference Heigold, Varanasi, Neumann and van Genabith2018); Passban et al. (Reference Passban, Saladi and Liu2021); Sperduti et al. (Reference Sperduti, Moreo and Sebastiani2021). Given that the phenomenon of misspellings is orthogonal to the downstream tasks in which they are studied, we refrain from listing typical datasets customarily used across different disciplines (a glance at Table 1 reveals many of them).

The most common strategy comes down to generating synthetic misspelt variants of the original words in a text. Some techniques that have been proposed for this purpose (see, e.g., Belinkov and Bisk Reference Belinkov and Bisk2018; Kumar et al. Reference Kumar, Makhija and Gupta2020; Moradi and Samwald Reference Moradi and Samwald2021) and that have been reproduced in other papers are listed below. The terminology we use for naming these types of misspellings is not standard in the literature, but is, we believe, appropriate for describing them. A list of methods with the types of misspellings they used is summarised succinctly in Table 2.

Table 2.

Types of misspellings applied in each paper. We included in this table only works that used synthetic misspellings and that explicitly described the kind of misspellings used

• • Full Permutation: involves generating a new token by completely permuting the characters of a term, for example, misspell $\rightarrow$ pseilmls .

• • Middle Permutation: generates a new token by permuting all characters of a term except the first and last, which remain in place, for example, misspell $\rightarrow$ mpseisll. This is also known as garbling (Sperduti et al. Reference Sperduti, Moreo and Sebastiani2021) or scrambling (Heigold et al. Reference Heigold, Varanasi, Neumann and van Genabith2018).

• • Swap: consists of choosing two adjacent characters at random from a word and interchanging their positions, for example, miss pell $\rightarrow$ misp sell.

• • Qwerty: consists of emulating typographical errors that are likely to arise when employing a QWERTY layout, for example, misspell $\rightarrow$ mi9sspell (note the key “9” is placed nearby the key ‘i’ in this layout). This kind of error is a special subtype of Addition (see below).

• • Addition: comes down to adding one or more characters to the target word, for example, misspell $\rightarrow$ missprell.

• • Deletion: amounts to removing one or more characters from a word, for example, misspell $\rightarrow$ mispell.

• • Substitution: consists of choosing one character at random from a word and replacing it with another character, for example, misspell $\rightarrow$ mrsspell.

8.1 Instance-based tests

Instance-based tests come down to inserting misspellings into the test instances themselves. The primary approach to testing LLMs against misspellings has been through dedicated benchmarks, as those described in Section 7.4.4.

Wang et al. (Reference Wang, Chen, Pei, Xie, Kang, Zhang, Xu, Xiong, Dutta, Schaeffer, Truong, Arora, Mazeika, Hendrycks, Lin, Cheng, Koyejo, Song and Li2023, Reference Wang, Hu, Hou, Chen, Zheng, Wang, Yang, Ye, Huang, Geng, Jiao, Zhang and Xie2024) employ the AdvGLUE benchmark to evaluate the robustness of models like GPT-3.5 and GPT-4. This benchmark includes misspellings in the test instances, as outlined in Section 7.4.4.

The benchmark AdvGLUE++ (Wang et al. Reference Wang, Chen, Pei, Xie, Kang, Zhang, Xu, Xiong, Dutta, Schaeffer, Truong, Arora, Mazeika, Hendrycks, Lin, Cheng, Koyejo, Song and Li2023) has served to show that both GPT-3.5 and GPT-4 experience a notable drop in performance when exposed to misspellings. However, Wang et al. (Reference Wang, Hu, Hou, Chen, Zheng, Wang, Yang, Ye, Huang, Geng, Jiao, Zhang and Xie2024) found that these models are still more robust when compared to smaller models like BART-L, DeBERTa-L, and even bigger models such as text-davinci-002. Despite these insights, neither AdvGLUE nor AdvGLUE++ allow for a detailed ablation study, since the exact quantity and typology of misspellings in the dataset are not specified.

Pan et al. (Reference Pan, Leng and Xiong2024) evaluated the robustness of large language models (LLMs) to noisy input (including misspellings) in machine translation. Specifically, the authors tested Baichuan2-7B-Chat and Baichuan2-13B-Chat for Chinese–English translation, and Qwen-7B-Chat and Qwen-14B-Chat for Indonesian–Chinese translation. The models are subjected to various types of misspellings, including both synthetic and naturally occurring misspellings. The authors found that incorporating misspellings into the prompt, as demonstrated examples, can improve model robustness. The impact of misspellings varies depending on the method used to generate them and the prompting strategy applied.

In the next section, we turn to methods that incorporate misspellings in the prompt not as demonstrations, but as genuine errors, in order to test model resiliency.

8.2 Prompt-based tests

Prompt-based tests introduce misspellings into the prompts provided to the model. Notable examples include the PromptRobust (Zhu et al. Reference Zhu, Wang, Zhou, Wang, Chen, Wang, Yang, Ye, Zhang and Gong2023) and eBench (Zhang et al. Reference Zhang, Hao, Li, Zhang and Zhao2024) benchmarks.

In contrast to instance-based tests, these benchmarks include an ablation study, which enables further disentangling of how misspellings impact model performance. Zhu et al. (Reference Zhu, Wang, Zhou, Wang, Chen, Wang, Yang, Ye, Zhang and Gong2023) experimented with T5-large, Vicuna, LLaMA2, UL2, ChatGPT and GPT-4 on PromptRobust, while Zhang et al. (Reference Zhang, Hao, Li, Zhang and Zhao2024) used LLaMA, Vicuna, GPT-3.5 and GPT-4 in eBench. Both studies concluded that LLMs experience significant performance drops when faced with misspelt prompts, though GPT-4 appears to be much more resilient than other models.

A similar conclusion was reached by Cao et al. (Reference Cao, Kojima, Matsuo and Iwasawa2023), who introduced ScrambledBench, a benchmark designed to test LLMs on text with internally scrambled characters—a challenge closely related to the problems addressed by the models discussed in Section 5.2. Once again, GPT-4 demonstrated notable robustness to this type of misspelling.

Building on this line of research, Wang et al. (Reference Wang, Gu, Wei, Gao, Song and Chen2025) recently investigated the same phenomenon with the aim of identifying the factors that most influence this resilience. Specifically, they sought to disentangle the relative contributions of context and word form to LLMs’ robustness against misspellings, finding that word form plays a more important role than context in reconstructing scrambled text.

Among prompt-based evaluations under misspellings, Moffett and Dhingra (Reference Moffett and Dhingra2025) introduce a novel task called the recovery task, which assesses a model’s ability to reconstruct the correct surface form given a misspelt variant of a word. To this end, they present the Ad-Word dataset, built from the 10,000 most frequent words in the Trillion Word Corpus. Each word is perturbed using approximately nine cognitively motivated misspelling strategies (e.g., typo-based, phoneme-based, visual-based). Three experimental settings are proposed:

• • Prediction without context: Several language models, including open-source and commercial versions, were evaluated on isolated word recovery. Surprisingly enough, GPT-4 surpassed the accuracy of the human expert baseline, consisting of five annotators.

• • Prediction with context: LLaMA2 and Mistral were tested on the same task with the aid of sentence-level context. Contextual information improved recovery in some cases but degrades performance in others, depending on the nature of the misspelling (e.g., in LLaMA2-7B, several additional visual and typographic misspellings were recovered with the aid of context, but accuracy also dropped by about 15 per cent in cases where the model had performed well without it).

• • Hate speech detection: Words in the HOT (Hate, Offensive, Toxic) dataset were misspelt at varying levels. Both LLaMA2 and Mistral exhibit degraded classification performance, with the latter being more adversely affected.

The results suggest that open-source models are generally more vulnerable to misspellings than commercial counterparts.