2.1 Analytical work

Vidgen et al. (Reference Vidgen, Harris, Nguyen, Tromble, Hale and Margetts2019) describe significant challenges and unaddressed limitations, and how these issues constrain abusive content detection research and limit its impact on the development of real-world detection systems. The authors identify two major challenges. First, there is the research challenge which consists of three parts. Part one concerns the categorization of abusive content. For one, there should be more clarity in subtasks. Research that purports to address the generic task of abuse detection is actually addressing something much more specific. This can often be seen in the datasets, which may contain systematic biases toward certain types and targets of abuse. Hence, the authors propose that subtasks are further disambiguated by the direction of abuse. Another issue is clarity of terminology. Clarifying terminology will help delineate the scope and goals of research and enable better communication and collaboration. Some of the main problems are (i) researchers use terms that are not well-defined, (ii) different concepts and terms are used across the field for similar work, and (iii) the terms used are theoretically problematic. In particular, three aspects of the existing terminology have significant social scientific limitations: speaker’s intent, effect of abuse, and audience sensitivity. The authors advocate ignoring the speaker’s intent, not conflating two distinct types of abuse in terms of effect, and not making strong assumptions about the audience. The second part deals with the recognition of abusive content. The authors identify five linguistic difficulties that complicate the detection of abusive content: humor/irony/sarcasm, spelling variants, polysemy, extensive dependencies, and language changes. The last part concerns the context, which must be taken into account. Meaning is inherently ambiguous and depends on the subjective perspective of both the speaker and the audience, as well as the specific situation and power dynamics. The second challenge concerns the community. Creating appropriate datasets for training abusive language detection systems is an important but time-consuming task. Currently available datasets have several limitations. For many datasets, including those from Twitter, content cannot be shared directly, but IDs are shared and the dataset is recreated each time. This can lead to significant degradation in the quality of datasets over time. To address this issue, the authors suggest working more with online platforms to make datasets available. Furthermore, annotation is a notoriously difficult task, which is reflected in the low level of inter-annotator agreement reported in most publications, especially for more complex multi-class tasks. Few publications provide details on their annotation process or annotation guidelines. Providing such information is the norm in social scientific research and is considered integral to verifying others’ findings and robustness. Another issue is the quality, size, and class balance of datasets, which varies considerably. Understanding the decisions underlying the creation of datasets is crucial for identifying the biases and limitations of systems trained on them. The authors propose that datasets could be curated to include linguistically difficult instances, as well as “edge cases”: content that is non-abusive but very similar to abuse. The authors note that most research on abusive content detection focuses on text, but not on multimedia content. Additionally, the implementation of a system should be fair, explainable, and efficient. Finally, researchers should ensure that systems can be used in a variety of settings.

In a comprehensive study, Vidgen and Derczynski (Reference Vidgen and Derczynski2020) analyze 63 datasets designed for the development of abusive language classifiers. The subjects of the analysis are the classification tasks and corresponding target taxonomies of the datasets, the contents and annotations schemes of the datasets, as well as the source of the data and the motivations for creating datasets. Furthermore, the authors present papers that address the broad spectrum of online abusive language detection and its associated challenges, ranging from data collection, bias in the training data, varying definitions of abusive language, different target labels, the role of online platforms, the need for machine learning support, and annotation methods. In addition to the contributions of their paper, the authors are creating and maintaining the website https://hatespeechdata.com, which provides basic information about the datasets, such as a link to the dataset and the corresponding publication, size and language of the data, and the source of the content. The website is intended as a starting point for a data-sharing platform in this domain. The analysis shows, among other things, that datasets of abusive language come from heterogeneous sources (YouTube, Twitter, Wikipedia, Facebook), that they aim at different target classifications, that the level of expertise of the annotators varies, and that annotation schemes are often not transparent or loose. In addition to a comprehensive analysis of datasets for abusive language detection and a website to promote data sharing, the authors derive guidelines for creating new datasets for this purpose. The guidelines include instructions for the task, the actual dataset creation, the annotation process, and the documentation of best practices.

In a systematic review, Poletto et al. (Reference Poletto, Basile, Sanguinetti, Bosco and Patti2021) analyze 64 datasets including eight dictionaries for hate speech detection and point out their heterogeneous character. In particular, they try to illustrate the different concepts related to hate speech in the literature and datasets. Figure 1 (Poletto et al. Reference Poletto, Basile, Sanguinetti, Bosco and Patti2021) shows both the boundaries and the relationship of inclusion and exclusion between hate speech and the various related terms. One of the main goals of the study is to create a method for developing robust hate speech detection resources. Therefore, they investigate the resources found in the research papers according to five different dimensions: type, topical focus, data source, annotation, and language. Among these, topical focus and annotation make hate speech recognition confusing and difficult. As noted in Figure 1, they point out that the boundaries between hate speech and other concepts overlap and each concept can be interpreted differently depending on cultural background and individual perception. Moreover, they note that some papers do not even provide a clear definition of the investigated concepts. Besides the topical focus, the annotation scheme and the skills of the annotators are essential aspects for the creation of suitable datasets. But annotation schemes are often not clearly described by the dataset creators. Annotations can be done either by experts, non-experts like crowdsourcing, or automated classifiers. Organizers often use a measure of inter-annotator agreement, with values ranging from very high (Cohen’s $\kappa$ = 0.982) to extremely low (Krippendorff’s $\alpha$ = 0.18) implying the difficulty of the classification task. They also argue that the collected data are not representative due to the collection methodology. Most data collections are made up by keyword searches on social media, especially Twitter. In their experiment with the English HatEval dataset, the evaluation showed that this leads to biased data collection. On the one hand, there are unclear boundaries regarding hate speech and its related concepts. On the other hand, the datasets are biased and sometimes too specific, which can lead to overfitting. These two issues are the main problems why an optimal result cannot be achieved in automatic abusive language detection. To overcome the challenges related to biased datasets, they suggest an in-depth error analysis that investigates the results of systems trained on a given dataset. For shared tasks in particular, they emphasize that thorough error analysis by the organizers helps to identify critical problems in the dataset scheme and annotation.

Fig. 1: Relations and boundaries between hate speech and related concepts according to Poletto et al. (Reference Poletto, Basile, Sanguinetti, Bosco and Patti2021).

Although the main contribution of Risch et al. (Reference Risch, Schmidt and Krestel2021) is a software tool that provides easy access to many individual toxic comment datasets, the authors also analyze the datasets. For the creation of the collection, the authors consider all publicly accessible comment datasets that contain labels that are subclasses of toxicity (e.g., offensive language, abusive language, and aggression). The broad definition of toxicity as a higher-level concept builds a bridge between the different lower-level concepts (e.g., sexism, aggression, and hate). During collection, the different dataset formats are converted into the same standardized csv format. Overall, the collection contains comments in thirteen different languages—obtained from twelve platforms—with 162 distinct class labels. The total number of samples is currently 812,993, but according to the authors, they are constantly adding more datasets. It is noteworthy that the collection does not contain the datasets themselves, but that the tool retrieves these datasets from their source and converts them into the unified format. According to the authors, this gives easy access to a large number of datasets and the ability to filter by language, platform, and class label. The authors also address the problem of class imbalance in many datasets. While most datasets show a bias toward “normal” comments, the dataset by Davidson et al. (Reference Davidson, Warmsley, Macy and Weber2017) shows a bias toward “offensive” comments. However, the latter class distribution is not representative of the underlying data in general. Additionally, some datasets are collected by filtering comments by a list of keywords, for example, “muslim,” “refugee,” or hashtags (#banislam, #refugeesnotwelcome). But such filtering introduces a strong bias because all hateful tweets in the created dataset contain at least one of the keywords or hashtags. Hence, the data are not a representative sample of all hateful comments, and models trained on this data might overfit the list of keywords and hashtags.

In contrast to the presented works, our contribution is based on participation in shared tasks and the practical challenges encountered. While we reach the same conclusions in many aspects (inadequate annotation process, heterogeneous motivations, data sources, and content), we provide a deeper analysis on a smaller selection of datasets. This includes the quantitative analysis of nine datasets where we present key data including data size, data collection, annotation guidelines, and best system performance. For the qualitative analysis, we manually inspect a sample of each dataset and present common categories of controversial annotations.

2.2 Experimental work

Arango et al. (Reference Arango, Pérez and Poblete2019) closely examine the experimental methodology used in previous work and find evidence for methodological problems by analyzing two models that performed best on the dataset of Davidson et al. (Reference Davidson, Warmsley, Macy and Weber2017). One of the experiments conducted by Arango et al. (Reference Arango, Pérez and Poblete2019) tests the generalization to other datasets. For this purpose, they train both models of Badjatiya et al. (Reference Badjatiya, Gupta, Gupta and Varma2017) and Agrawal and Awekar (Reference Agrawal and Awekar2018) on the complete dataset of Waseem and Hovy (Reference Waseem and Hovy2016) and use the HatEval dataset (Reference Basile, Bosco, Fersini, Nozza, Patti, Rangel Pardo, Rosso and SanguinettiBasile et al. 2019) as test set. Both models achieve a macro-averaged F1 score below 50%, indicating that generalization to other datasets is not given. Another issue is the impact of user distribution on model generalization. For example, in the dataset by Waseem and Hovy (Reference Waseem and Hovy2016), one user is responsible for 44% of all “sexist” comments, while another user is responsible for 96% of “racist” comments. The authors hypothesize that user distribution is another source of overestimation of the performance of state-of-the-art classifiers. To prove this, they conduct another set of experiments in which the authors ensure that posts from the same user are not included in both training and test sets. This leads to a further drop in F1 scores for both models, and Arango et al. (Reference Arango, Pérez and Poblete2019) conclude that the models are susceptible to what they call “user-overfitting,” that is, the models learn to discriminate between users rather than comment content. Indeed, Arango et al. (Reference Arango, Pérez and Poblete2019)’s results help to understand why state-of-the-art models that show impressive performance in their original test sets do not generalize well to other datasets, even when new data come from the same domain. They also conduct experiments aimed at alleviating state-of-the-art problems and improving model generalization. Their proposed solutions show improved performance across datasets and better estimation of model performance, leading to a more accurate state-of-the-art. In addition, they conclude that it is important to pay more attention to experimental evaluation and generalization of existing methods. In particular, the authors believe that there is an urgent need to obtain more information about the distribution of users in existing datasets or, even better, to create datasets that do not contain significant user biases.

Moreover, MacAvaney et al. (Reference MacAvaney, Yao, Yang, Russell, Goharian and Frieder2019) identify and explore the challenges faced by automated online approaches to hate speech detection in texts. The authors identify the varying definitions of hate speech as one problem. Another problem is the limitation of publicly available data for training and testing. Many of the available datasets come from Twitter, but because of the character limit, these texts are short and concise. Therefore, models trained on such data cannot be generalized to texts from other sources. In addition, there are not many datasets that more accurately distinguish between hateful, aggressive, sexist, and similar content. Furthermore, datasets vary significantly in terms of their size and the characteristics of the hate speech considered. The authors use a multi-view SVM model and evaluate its performance on four datasets. Their experiments show that this approach can outperform existing systems but the authors also note there is need for more research on both technical and practical aspects given the remaining challenges.

Kovács et al. (Reference Kovács, Alonso and Saini2021) find the automatic detection on social media posts difficult because they contain paralinguistic signals (e.g., emoticons and hashtags), and their linguistic content contains a lot of poorly written text. The authors encounter several challenges. First, there is word obfuscation, which is the intentional or unintentional misspelling of words. In addition, there are expressions that are considered offensive only in context. Another challenge is the use of ethnic slurs by in-group and out-of-group speakers, as the former are not considered offensive. Furthermore, the authors state that there is no universally accepted definition of hate speech and that even people have difficulty annotating hate speech using annotation guidelines. In their experiments, the authors study the effect of leveraging additional data on the HASOC dataset. They use several classical machine learning methods as well as different deep learning models. Their results show that additional data is useful in most cases, even if the data comes from another collection.

3.1 OLID

Zampieri et al. (Reference Zampieri, Malmasi, Nakov, Rosenthal, Farra and Kumar2019b) present the Offensive Language Identification Dataset (OLID) as part of the SemEval 2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval).

3.2 HatEval

The shared task HatEval describes the detection of hate speech against immigrants and women in Spanish and English tweets extracted from Twitter at SemEval 2019 (Basile et al., Reference Basile, Bosco, Fersini, Nozza, Patti, Rangel Pardo, Rosso and Sanguinetti2019).

3.3 TRAC-2020

The shared task on Aggression and Gendered Aggression Identification was organized as part of the Second Workshop on Trolling, Aggression, and Cyberbullying (TRAC-2) at the 12th Language Resources and Evaluation Conference (LREC 2020; Kumar et al. (Reference Kumar, Ojha, Malmasi and Zampieri2020)).

3.4 HateBaseTwitter

Davidson et al. (Reference Davidson, Warmsley, Macy and Weber2017) present their approach on automated hate speech detection while addressing the problem of offensive data. Unlike other work, the authors do not conflate hate speech and offensive speech, but distinguish between the two. The corpus was compiled by the authors themselves.

3.5 HASOC

Mandl et al. (Reference Mandl, Modha, Majumder, Patel, Dave, Mandlia and Patel2019) describe the identification of hate speech and offensive content in Indo-European languages, that is, English, Hindi, and German, at FIRE’19 (Majumder et al. Reference Majumder, Mitra, Gangopadhyay and Mehta2019).

3.6 GermEval 2018

Wiegand et al. (2018) present the pilot edition of the “GermEval Shared Task on the Identification of Offensive Language.” This shared task deals with the classification of German tweets from Twitter.

3.7 GermEval 2019

Following GermEval 2018, Struß et al. (Reference Struß, Siegel, Ruppenhofer, Wiegand and Klenner2019) presented the second edition of the “GermEval Shared Task on the Identification of Offensive Language.” The topics and political orientation were expanded to include left-wing extremism, right-wing extremism, and antisemitism.

3.8 GermEval 2021

The GermEval 2021 workshop on “Identification of Toxic, Engaging, and Fact-Claiming Comments” was organized as part of KONVENS 2021 (Conference on Natural Language Processing). The following descriptions are either taken from the official workshop website,Footnote ^b or from our own dataset analysis unless otherwise noted.

GermEval workshops are usually organized informally by groups of interested researchers. The GermEval 2021 workshop is endorsed by the IGGSA (Interest Group on German Sentiment Analysis), a special interest group within the GSCL (German Society for Computational Linguistics).

3.9 Summary of datasets

In Table 1, we provide an overview of the datasets by specifying the name, language, number of training and test samples, percentage of abusive samples, and type of abusive content. The datasets vary significantly in size, which risks overfitting deep learning models on the smaller datasets. The percentage of abusive samples is calculated by dividing the number of abusive samples by all samples. For the HateBaseTwitter dataset, this was done by combining “HATE” and “OFFENSIVE,” and for the TRAC-2020 dataset, this was calculated by combining “OAG” and “CAG.” The percentage of “HATE” is 6%, that of “OFFENSIVE” is 77%, that of “OAG” is 12.78%, and that of “CAG” is 12.16%. The resulting numbers show that most of the datasets are imbalanced.

Table 1. Overview of datasets used and their corresponding sizes. The percentage of abusive content is calculated by considering all types of abuse (e.g., hate, offense, aggression)

From the information in the previous sections, the values for the inter-annotator agreement are not very high in most of the datasets. The degree of agreement for English and German HASOC is minimal to weak, while it is weak to moderate on GermEval 2018 and GermEval 2019. The level of agreement for HatEval is considered high, but was calculated using only 21 tweets, so this value is not statistically significant. The value for TRAC-2020 is moderate, while the authors give a very high intercoder agreement score for HateBaseTwitter, without disclosing its calculation. These values show that people have difficulty agreeing on what constitutes abusive language.

In terms of data collection, only HateBaseTwitter and GermEval 2018 provided information on user distribution, that is, from how many different users’ tweets/posts were collected. As pointed out by Arango et al. (2019), this is crucial information to ensure that the classification is not a distinction between users, but based on content. While the user distribution of HateBaseTwitter is very good, this is not the case for GermEval 2018 (only 100 users). Future dataset creators should use a broad base of users and also provide at least an internally generated ID for each user.

In terms of annotation guidelines, there is great variation. HatEval is limited to hatred against two specific targets, women and immigrants. OLID defines offensive language as threatening, insulting, or profane. This is almost the same as GermEval 2018 and GermEval 2019, whose definition of offensive includes insult, abuse, or profanity. There seems to be a discrepancy between OLID’s threats and GermEval’s abuse. HateBaseTwitter’s definition seems to only include hate speech toward a group or member of a group. They also exclude instances of abusive language uttered by in-group speakers. A model without knowledge of a user’s ethnic background cannot correctly distinguish these cases. HASOC’s definition of hate speech also refers to negative characteristics toward a group or person of a group, while their definition of offensive language refers to individuals. Interestingly, the authors consider threats to be offensive but not hateful, which is in contrast to OLID, HatEval, and HateBaseTwitter. GermEval 2021 provides guidelines for toxic language, which seems to be a fairly broad definition covering six different topics. The most interesting case is TRAC-2020, where no strict guidelines were given. Despite the lack of clear instructions, a Krippendorff’s $\alpha = 0.75$ was reached. One factor that many of the guidelines have in common is the instruction to pay attention to context when annotating. This means that when training models on such data, context should also be considered and that context should be provided in the dataset.

In Table 2, we present the best-performing systems for each dataset evaluated using the macro-averaged F1 score, with the exception of TRAC-2020, which was evaluated using the micro-averaged F1 score. As the scores suggest, the datasets differ in difficulty. While the classification of HatEval and the German HASOC appears to be difficult in terms of reported F1 scores, this is not true for OLID, TRAC-2020, and HateBaseTwitter. The scores for the English HASOC, GermEval 2018, and GermEval 2019 are in the middle range. It is also worth noting that different models were used. While traditional machine learning models were applied on the smaller datasets, deep learning models can be used on the larger datasets.

Table 2. Overview of the systems that achieved the best results on each dataset

4.1 OLID

When we manually reviewed the data, we found a total of 68 controversial annotations, a rate of 13.6%. Many of them are insults that were marked as non-offensive. For instance, in example (1), the author calls someone stupid in a distasteful way and (2) suggests that the referred person will die because of his obesity. Although these examples are undoubtedly wrongly annotated, insults are generally very subjective. We identified a number of borderline cases that we could not label without a doubt, neither intuitively nor by following the annotation guidelines. The examples (3)–(5) illustrate this problem. Another common source of controversial annotations are tweets marked as offensive, which are not offensive in themselves, but in context. Although the OLID dataset does not contain contextual information, we identified some cases where we suspect that context “makes” the tweet offensive, for example, examples (6)–(8). In example (7) is “kids” used here as a pejorative? Or is this user more defending young people fighting for gun control?

1. (1) @USER [brain emoji] damages she is—NOT

2. (2) @USER 2020? Michael Moore is a cheeseburger away from not making it to 2019!—NOT

3. (3) @USER @USER No he is in some other world he dreamed of when he was on drugs in college—NOT

4. (4) @USER Take a stand against all violence! Talk to your ANTIFA friends.—NOT

5. (5) @USER This all I get [sad smiley] damn these haters deleting you IG for no reason $\ldots$ URL—OFF

6. (6) @USER She should ask a few native Americans what their take on this is.—OFF

7. (7) @USER @USER @USER gun control! That is all these kids are asking for!—OFF

8. (8) @USER @USER What a waste of one’s short life.[shrugging emoji][U.S. flag emoji]—OFF

4.2 HatEval

From the random samples we examined, only 21 annotations are controversial (4.2%). These include thirteen posts targeted at “women” and eight directed at “immigrants.” It is important to keep in mind that the tasks specifically target women or immigrants, and all other abusive posts targeting others are not classified as “hate speech” (HS). For example, the sentences in the examples (9) and 10 are not classified as “HS” because they are not targeted at women or immigrants. We assume that a model trained on such data does not generalize well to other target tasks related to abusive language. The examples (11) and (12) are labeled as “HS,” but the annotator needs background knowledge and/or contexts to infer this classification. The examples (13) and (14) show contradictory cases. We found many cases where facts were asserted that would need to be verified to determine the correct class. If the claimed facts in example (15) are false, then this tweet is hateful; otherwise, it is not.

1. (9) Im smack if out one of you pussy boys—NOT

2. (10) ALL YOU PUSSY BOYS FALL BACK!—NOT

3. (11) Let’s build that wall.#BuildThatWall—HS

4. (12) Thank You #ICE !! #BuildThatWall !—HS

5. (13) Dat Whore Karma ah Bitch—NOT

6. (14) Ewk man this bitch be with mad whores—HS

7. (15) It is way overdue to admit Europe cannot take all of the migrants which are trying to get there. Nor can places like Canada. @Europarl_EN @PopulationIC @PopnMatters @BBCNews @dwnews @CBC Please look at this dramatic graphic demonstration $\ldots$ https://t.co/wZ3v1J3Uqh—HS

During our analysis of this dataset, we noticed that many tweets did not contain any hateful content other than the hashtag content. For this reason, we extracted all hashtags from the whole dataset. The top five hashtags (lowercased) and their frequency per class are shown in Table 3. The values in Table 3 confirm our expectation that hashtags play an important role in classification. For example, the tweets with the hashtag #buildthatwall are all classified as HS, while the tweets with the more neutral hashtag #immigration compete for each class.

Table 3. Top five hashtags in HatEval

4.3 TRAC-2020

A total of 54 controversial annotations were found, resulting in an error rate of 10.8%. Due to the loose annotation guidelines, the difference between overtly aggressive (OAG) and covertly aggressive (CAG) is not further specified, leaving the judgment to the annotators’ intuition. From our point of view, covertly aggressive means aggression expressed only through content, while overt aggression is enhanced by tone of voice, swear words, and aggressive emojis. Considering this definition, we identified many comments which should be labeled as OAG but were labeled as CAG, such as the examples (16)–(18). We also found many comments that are labeled as CAG even though they do not contain aggression, for example, the examples (19)–(22). This indicates that the CAG label was used in cases where the annotators were unsure whether the comment should be considered aggressive, rather than a label with clear characteristics. It may be necessary to define the annotation scheme more clearly and establish new labels as appropriate. As in the other datasets, we found many controversial annotations that lacked contextual information. In example (23), the author might be referring to another comment with inappropriate content—should the comment still be considered aggressive? In example (24), the annotators seemed to consider the comment as sarcastic, but depending on the context, it could have been a positive comment.

1. (16) Arundathi roy has no right to live in India, if she instigates violence through spreading her dangerous ideology. We Indians should not fall for these kind of TRAITORS. PLEASE BEWARE OF THESE KIND OF PEOPLE. SPREAD AWARENESS. WE NEED TO STOP THESE NON SENSE.—NAG

2. (17) We will not going to show any documents to govt.. We will see what they can do.. The mother f**ker BJP ?—NAG

3. (18) This is the fucking true which today’s generation pretend $\ldots$ [angry emoji]—NAG

4. (19) @Rishita Pandey these guys don’t know meaning of feminism. In simple words it means equality. We are not property of anyone. Thank you.—CAG

5. (20) You’re watching the movie in a completely negative way. I get it that it does have a negative impact, but you’re not giving a balanced analysis.—CAG

6. (21) worst video—CAG

7. (22) Feminist mean the equal right of women and men, why people take it wrong way—CAG

8. (23) Shame $\ldots \ldots$ shame $\ldots \ldots$ —CAG

9. (24) Brother, you’re a Savior for this generation—CAG

4.4 HateBaseTwitter

Overall, the random sample is well annotated. There are only a few samples that can be classified as “contradicting.” Example (25) is labeled as “NEITHER” but contains a derogatory term for women. Other samples with the same word are labeled as “OFFENSIVE.” The same is true for example (26) and example (27). While example (26) contains the same ethnic slur as example (27), the former example is labeled as “NEITHER,” the latter is marked as “OFFENSIVE.” Furthermore, we think that example (27) should rather be marked as “HATE” because it sounds like an insult to a certain group. The examples (28) and (29) contain hateful expressions that are not easy to understand without background knowledge, due to the terms “hopper” and “beaner.”

1. (25) You wanna find hoes on here? Just follow the chicks who reply with heart eyes under sex gifs—NEITHER

2. (26) Log off nigger RT @PoloKingBC: #relationshipgoals—NEITHER

3. (27) No way all u niggers are employees of the month—OFFENSIVE

4. (28) in real life and online I follow that. sorry folks but just cause I look like a border hopper don’t mean I am one. born and raised here—HATE

5. (29) @HuffingtonPost im American, don’t give 2 shits about the beaners, ship them beaners back—HATE

Upon inspection, we noticed that almost every HOF instance contained an instance of a small set of abusive words. To validate our observation, we preprocessed the whole dataset by first locating all “OFFENSIVE” and “HATE” instances and removing stop words and numbers. We then computed the fifteen most frequent terms listed in Table 4. The frequencies of all abusive words amount to 27,465. Out of the 15 most frequent terms, eight are offensive/hateful. Therefore, models trained on such data cannot be generalized to texts from other sources because they will likely focus on these terms.

Table 4. Fifteen most frequent terms (after preprocessing) in HateBaseTwitter

4.5 HASOC

4.6 GermEval 2018

From the samples we manually reviewed, we found only four controversial annotations, a rate of 0.8%. The examples (39) and (40) involve a politician and are tagged as “OTHER,” but we would classify them as abusive. The examples (41) and (42) are probably cases of “missing context” because they do not contain obvious abusive content.

1. (39) Das EU Experiment soll einer kosmetischen Behandlung unterzogen werden, nur taugt die Kosmetikerin Merkel in der Wurzel nichts [Winking Face with Tongue] ÜBERFORDERT—OTHER “The EU experiment is to be subjected to a cosmetic treatment, but the beautician Merkel is fundamentally useless [Winking Face with Tongue] OVERLOADED”

2. (40) Merkel lernte u. studierte in der DDR für “Null” $\gt$ ihre wichtigste Erkenntnis, halte deine Wähler DUMM, denn dann keine “dummen Gedanken” [Winking Face with Tongue]—OTHER “Merkel learned and studied in the GDR for ‘free’ $\gt$ her most important insight, keep your voters STUPID, because then no ‘stupid thoughts’ [Winking Face with Tongue]”

3. (41) Liebe Freunde, viel zu lange geschwiegen!Was hat diese Person verbrochen?—OFFENSE “Dear friends, silent for far too long! What has this person done wrong?”

4. (42) @Sakoelabo @Padit1337 @SawsanChebli Du bist jung und schön—OFFENSE “@Sakoelabo @Padit1337 @SawsanChebli You are young and beautiful”

4.7 GermEval 2019

From the sample, only eight posts are controversial, a rate of 1.6%. They are related to lack of context or the vagueness between factual aspect and abusive attitude. As discussed with other datasets, some posts cannot be clearly classified as such without the proper context. The example (43) is annotated as “OFFENSE,” but it is difficult to understand this decision without the proper context. The word “Mist” (en: crap) could imply an offensive stance, but it is a very common word in German that can be used without insulting intent. Example (44) is marked “OTHER,” but the person “Wagenknecht” and the party “Die Linke” are humiliated in this example. In example (45), the author considers #bolsonaro a fascist and the post is labeled as “OTHER.” Even though Bolsonaro is often criticized for his authoritarian dictatorial tendencies, it is debatable whether this post should rather be labeled as “OFFENSE.” In general, criticism of politicians is widely accepted, even if it is often abusive. Example (46) is classified as “OFFENSE,” but if the claimed facts in this example are true, then this tweet is not offensive.

1. (43) @maxihome27 für Ohne Worte $\ldots$ schreibste ganz schön viel $\ldots$ Mist! =)—OFFENSE “@maxihome27 for without words $\ldots$ you write a lot $\ldots$ crap! =)”

2. (44) @JaMa08768523 @b_riexinger @dieLinke $\ldots$ weil es die Wagenknecht-Partei ist $\ldots$ was erwartest du denn? [face with rolling eyes]—OTHER “@JaMa08768523 @b_riexinger @dieLinke $\ldots$ because it’s the Wagenknecht party $\ldots$ what do you expect? [face with rolling eyes]”

3. (45) @SPIEGELONLINE #bolsonaro ist ein Faschist sehr geehrtes Nachrichtenmagazin—OTHER “@SPIEGELONLINE #bolsonaro is a fascist dear news magazine”

4. (46) Nicht nur Poroschenko verdient gut,auch Deutsche Politiker verdienen gut an das Schwarzgeschäft Rüstung—OFFENSE “Not only Poroshenko earns well, German politicians also earn well from the black market of armaments”

4.8 GermEval 2021

When we manually reviewed the data annotations, we found a total of 41 controversial annotations, a rate of 8.2%. One problem with the annotation, which also occurs with other datasets, is the lack of context. Specifically for the creation of the GermEval 2021 dataset, context is known to be considered in the annotation process, but it is not provided in the dataset. Many comments such as the examples (47)–(49) do not in themselves satisfy any of the features in the annotation guidelines. However, veiled toxic features such as discredit, discrimination, or sarcasm could apply in these cases if the context annotation and/or higher-level annotations are taken into account. This means that such samples have ambivalent meanings that lack crucial information, which will most likely “confuse” a classification model during training. The examples (50) and (51) show that there are borderline cases where the evaluation is subjective and labeling can easily become arbitrary. This starts getting an issue as soon as there are conflicting labels, as it can be seen in the examples (52) and (53). Example (52) was classified as “TOXIC,” most likely due to the word “kotzen” (en: puke) being profane. Example (53) was classified as “NOT TOXIC,” even though it contains the same word. In both examples, there are no other indicators for abusive language.

1. (47) @USER Oh weh das möchten wir uns garnicht vorstellen.—TOXIC “@USER Oh dear we do not want to imagine that at all”

2. (48) @USER hat eine Ähnlichkeit mit ihnen—NOT TOXIC “@USER has a resemblance with you”

3. (49) Nach mir die Sintflut [Smiling emoji]—TOXIC Difficult to translate. It is a mostly negatively connotated German proverb to express one’s indifference about a situation.

4. (50) @USER weil man nicht über Islmaistengewalt öffentlich reden will wächst die Afd!!—TOXIC “@USER the Afd grows because no one publicly talks about Islamist violence!!”

5. (51) Klatscht das Publikum für jeden Mist—NOT TOXIC “The audience claps for every crap”

6. (52) wo kommt man live in die Sendung mit den nachrichten !!!! ich könnt kotzen—TOXIC “which show takes such news live on air !!!! this sucks”

7. (53) Und wieder weg – es geht um diese linksgrünen und jedes 2 Wort AfD zum kotzen—NOT TOXIC “And away again—it’s about these left-green and every 2 word AfD this sucks”

4.9 Summary

The analysis revealed that the percentage of controversial annotations varies significantly from dataset to dataset. As a result of our analysis, we define the following main classes of controversial annotations: (i) Contradictory: content with the same abusive words is labeled differently, (ii) missing context: the abusiveness of a content cannot be determined if there is no context, (iii) fact check: the decision on abusiveness requires verification of the facts presented in the content, and (iv) borderline cases: content cannot be easily annotated.

OLID contains a lot of insulting instances that have not been labeled as such. This is mainly due to the fact that insults are perceived subjectively. The problem with HatEval arises from the specific targets, that is, even very abusive content is not labeled as such if it is not directed against women and/or immigrants. The loose definition of “OAG” and “CAG” in TRAC-2020 leads annotators to rely on their intuition. As a result, we believe that some comments are wrongly classified. In the HateBaseTwitter sample, we found hardly any controversial annotations but it became obvious that the data were collected by using abusive keywords. The same is true for the German HASOC part, where hardly any controversial annotation was found. In contrast, we found 25% of the English HASOC part to be controversial. In opposition to GermEval 2018 and GermEval 2019, where hardly any controversial annotations were found, GermEval 2021 contains particularly many controversial annotations. Mostly, this is attributed to a lack of context. Posts do not sound abusive per se, but in a given context they obviously are. Table 5 gives an overview of which dataset falls into which classes.

Table 5. Classes of controversial annotations in each dataset

To achieve higher-quality annotations, we recommend the following actions for the respective classes. For missing context, we recommend providing context information, for example, the preceding and following post. For borderline cases, we recommend providing a more distinguishable annotation scheme and clear definitions for each target class. The other two classes would require additional resources in the annotation process. Contradictory cases could be prevented by having repeated discussions between the annotators. Fact check would require additional time to research the truthfulness of claimed facts.

5.1 Machine learning models

Three machine learning classifiers are used in the experimental setup, that is, Logistic Regression (LG) from the linear model series, Complement Naive Bayes (CNB) from the Naive Bayes model series, and Linear Support Vector Classifier (LSVC) from the SVM model series of the scikit-learn library.Footnote ^c Features are obtained from the default CountVectorizer with case-insensitive word representation. During preprocessing, we replaced “&” by “und” (en: and) and “ $\gt$ ” by “folglich” (en: consequently; users use this character to indicate a conclusion), masked user mentions by “user” and URLs by “http” (if not already done by the organizers), replaced emojis with their literal equivalents using the emoji library,Footnote ^d and segmented hashtags into their word components using the current state-of-the-art hashformer libraryFootnote ^e (a German GPT-2 modelFootnote ^f was chosen as segmenter model, but not a reranker model) in all three datasets. We keep the original training and test data split as provided by organizers of the shared tasks.

5.2 Intra-dataset results

To put our generalization results into perspective, we first perform intra-datasets experiments as to obtain a baseline. These experiments are not designed to achieve state-of-the-art results. Table 6 shows the performance results for these models.

Table 6. Results of the intra-dataset experiments that serve as our baseline (top) and the results of the generalization experiments (bottom). All values shown are macro-averaged F1 scores

5.3 Cross-dataset results

For our generalization experiments, we start with GermEval2018 as training data and evaluate on the other test sets. Then, we incrementally add the other GermEval training sets and evaluate on each test set. The generalization performance results of our experiments are also shown in Table 6.

The GermEval2018 dataset shows a decent generalization ability on the GermEval2019 test set with a plus of 0.19% in F1 score on average across all models compared to the GermEval2019 baseline results. For GermEval2021, there is an average loss of –3.23% compared to the baseline models.

Since the GermEval2018 and GermEval2019 datasets are the most similar, we expected to see an increase in performance when combining the data. As the results show, this expectation was confirmed with an increase of 2.51% on the GermEval2018 test set and 3.7% on GermEval2019 on average across all models. However, there is an average loss of –3.76% for the GermEval2021 test set compared to the GermEval2021 baseline.

Combining all GermEval training sets does not further improve generalization ability, but produces similar results as combining GermEval2018 and GermEval2019. Adding the GermEval2021 training data helps with the test set, but still results in a loss of –1.87%.

In summary, the generalization from GermEval2018 to GermEval2019 worked significantly better than to GermEval2021. We assume that three factors are responsible for this. First, the source of data collection differs: GermEval2018 + 2019 was sourced from Twitter and GermEval2021 was from Facebook. Second, an important difference in the annotation criteria was that context was explicitly considered in the labeling of GermEval2021, but not in GermEval2018 and GermEval2019. Third, the abusive categories in GermEval2021 contain more types of abusive subcategories than the other datasets. Since all experiments resulted in a decrease in F1 score on GermEval2021, we tested the generalization properties of the GermEval2021 training data on the other test sets (see Table 6). The results show an average loss of –8.22% on GermEval2018 and –5.23% on GermEval2019, hence confirming that the differences between GermEval2018 + 2019 and GermEval2021 play a role in generalization. The GermEval2018 and Germeval2019 datasets can be successfully used in combination for training of more effective models when tested with either test set. The differences in political opinions (left-wing vs. right-wing) do not seem to affect the models.