Characteristics of Group Mentions in Political Texts

One of the central methodological challenges in identifying mentions of social groups in political text and speech is that they are linguistically extremely diverse. First, the number of social groups that can be referred to in a given political context is typically large. The list is already long if one considers only groups that are defined based on socio-demographic characteristics such as age or generation, gender, race, or ethnicity (cf. Chandra Reference Chandra2012). And if one considers that objective membership in different group categories is often nested and intersectional, the list grows further. For example, a mention of ‘people living and working in rural areas’ refers to members of the rural population who are workers. As a case in point, Thau (Reference Thau2019) conducted a manual content analysis of group appeals in British party manifestos and identified more than 2.7 thousand unique ways in which the Conservative and Labour parties referred to economically or socio-demographically defined groups (see Figure 1 and Table F2).

Figure 1.

Unique n-grams in human-annotated data collected by Thau (Reference Thau2019) and in the Dolinsky-Huber-Horne (DHH) dictionary compiled by Dolinsky et al. (Reference Dolinsky, Horne and Huber2023) by social group category.

Second, political actors do not only refer to groups using socio-demographic markers but also discursively construct groups by emphasizing people’s shared values, norms, circumstances, and commonalities in other attributes. For example, phrases like ‘the needy in our country’ and ‘the wretched of the world’ (see Table 1), ‘those with the broadest shoulders,’ or ‘those who work hard and do the right thing’ do not refer to clearly circumscribed socio-demographic groups, but they likely still appeal strongly to people with corresponding self-conceptions and identities (Bornschier et al. Reference Bornschier, Häusermann, Zollinger and Colombo2021). Drawing again on the data collected by Thau (Reference Thau2019), we argue that this phenomenon should not be neglected. Thirty-one per cent of social group appeals in his data were assigned to the ‘other’ social group category as the mentioned groups did not fit into any of his economic or socio-demographic group categories (see Table F2).

Table 1.

Examples of group mentions in sentences drawn from British mainstream party manifestos: the highlighted text spans the identified groups mentioned in each sentence

A third reason why social group mentions in political texts are linguistically extremely varied is that for any given social group, there are various lexically different ways to refer to it. For one, there are many indirect ways to refer to a group. For example, the phrases ‘the unemployed’ and ‘those out of work’ refer to the same social group. For another, many references to groups use descriptive language, such as ‘the first generation to know we are destroying the environment, and the last generation with a chance to do something about it before it is too late’.

Established Methods and their Limitations

The linguistic diversity of social group mentions in political rhetoric has two important methodological implications. First, as illustrated in Table 1, the phrases used to mention, refer to, or address social groups in political text often span multiple words. Second, any sentence can mention no, one, or several social groups. Consequently, reliable detection and extraction of social group mentions require identifying the words used to refer to or describe social groups in a text while not knowing a priori how many unique mentions it contains, where the mentions are located in the text, and how many words a given mention spans.

To cope with these challenges, researchers studying groups-based rhetoric based on political text currently have two options: manual content analysis and automated dictionary measurement. These two approaches are well-established in the applications to sentence- and document-level classification (cf. Barberá et al. Reference Barberá, Boydstun, Linn, McMahon and Nagler2021; Quinn et al. Reference Quinn, Monroe, Colaresi, Crespin and Radev2010). However, both approaches have clear limitations when applied to extract group mentions from large text corpora.

Manual content analysis identifies group mentions in political texts by tasking coders to locate and extract the relevant text segments referring to groups (for example, Huber Reference Huber2021; Stückelberger and Tresch Reference Stückelberger and Tresch2022; Thau Reference Thau2019, Reference Thau2021) or by indicating this information at the sentence level (Hopkins et al. Reference Hopkins, Lelkes and Wolken2024; Horn et al. Reference Horn, Kevins, Jensen and Van Kersbergen2021). As in other applications (cf. Grimmer and Stewart Reference Grimmer and Stewart2013; Quinn et al. Reference Quinn, Monroe, Colaresi, Crespin and Radev2010), this approach can be considered the most valid compared to semi- or fully automated methods. Human coders can read and interpret texts, allowing them to spot simple group mentions but also more complex ones, like the abstract or descriptive multi-word examples included in Table 1 above.

However, manual content analysis is relatively costly (but see Benoit et al. Reference Benoit, Conway, Lauderdale, Laver and Mikhaylov2016). Researchers need to hire annotators. Moreover, collecting manual annotations is time-consuming for large corpora.Footnote ¹ Consequently, studies that have applied manual content analysis to study group-related rhetoric use either text corpora of limited size, focus on a small set of political parties, and/or limited periods.

The dictionary approach is more resource-efficient as it enables detection mentions of predefined groups automatically by searching for matches to a list of group keywords (cf. Dolinsky et al. Reference Dolinsky, Horne and Huber2023). The only required human input to dictionary-based measurement is to define a list of keywords that reflect the potential ways the social group(s) of interest are mentioned in a corpus.

However, considering how linguistically varied social group references are in political texts, we should expect that compiling a comprehensive list of relevant keywords will be very challenging in many applications, especially since group mentions usually span multiple words, are often indirect, and potentially discursively invoke groups in abstract ways. For example, a dictionary might contain the keyword ‘the unemployed’ but fail to recognize semantically similar phrases like ‘those out of work’.Footnote ² Figure 1 underscores this argument, showing across different social group categories how the number of keywords and keyword patterns in a dictionary compiled by Dolinsky et al. (Reference Dolinsky, Horne and Huber2023) for detecting social group mentions in British party manifestos compares the number of unique mentions Thau’s coders have identified. This shows that even experts in group appeals research who have employed an iterative strategy to identify relevant keywords and patterns arrive at much shorter lists of phrases than is possible through direct human annotation of the target corpus.

In the supplementary materials, we present analyses that support our argument and justify our concerns. First, we apply the dictionary from Dolinsky et al. (Reference Dolinsky, Horne and Huber2023) to our and Thau’s human-annotated texts,Footnote ³ finding modest precision but poor recall at both mention and sentence levels (see Tables G1 and G2). Additionally, our analyses suggest that semi-automated dictionary expansion techniques are not a simple solution. For instance, when using a pretrained word embedding model to find relevant keywords (cf. King et al. Reference King, Lam and Roberts2017), many multi-word phrases are missing from the model’s vocabulary. We estimate that considering the top $k =$ 10 most similar words for each ‘seed’ keyword would require reviewing 1,412 words and phrases (see Table G3). Furthermore, skipping human review in dictionary expansion (cf. Osnabrügge et al. Reference Osnabrügge, Hobolt and Rodon2021b) by adding all $k$ most similar words do not improve reliability (see Table G4), as it increases recall but reduces precision (see Figure G3).

To summarize, manual content analysis allows valid measurement of social group mentions in political texts but is resource-intensive, and, when adopting a sentence-level classification approach, it means discarding empirically interesting variation. By contrast, dictionary-based measurement promises resource efficiency, but it limits reliable detection demonstrably, likely especially so for groups without clear-cut membership criteria and groups that can be referred to in many lexically different ways.

A Supervised Token Classification Approach

We propose a method that allows researchers to automatically identify and extract mentions of groups in political texts with a limited manual labeling effort. Our method applies supervised learning to detect and extract mentions of social groups in political texts. It strikes a favourable balance between the objectives of reliable and valid detection on the one hand and scalability on the other.

After theoretically defining the concept, the first step of our supervised learning approach is to task human coders to highlight all mentions of social groups in a set of sentences sampled from a target corpus. This step mirrors the procedures adopted in existing manual content analysis studies. However, what distinguishes our approach is that we preserve the verbatim mentions of groups where and how they occur in texts.Footnote ⁴ The first row in Figure 2 illustrates what the annotations we collect look like. By tasking coders with highlighting all group mentions in a sentence, we can determine the characters that belong to individual group mentions. This means that in each labeled sentence, no, one, or several spans of characters might be marked as mentioning a group (see Table 1 for examples).

Figure 2.

From sentence annotation to extracted mention. Highlighted spans are converted into token-level labels. Labels ‘B’ and ‘I’ indicate tokens that are at the ‘beginning’ or ‘inside’, the ‘O’ those outside of a group mention. The token classifier predicts label probabilities, which indicate a token’s most likely label. Predicted mentions can be determined from token-level predicted labels.

In the second step, we use this information as data for supervised learning. Specifically, we train a supervised classifier for token classification. Token classification means to assign each word in a sentence a single label from some predefined categories. Enabling this requires converting the annotations into word-level labels. This is illustrated in the second panel of Figure 2. From the annotations we have collected in the first step, we know for each group mentioned in a sentence at which character it starts and ends. Tokenizing the sentence into words, we can determine for each word in the sentence whether or not it belongs to a mention of a group. Further, for words that belong to such a mention, we can determine whether the word is at the beginning of the mention or inside of it. As shown in the second row of Table 2, words that do not belong to a mention are labeled ‘O’ to indicate that they are outside of a social group mention. By contrast, words at the beginning or inside of a mention are labeled ‘B’ respectively ‘I’ (cf. Ramshaw and Marcus Reference Ramshaw and Marcus1995).

Table 2.

Summary of test set performances of DeBERTa group mention detection classifiers fine-tuned and evaluated on our corpus of labeled UK manifesto sentences. Values (in brackets) report the average (90 per cent quantile range) of performances of 25 different classifiers fine-tuned in a 5-times repeated 5-fold cross-validation scheme. Columns distinguish between different evaluation schemes (i.e., different ways to compute the eval. metrics)

Note: seqeval is the strict metric proposed by Ramshaw and Marcus (Reference Ramshaw and Marcus1995) and implemented by Nakayama (Reference Nakayama2018).

With word-level labels at hand, the supervised token classification task is to predict each word’s label in a sentence. Provided with multiple labeled sentences in this format, we fine-tune a transformer-based neural network for this task. This approach is commonly applied in named entity recognition, and it has already been adopted for event data extraction (Skorupa Parolin et al. Reference Skorupa Parolin, Hosseini, Hu, Khan, Brandt, Osorio and D’Orazio2022) and the section of references to the people and the elite in German parliamentary speeches (Klamm et al. Reference Klamm, Rehbein and Ponzetto2023). Relying on a pretrained transformer-based model like BERT (Devlin et al. Reference Devlin, Chang, Lee and Toutanova2019), RoBERTa (Liu et al. Reference Liu, Lin, Shi and Zhao2021), or DeBERTa (He et al. Reference He, Liu, Gao and Chen2021) for this task allows accounting for words’ sentence context when learning to predict their labels. This is impossible with standard bag-of-words methods (cf. Timoneda and Vallejo Vera Reference Timoneda and Vallejo Vera2025).

The result of this second step is a fine-tuned token classification model that can be applied to detect and extract mentions of social groups in political texts. As shown in the third panel in Figure 2, the label class that receives the highest predicted probability for a word is treated as its predicted label. And, as shown in the last panel of Figure 2, this classifier output can be parsed to extract the words belonging to the (predicted) group mention(s) in a sentence.

In the third step, the fine-tuned supervised token classifier can be applied to unlabeled texts to identify and extract mentions of social groups that have not been in the training data. This enables automated labeling and extraction of group mentions in large text corpora.

Our proposed method contrasts with established approaches to quantifying group-based rhetoric in political texts in three ways. First, it contrasts with dictionary-based measurement in that we presume that recognizing concrete group mentions in a text is more reliable than selecting indicative words or phrases a priori. Second, in contrast to the manual content analysis approach, we leverage the benefits of automation through supervised learning. This saves researchers the time and costs associated with manual content analysis (cf. Barberá et al. Reference Barberá, Boydstun, Linn, McMahon and Nagler2021; Grimmer and Stewart Reference Grimmer and Stewart2013). Third, in contrast to sentence-level classification approaches, we annotate, model, and predict the text passages that represent group mentions at the word level. Consequently, our approach preserves the lexical diversity and linguistic variability of group mentions as they occur in political texts, which will enable more detailed analyses of group-centered political rhetoric.

Reliability: Evaluation in British Party Manifestos

We first focus on detecting and extracting social groups mentioned in British parties’ election manifestos. Our case selection is motivated by substantive as well as methodological considerations. From a substantive perspective, we are interested in comparing parties’ social group mentions across elections and parties, for example, to study what distinguishes the groups mentioned by parties with different ideological profiles and programmatic platforms. From a methodological point of view, studying cases that have already been studied in parts in the influential work by Thau (Reference Thau2019) allows us to assess whether the measurements we obtain with our supervised learning method align with those obtained through manual content analysis and, in turn, allows us to assess the validity of our approach.

Convergent Validity with Measurements by Thau Reference Thau (2019)

We next demonstrate that the measurements generated with our approach also converge with those Thau (Reference Thau2019) has obtained through manual content analysis. Thau (Reference Thau2019) has tasked trained coders with manually coding group-based appeals made in UK Labour and Conservative party manifestos (1964–2015). Part of this task is identifying the explicit mentions of targeted social groups.

We use Thau’s data to validate our approach in two ways. First, we assess whether the social group mentions Thau’s coders have identified are also detected by our supervised token classification approach. To answer this question, we have matched the group mentions extracted by Thau’s coders to the manifesto sentences from which they were retrieved,Footnote ¹⁶ applied a group mention detection classifier fine-tuned on our human-labeled UK party manifesto sentences,Footnote ¹⁷ and computed the average mention-level recall per group category in Thau’s coding scheme.Footnote ¹⁸ As shown in Figure F1, our classifier performs overall consistently in his group categories, achieving average recall values above 0.90 in most categories. As discussed in greater detail in Supplementary Material F, the three exceptions to this pattern are explained by how our coding instructions diverge from Thau’s.

Second, we use Thau’s data to compare document-level indicators obtained with our automated method to those obtained with his manual approach. Specifically, we count the number of social group mentions in each party manifesto according to his records and our classifier’s predictions and compare how they correspond. Figure 3 shows a high positive correlation between our and Thau’s estimates. Moreover, our counts are systematically higher, which is expected since Thau has coded group-based appeals, and a group-based appeal implies a group mention but not vice versa.

Figure 3.

Cross-validation of RoBERTa group mentions detection classifier’s predictions against data collected by Thau (Reference Thau2019). Figure compares the numbers of social group mentions identified in a manifesto by Thau (Reference Thau2019, see x-axis) and our classifier (y-axis) in Labour and Conservative party manifestos (1964–2015). Colors indicate parties. The correlation coefficient (with 95 per cent confidence interval) is shown in the top left of the plot panel.

Transfer to Other Parties, Domains, and Countries

The results presented thus far underscore the reliability and validity of our supervised group-mention detection method. However, applied researchers might want to adopt our approach to study group-based rhetoric in texts from other domains, countries, or languages. After all, in comparative politics and neighbouring fields, researchers typically want to compare political elites’ communication behaviour across contexts.

To demonstrate the practical utility of our method, we assess the ‘transferability’ of the models we train on UK party manifesto data. By transferability, we mean the degree to which a classifier fine-tuned on labeled data from a ‘source’ context reliably classifies data from a ‘target’ context, which we consider an important dimension of generalization.

We examine the transferability of the classifiers obtained with our method in three scenarios. First, a cross-party transfer scenario in which we use labeled data from the Conservative and Labour Party manifestos as source data and that of the smaller British parties in our corpus (DUP, Greens, SNP, and UKIP) as target data. Second, we examine cross-lingual transfer using British parties’ English-language manifestos as source documents and German parties’ German-language manifestos as target documents (cf. Licht Reference Licht2023). Third, we examine cross-domain transfer using British parties’ manifestos as source documents and sentences from British House of Commons speeches as target documents (cf. Osnabrügge et al. Reference Osnabrügge, Ash and Morelli2021a). The datasets for these experiments are described in Supplementary Materials A and B.

In all three scenarios, we study zero- and few-shot transfer. Zero-shot transfer means classifying sentences from a ‘target’ context using a classifier solely fine-tuned on labeled sentences from the ‘source’ context. Few-shot transfer, in turn, means to continue fine-tuning this classifier on a few labeled sentences from the target context before applying it to other sentences from this context.

To examine how well our classifiers transfer in these scenarios, we have sampled the labeled data from the source context 50:50 into training and test splits. We have then started with evaluating the zero-shot setup by evaluating a classifier fine-tuned only on labeled sentences from the source context (for example, British manifestos) in the target-context test set (for example, sentences from German party manifestos).Footnote ¹⁹ By also evaluating the classifier in a source-context test set, we can compare the zero-shot transfer performance to the baseline of no transfer. We have then used portions of labeled sentences in the target-context training split to incrementally continue fine-tuning the classifier. For each scenario, we repeated this process with five different random seeds and averaged results across runs to account for uncertainty in fine-tuned classifiers’ performances.

The results from these experiments are reported in Figure 4. The data points in the left-hand plot panels report the results without transfer, that is, from evaluating the classifiers in held-out test set examples from the respective source contexts. In the right-hand plot panels, the data points at x-axis values of 0, in turn, report the results for zero-shot transfer. Across scenarios, we find that zero-shot transfer comes with reliability losses. This should caution applied researchers against applying our pretrained classifiers to detect social group mentions in texts from other domains or languages. However, at least in the cases of cross-party transfer, the reliability losses are relatively modest.

Figure 4

Summary of test set performances in cross-party, cross-lingual, and cross-domain transfer, respectively. The y-axis indicates the performance of classifiers trained on annotated manifesto sentences from the source context (for example, British manifestos) when evaluated on sentences from the target context (for example, German manifestos) in terms of the seqeval F1 score. Points (line ranges) report the average ( $pm$ 1 std. dev.) of performances of 5 different classifiers trained with different random seeds. Cross-party and cross-domain transfer results are based on fine-tuning DeBERTa models, and cross-lingual transfer results are based on fine-tuning XLM-RoBERta models.

However, Figure 4 also shows that the reliability of transfer to the target context can be improved through few-shot fine-tuning, that is, continuing to fine-tune the classifier pretrained on source-context sentences with a few labeled sentences from the target context. In all three transfer scenarios, classifiers’ reliability in classifying target-context examples improves compared to the zero-shot baseline when continuing to fine-tune the classifier with a few hundred labeled sentences from the target context. As a point in case, in the cross-party transfer experiment (Figure 4a), continuing to train it with only 176 labeled sentences (10 per cent of the target corpus) allows matching the F1-score achieved in the source-context test set. Continuing to train with more labeled data from the target context does not improve classification performance in the target context further.

Figure 5.

Social group mentions in Labour and Conservative party manifestos (1983-2015) by Comparative Agendas Project (CAP) policy topic. Note: Sentences CAP-coded using multiclass classifier trained on human-labeled manifestos of same cases (Jennings et al., Reference Jennings, Bevan and John2011) Infrequent CAP policy topics grouped into the ‘other’ category. Topic ‘Immigration’ recoded to topic ‘Civil Rights, Minority Issues, Immigration and Civil Liberties.’

We find a similar initial improvement for cross-domain transfer from UK manifestos to parliamentary speech (see Figure 4c). However, as we continue to adapt the source-context classifier with more and more labeled parliamentary speech sentences, the classifier’s target-context performance becomes more uncertain.

The results for cross-lingual transfer are not as strong (see Figure 4b). This might be explained by the fact that, in this setup, we transfer not only across languages but also party systems and political cultures. Nevertheless, even in the few-shot cross-lingual transfer experiment, 10 per cent of the labeled target corpus (361 labeled sentences) already yields substantial performance improvements relative to the zero-shot baseline.

Overall, our findings on the transferability of our classifiers suggest that, in practice, researchers can start with our pretrained classifiers and adapt them to their target context with a few labeled examples. We thus believe that our approach enables even less well-endowed researchers to size the scalability advantage of our proposed approach. Further, our results suggest that by fine-tuning using a small but diverse and potentially multilingual set of labeled sentences from different domains or countries, our approach could enable reliable detection and retrieval of social group mentions across political contexts. Our results thus highlight our approach’s great promises for large-scale comparative research projects.

What Distinguishes British Parties’ Social Group Focus?

We first examine differences in British parties’ social group focus in how much they emphasize groups in different policy areas, using data from the UK Comparative Agendas Project (CAP; Jennings et al. Reference Jennings, Bevan and John2011). Specifically, we classify manifesto sentences according to the CAP policy topic they discussFootnote ²⁰ and then estimate the prevalence of social group mentions in Labour and Conservative party manifestos sentences (1983–2015) by CAP category.Footnote ²¹

From the group appeals literature, we know that political parties combine policy and group appeals to cater to voters (cf. Huber et al. Reference Huber, Meyer and Wagner2024; Robison et al. Reference Robison, Stubager, Thau and Tilley2021; Thau Reference Thau2023). Since group mentions can reflect parties’ attempts at addressing groups’ interests and shaping their opinions, we generally expect more mentions in policy areas marked by (re)distributive conflict, such as social welfare, compared to discussions about regulatory issues like the economy (Majone Reference Majone1997). But we also expect differences in the emphasis parties place on social groups in policy areas due to divergent incentive systems for acquiring issues (Petrocik, Reference Petrocik1996) and group yield (Huber Reference Huber2021).

Figure 5 presents evidence that supports both expectations. The overall salience of social group mentions in different policy topics aligns with our expectations. Distributive and redistributive policy areas (for example, social welfare, education, and civil rights) are more likely to include social group references than sentences about regulatory matters (for example, transportation, environment). In addition, we observe differences between parties in the degree to which they emphasize social groups when addressing these policy issues. Labour mentions social groups more in their manifestos than the Conservatives when talking about the topics of ‘Social welfare’ and ‘Law, Crime, and Family issues’. A reverse pattern emerges tentatively in their discussion of macroeconomics topics. This suggests that parties emphasize social groups more in areas considered their core competencies, indicating an association between emphasis on social groups and issue ownership (Petrocik Reference Petrocik1996).

Next, we analyze how British political parties distinguish themselves through their references to social groups. Previous studies emphasize that it is not only important whether groups are mentioned but also which groups (Huber, Reference Huber2021; Thau, Reference Thau2021) and how they are referred to (Graf et al. Reference Graf, Rubin, Assilamehou-Kunz, Bianchi, Carnaghi, Fasoli, Finell, Sendén, Shamloo and Tocik2023).

To investigate this, we employ the ‘fightin’ words’ method by Monroe et al. (Reference Monroe, Colaresi and Quinn2008) to the social group mentions identified and extracted by a RoBERTa classifier fine-tuned on our labeled British party manifestos corpus. In this analysis, we focus on manifestos from 2015 to 2019 to allow the inclusion of smaller British parties.

The ‘fightin’ words’ algorithm (Monroe et al. Reference Monroe, Colaresi and Quinn2008) is a bag-of-words method for quantifying differences in word choices between speakers, parties, or any other binary indicator. We use this method to compare the parties’ social group mentions between pairs of parties. Specifically, we apply it to the predicted group mentions extracted from parties’ manifestos after removing common stop words, retaining uni- and bi-grams, and adding skip-grams.

Figure 6 summarizes our findings. The x-axis shows term frequency. The y-axis displays $z$ -scores that quantify how distinctive the words a party uses to refer to social groups when comparing pairs of parties. Higher $z$ -scores indicate more distinctive words.

Figure 6.

Different pairs of parties in terms of the words and phrases that distinguish the social groups the mention in their manifestos for the elections 2015, 2017, and 2019. Note: $z$ -scores indicate words ‘distinctiveness’ and have been obtained by applying the ‘fightin’ words’ method proposed by Monroe et al. (Reference Monroe, Colaresi and Quinn2008) to the social group mentions retrieved by our classifier.

Analyzing Conservative and Labour manifestos, Labour emphasizes workers’ and disadvantaged groups like people [with] disabilities,’ refugees,’ women, and ‘BAME’ (Black, Asian, and minority ethnic) and LGBT communities. By contrast, the Conservative Party focuses on ‘ordinary working [people]’, ‘working families’, ‘British people’, and the middle class (for example, ‘doctors’, ‘entrepreneurs’, and ‘professionals’).

Examining Greens and UKIP along the GAL-TAN dimension, Greens refer distinctively to age- and gender-based groups and disadvantaged communities, while UKIP, like the Conservatives, focuses on ‘the nation’ and ‘British people’, also mentioning immigrants and criminals.

Comparing Labour and the SNP, the center-periphery issue of Scottish independence is evident, with the SNP mentioning ‘[the] people [of] Scotland’, ‘Scottish’, ‘Scotland’s’ people, citizens, etc., as well as ‘Scots’.

The insights into differences in British parties’ social group focus we have generated in the analyses above underscore the practical value of our method. Automating the detection of references to social groups at a very granular level of measurement, our method allows detailed insights into how parties’ group- and issue-based appeals correspond. Further, by extracting the exact words with which parties refer to social groups, our method facilitates inductive discovery and analysis of party rhetoric based on a limited set of human-annotated sentences.

What is more, our evidence presented in section 4.3 suggests that researchers can also harness these advantages of our method for analyzing texts from other domains and languages. For example, this promises new insights into individual legislators’ group-based rhetoric.

Is Group-Based Rhetoric Linked to Emotional Appeals?

Like directly mentioning social groups, emotional language is a powerful rhetorical strategy to appeal to voters (Crabtree et al. Reference Crabtree, Golder, Gschwend and Indriđason2019; Gennaro and Ash Reference Gennaro and Ash2022; Osnabrügge et al. Reference Osnabrügge, Hobolt and Rodon2021b). However, we do not know whether parties combine these two strategies in their campaign communication or use them separately.

We investigate the link between group-based rhetoric and emotional appeals through logistic regression analysis.Footnote ²² We use our sentence-level corpus of automatically labeled Labour and Conservative party manifestos from 1964 to 2019. Our dependent variable measures whether a sentence includes emotional language based on the Linguistic Inquiry and Word Count (LIWC) dictionary (Pennebaker et al. Reference Pennebaker, Boyd, Jordan and Blackburn2015). Specifically, we classified a sentence as containing emotional language (coded 1) if at least one word matched the list of positive and negative emotion words in the LIWC. If a sentence contained no emotional words, we coded it as zero (0). Further, we have created two additional indicators using only positive and negative emotion words, respectively. These alternative outcomes allow us to assess whether positive, negative, or both emotions contribute to the overall association.

Our main explanatory variable measures whether a sentence mentions one or more social groups, and we classify all sentences that contain at least one (predicted) social group mention as 1s and all others as 0s. To account for potential confounders, we control for parties’ positions on the economy and cultural topics using Manifesto Project Data indicators (Lehmann et al. Reference Lehmann, Burst, Matthieß, Regel, Volkens, Weßels and Zehnter2022), whether a party was the prime minister’s party in the year leading up to the election for which the manifesto was written, and the number of words in a sentence. We use these indicators to fit logistic regression models with the binary emotion indicator as the outcome. All our models include election fixed effects, and we cluster standard errors at the level of parties and elections.

Figure 7 presents the coefficient estimates of our logistic regression models for our binary, sentence-level social group mention indicator as odds.Footnote ²³ The odds measure how much more likely a sentence is to contain emotional words when it contains at least one social group mention compared to when it contains no social group mention. Figure 7 shows that sentences that contain at least one social group mention are about 1.2 to 1.4 times more likely to contain emotional words. This association exists with positive and negative emotional language use, as we find positive and statistically significant associations when measuring emotional language use only with positive or negative emotion words in the LIWC dictionary.Footnote ²⁴

Figure 7.

Estimates from logistic regressions analyzing whether sentences that contain group mentions are more likely to contain emotion words. The x-axis reports our estimates of the odds that a sentence contains emotional language when it contains at least one social group mention compared to when it contains no social group mention. Points (line ranges) report the coefficients point estimates (95 per cent confidence intervals) of logistic regression models. The y-axis values differentiate between different emotion dictionary categories.

This analysis underscores that applying our method for automatically detecting social group mentions in political texts enables new empirical insights into the relation between group-based rhetoric and emotional appeals in parties’ campaign communication.