1 Introduction
The way citizens judge politicians is influenced by how they evaluate politicians’ character or personality traits (Bittner and Peterson Reference Bittner and Peterson2018; Hayes Reference Hayes2009; Laustsen and Bor Reference Laustsen and Bor2017). A more favorable evaluation of a politician’s personality increases the likelihood of voting for them and contributes to the formation of positive partisan attitudes (Funk Reference Funk1997; Miller, Wattenberg, and Malanchuk Reference Miller, Wattenberg and Malanchuk1986). The abiding trends toward greater personalization of politics renders voters’ perceptions of politicians’ personality even more important, as individual political actors become more prominent at the expense of collective actors and institutions such as political parties (Pedersen and Rahat Reference Pedersen and Rahat2021; Rahat and Kenig Reference Rahat and Kenig2018).
While a vast literature has studied which personality traits voters look for in politicians (see Van Aelst, Sheafer, and Stanyer Reference Van Aelst, Sheafer and Stanyer2012 for an overview), less is known about the ways in which politicians present their personality in public communication, such as through speeches or social media posts. Previous studies have explored various aspects of politicians’ strategic self-presentation, including their style of representation (Fenno Reference Fenno2003; Gulati Reference Gulati2004; Schürmann and Stier Reference Schürmann and Stier2022) or their private lives (Colliander et al. Reference Colliander, Marder, Falkman, Madestam, Modig and Sagfossen2017; Perloff Reference Perloff2013; Stanyer Reference Stanyer2008; Van Zoonen Reference Van Zoonen2005). However, little research has specifically examined how politicians use public communication to convey personality traits to the public, either strategically or inadvertently. One reason for this gap may be the difficulty of measuring a politician’s public personality when the available information is often fuzzy and ambiguous.
Leveraging recent advances in computational text analysis and political psychology, our study establishes the feasibility of inferring politicians’ public personality from purely verbal personality cues. Although public personality can also be conveyed through nonverbal (e.g., posture, gestures) and paraverbal (e.g., prosodic features) cues, verbal cues (i.e., what politicians say) stand out as both informative (Chen, Qiu, and Ho Reference Chen, Qiu and Ho2020) and widely available for analyzing political figures in public communication. The core methodological contribution of our study is a computational text analysis workflow that detects verbal personality cues in political communication and infers politicians’ public personality traits from these cues.
We build upon a validated personality framework and attendant survey inventory to operationalize two key traits in political psychology: agency and communion. We then apply different methods to automatically infer them at scale in a text corpus comprising social media posts, political speeches, and political interviews. We systematically compare multiple modeling strategies and validate our approach within a structured evaluation framework. Our results suggest that prompt-based approaches using state-of-the-art models, such as DeepSeek-V3 and GPT-4o, show the strongest evidence of measurement validity and are best suited to identify relevant personality cues.
To support transparency and future research, we provide all trained models, code, and data openly.Footnote 1
2 Literature
2.1 Politicians’ Personality Traits Matter
Politicians’ personality plays a central role in explaining political actions, such as partisan attitudes or voting behavior (Best Reference Best2011; Federico and Malka Reference Federico and Malka2018; Schoen and Schumann Reference Schoen, Schumann, Falter, Gabriel and Weßels2005). While politicians and their characteristics have always played a key role in politics, there are growing claims that a new, increasingly personalized political culture of “candidate-centered politics” has emerged (McAllister Reference McAllister, Dalton and Klingemann2007; Pedersen and Rahat Reference Pedersen and Rahat2021). This shift is attributed to a decline in the traditional emotional connections between voters and political parties (Mair, Reference Mair2005) and the transformative impact of a media landscape that prominently includes decentralized social media (Stier et al. Reference Stier, Bleier, Lietz and Strohmaier2018). The growing interest in the characteristics of politicians has led to an increased focus on analyzing politicians’ personalities.
For example, previous research has examined the role of politicians’ personality as a means to understand their ideological orientations and political behavior (George and George Reference George and George1998; Walker, Schafer, and Young Reference Walker, Schafer and Young1998).Footnote 2 This strand of research indicates that conservative politicians often show lower levels of agreeableness and openness but higher levels of conscientiousness, whereas the link to other personality traits is less clear (Hanania Reference Hanania2017; Schumacher and Zettler Reference Schumacher and Zettler2019). Evidence also suggests that politicians, compared to the general population, show higher levels of agreeableness, emotional stability, and openness (Caprara et al. Reference Caprara, Barbaranelli, Consiglio, Picconi and Zimbardo2003). Likewise, personality, particularly higher levels of extraversion and lower levels of agreeableness, is related to higher in-office performance (Joly, Soroka, and Loewen Reference Joly, Soroka and Loewen2019; Lilienfeld et al. Reference Lilienfeld, Waldman, Landfield, Watts, Rubenzer and Faschingbauer2012; Watts et al. Reference Watts2013), media visibility (Amsalem et al. Reference Amsalem, Fogel-Dror, Shenhav and Sheafer2020), or electoral success (Nai Reference Nai2019; Pillai et al. Reference Pillai, Williams, Lowe and Jung2003; Scott and Medeiros Reference Scott and Medeiros2020).
Naturally, voter preferences also display influences of personality-based judgments. For example, voters appear to consistently prefer characteristics such as competence, integrity, or energy in political figures (Aichholzer and Willmann Reference Aichholzer and Willmann2020; Mondak Reference Mondak1995). Caprara and Zimbardo (Reference Caprara and Zimbardo2004) further elaborate that voters’ evaluation of politicians depends on how closely politicians resonate with the voters’ own traits and values. This can manifest through emotional attachments (likability heuristic) or the attribution of positive characteristics (dispositional heuristics). Cross-national studies confirm varying preferences for traits such as emotional stability, extraversion, and conscientiousness in ideal politicians (Aichholzer and Willmann Reference Aichholzer and Willmann2020).
2.2 Politicians’ Public Personality Is Measured Through Cues
In nearly all cases, voters must rely on secondary sources (e.g., media) to assess a politician’s personality. Therefore, voters’ judgments are exclusively based on the public persona projected by the politician. Figure 1 outlines this process in an adaptation of the lens model proposed by Brunswik (Reference Brunswik1952).
Figure 1 Conceptual model (adapted from Brunswik, Reference Brunswik1952).
The key message of the model is that voters and researchers alike never have direct access to a politician’s intrinsic personality. Rather, they have to infer politicians’ personality from (i.e., through the lens of) observable personality cues
${X}_1$
…
${X}_n$
communicated by the politician. These cues, whether verbal, such as statements in speeches or debates, or nonverbal, such as gestures, posture, or facial expressions, are the basis upon which observers form their judgments of a politician’s personality.
The observable personality cues
${X}_1$
…
${X}_n$
are influenced by two main factors: a politician’s intrinsic personality (
${Y}_{IPT}$
) and their strategic self-presentation (
${Y}_S$
), in which they deliberately emphasize or suppress certain traits to appeal to voters (Fenno Reference Fenno2003; Goffman Reference Goffman2016). The relative influence of intrinsic dispositions and strategic behavior in any given cue cannot be determined. However, this uncertainty is not a limitation when the focus is, as in our study, precisely on automating the personality judgments that the public (i.e., voters or news media) would also form based on the cues.
The challenge for inferring politicians’ personality through automated text analyses, however, is twofold: (1) to identify valid cues
${X}_1$
…
${X}_n$
that can serve as indicators for a given personality trait (cue validity), and (2) to correctly utilize these cues to form judgments of that trait (cue utilization).
2.3 Personality Cues Are Inherently Ambiguous
Identifying personality cues and utilizing them to form coherent trait judgments is inherently challenging for two main reasons.
2.3.1 Cue Validity
First, personality traits themselves are not sharply defined or neatly separable—they have the nature of fuzzy concepts (Thielmann and Hilbig Reference Thielmann and Hilbig2019). As such, traits like extraversion and agreeableness do not always have clear-cut boundaries, making it difficult to determine where one ends and another begins (Zettler et al. Reference Zettler, Thielmann, Hilbig and Moshagen2020). In addition, not all traits are equally observable to begin with. Traits such as extraversion often have relatively clear behavioral cues—like a politician’s public speaking style or social engagement—whereas others, such as openness or honesty, are harder to detect.
2.3.2 Cue Utilization
Second, the observable manifestations of these traits—in our case the personality cues
${X}_1$
…
${X}_n$
—are themselves ambiguous. A single behavior can be driven by multiple traits (equifinality), and conversely, a single trait can result in a wide range of behaviors (multifinality). This means that very few behaviors can be considered “pure” indicators of a single trait; instead, personality cues tend to remain ambiguous stimuli open to interpretation. In political contexts, these ambiguities are amplified by the viewer’s own perspective. Partisan or cultural biases can shape how observers interpret the same cue—what one supporter sees as “bold” or “charismatic,” an opponent might perceive as “reckless” or “showboating.” This means that not all observers will use the same available cues in the same way, let alone use them correctly, to infer personality traits.
In sum, personality as a construct is both conceptually and empirically fuzzy, and its behavioral indicators are often ambiguous and context dependent. It is not always straightforward to identify valid, let alone pure, cues of each given personality trait (cue validity). It is also challenging for observers, whether human or computers, to utilize the right cues to correctly infer a given personality trait (cue utilization). This makes the reliable measurement of personality in public discourse particularly demanding.
2.4 How Can We Utilize Personality Cues?
Due to these challenges, methods for measuring politicians’ personalities based on personality cues face considerable limitations (McDonald Reference McDonald2008). Broadly speaking, these methods can be categorized into self-report and observer-report approaches.
Self-report approaches include participants being asked to fill out surveys (Caprara et al. Reference Caprara, Barbaranelli, Consiglio, Picconi and Zimbardo2003; Joly, Soroka, and Loewen Reference Joly, Soroka and Loewen2019; Maier et al. Reference Maier, Oschatz, Stier and Zettler2023; Nørgaard and Klemmensen Reference Nørgaard and Klemmensen2019; Schumacher and Zettler Reference Schumacher and Zettler2019).Footnote 3 However, politicians—especially high-profile politicians—are a hard-to-survey population, and surveys among politicians often suffer from low response rates, which not only make data access challenging but may also lead to biased response patterns. Likewise, time limitations frequently lead researchers to use brief personality inventories, which may fail to fully represent the complexity of a politician’s personality profile. Also, fundamentally measuring politicians’ intrinsic personality in this way is not the same thing as measuring their personality from the observable cues based on which the public would form their personality judgments.
On the other hand, observer-report approaches typically rely on observer ratings derived from various forms of observational data, such as everyday interactions or digital traces from social media. In the study of politicians’ personalities, this can be done using ratings by experts or the general public (de Vries and van Prooijen Reference Vries and van Prooijen2019; Rubenzer, Faschingbauer, and Ones Reference Rubenzer, Faschingbauer and Ones2000; Visser, Book, and Volk Reference Visser, Book and Volk2017; Wright and Tomlinson Reference Wright and Tomlinson2018; Nai and Maier Reference Nai and Maier2024). Although these evaluations are explicitly based on observable personality cues (
${X}_1$
…
${X}_n$
), they can be skewed by the raters’ political beliefs, and these raters have, by definition, only a partial and biased view of the politicians they are evaluating (Wright and Tomlinson Reference Wright and Tomlinson2018). In addition, the interpretation and aggregation of personality cues into traits by raters often hides the direct connection between specific cues and the final evaluation, hampering the observation of short-term changes and specific settings.
2.5 The Need for Computational Approaches
In this article, we therefore argue that computational text analysis provides a valuable yet underutilized tool for measuring politicians’ public personality. Generally, political communication serves as the primary medium for self-presentation, with every statement made by a politician potentially being a personality cue that voters perceive. In the past, research has aimed to study political self-presentation using manual content analysis (Hermann Reference Hermann1980; Winter Reference Winter2005). However, content analysis usually entails a targeted, case-study-specific angle for politicians like Donald Trump and Hillary Clinton (Lee and Lim Reference Lee and Lim2016) or George Bush and Mikhail Gorbachev (Winter et al. Reference Winter, Hermann, Weintraub and Walker1991). Therefore, while manual content analysis likely provides the most valid research approach in a case-study design, it is unnecessarily effortful for analyzing politicians public images on a large scale, given that automated personality prediction from behavioral data has demonstrated performance comparable to traditional approaches (see the meta-analysis by Azucar et al., Reference Azucar, Marengo and Settanni2018).
A notable exception in the use of automated methods is the study by Ramey et al. (Reference Ramey, Klingler and Hollibaugh2019), who used computational techniques to measure the personalities of U.S. Congress members based on their floor speeches. Their approach builds on a rich literature that uses tools like Linguistic Inquiry and Word Count (LIWC) to capture linguistic features—such as the use of second-person pronouns, punctuation, or word length—and link them to personality traits (Chen, Qiu, and Ho Reference Chen, Qiu and Ho2020; Park et al. Reference Park2015; Mairesse et al. Reference Mairesse, Walker, Mehl and Moore2007). Using the pretrained Support Vector Machines for Regression models from Mairesse et al. (Reference Mairesse, Walker, Mehl and Moore2007) and applying it to U.S. Congress floor speeches, they found that their text-based measures were correlated with survey response rates for members of Congress.
While the work of Ramey et al. represents a significant contribution by introducing a computational method to extract politicians’ personality cues, it also has limitations. Their use of a pretrained support vector machine lacks validation against human-annotated data and appears outdated given advances in large language models (Laurer et al. Reference Laurer, van Atteveldt, Casas and Welbers2024; Widmann and Wich Reference Widmann and Wich2022). Moreover, the Big Five traits, though common in personality research, offer little domain-specific nuance for political contexts.
Building on the work of Ramey et al., we, therefore investigate whether computational methods can be used to develop valid, domain-specific automated measures of politicians’ public personalities based on the personality cues they exhibit.
3 Method
3.1 Data
We compiled a corpus comprising a diverse set of public statements from leading German politicians covering the period from 2017 to 2024, encompassing two full legislative terms of the German Bundestag. Our definition of “leading politicians” includes those who served as leaders or co-leaders of both legislative factions and political parties, covering all six parties represented in the German Bundestag. For these 46 politicians (see Appendix A of the Supplementary Material), we collected data from three sources commonly used to study politicians’ public communication: interviews, social media posts, and parliamentary speeches.
3.1.1 Interviews
First, we rely on a novel dataset comprising 99 interviews with political leaders broadcasted in five different media formats (Birkenmaier, Sieber, and Bergstein, Reference Birkenmaier, Laureen and Felix2025). Interview statements provide one of the most natural and authentic sources of politicians’ interactions and self-presentation to the public, ideally lending themselves to capture the public personality of politicians (Schoonvelde, Schumacher, and Bakker Reference Schoonvelde, Schumacher and Bakker2019).
3.1.2 Social Media Posts
Second, we rely on social media data as another highly popular resource for politician’s public communication. The initial data consist of a subsample of 300,000 Facebook posts from a raw corpus of 6.7 million politicians’ posts utilized in Birkenmaier, Sältzer, and Wurthmann (Reference Birkenmaier, Wurthmann and Sältzer2025).
3.1.3 Parliamentary Speeches
Finally, we rely on a corpus of politicians’ speeches in the German Bundestag. We utilize a subset of German MP speeches provided by the Parlspeech database, which provides transcripts of parliamentary speeches of nine representative democracies (Rauh and Schwalbach Reference Rauh and Schwalbach2020). We used only the German subset for our present analyses.
Across all data sources, we defined individual sentences and statements uttered or written by politicians as our unit of analysis. Sentences represent personality cues at a granular level that can be detected and potentially aggregated across texts. Because our primary focus is on detecting verbal personality cues in political communication and inferring politicians’ public personality traits from these cues in text rather than analyzing explanatory factors influencing changes or differences in the public personality, we draw a random sample of 15,000 sentences from each data source as our main corpus.
3.2 Measurement
3.2.1 Workflow
Figure 2 illustrates our workflow. As detailed next, we begin with manual coding to develop and validate a codebook and create a set of labeled data. We then employ computational text analysis using different models and measurement strategies and assess the validity of our measures using a comprehensive validation framework (Birkenmaier, Wagner, and Lechner Reference Birkenmaier, Wagner and Lechner2024).Footnote 4
Figure 2 Workflow.
3.2.1.1 Trait Selection.
To select personality traits that can be measured using computational text analysis, we focus on two traits that are widely used in personality and political science research: agency and communion (Bakan Reference Bakan1966). These so-called “meta-traits” provide a higher order structure that captures common variance among lower order traits, including the Big Five, and consistently emerge across diverse personality models (e.g., Partsch, Bluemke, and Lechner Reference Partsch, Bluemke and Lechner2022; Saucier et al. Reference Saucier2014). Adopting the terminology of Hogan (Reference Hogan1982), agency can be thought of as the drive to “get ahead,” whereas communion reflects the drive to “get along.” These dimensions constitute key dimensions of person perception and important behavioral determinants in both personal and political environments (Entringer, Gebauer, and Paulhus Reference Entringer, Gebauer and Paulhus2022).
To guide our operationalization of agency and communion, we rely on a validated framework and survey inventory to assess politicians’ personalities proposed by Lechner et al. (Reference Lechner, Aichholzer, Birkenmaier and Bluemke2025). This inventory comprises 15 domain-specific personality facets specifically tailored to assessing politicians’ personality. Compared to global and general personality frameworks such as the Big Five (Costa Jr. and McCrae Reference Costa and McCrae1995), the dimensions in the Framework by Lechner et al. offer a validated, domain-specific framework for politicians’ personality traits, providing both a clear definition and a conceptual basis for subsequent computational text analysis. In our analysis, we focus on the two prototypical traits from their framework that capture the essence of agency and communion, respectively. This keeps our method interpretable, theoretically grounded, and capable of focusing on the validation of measuring these traits using personality cues in textual data.
Agency: In the Lechner et al. inventory, the trait Assertiveness & Will to Take Control is the key trait representing agency. It reflects the willingness to be a leader as well as to achieve power and one’s assertiveness. The items for this trait characterize a politician who is “power-hungry, claims leadership,” “assertive, dominant,” and “a born leader.”
Communion: In the Lechner et al. inventory, Empathy & Compassion constitutes the key trait representing communion.Footnote 5 It reflects the ability to express and act on feelings of empathy and compassion toward others. The items for this trait characterize a politician who “is empathetic, shows genuine compassion,” “understands the needs of disadvantaged people,” and “is compassionate, has a soft heart.”Footnote 6
3.2.1.2 Trait Conceptualization.
For each trait, we define both weak and strong cues that reflect varying degrees of explicitness in the signaling of personality cues. Personality cues can differ in both strength and relevance, as they vary in how clearly and accurately they reflect the traits being judged in real-world context (Funder Reference Funder1995). In our case, strong cues refer to clear and direct statements that contain self-disclosure or direct references to a personality trait ascribed to oneself (e.g., “I am a very stable genius”), whereas weak cues are more ambiguous or indirect and might be more open to subjective interpretation by the receiver of the cue. Agency, for example, can be explicitly signaled through strong statements such as “As the leader of this party, I will ensure that this policy is implemented, regardless of who stands in the way.” In contrast, weaker agency cues might include phrases that can potentially be subject to human interpretation, such as “We will push this policy forward and ensure its implementation.” While the latter statement suggests a call to action, it is much more passive and does not clarify whether the action entails the speaker actively overcoming resistance.
3.2.1.3 Codebook Development and Validation.
We developed the codebook in multiple steps. Initially, we only coded a subset of sentences containing elements of personal attributes using self-referential words as filtering keywords (e.g., “I,” “we,” “mine”; see Appendix C of the Supplementary Material). Coders annotated these sentences while providing continuous feedback to refine the codebook. Once coders achieved a consensus on trait assignments, we expanded the coding to a random selection of sentences from the full dataset to improve generalizability. Ultimately, a team of three annotators (i.e., the first author and two research assistants) annotated 1,000 sentences. We computed Krippendorff’s alpha as a measure of multilabel agreement, with values above α > .70 deemed as acceptable (Krippendorff Reference Krippendorff2018). After demonstrating sufficient agreement on the 1,000 sentences (see Section 4), we proceeded to generate a human-labeled corpus containing at least 200 examples per class (“agency,” “communion,” or “none”) for both weak and strong cues that we can use to compare our computational scores against (see Appendix D of the Supplementary Material for the final codebook).
3.2.2 Computational Analysis
With the growing application of large language models (LLMs) in the social sciences, there is an ongoing debate concerning the best strategies for computational text classification (Gilardi, Alizadeh, and Kubli Reference Gilardi, Alizadeh and Kubli2023; Ziems et al. Reference Ziems, Held, Shaikh, Chen, Zhang and Yang2024). Therefore, in this article, we systematically test different strategies and computational models to determine the best-performing architecture to detect valid personality cues in textual data.
For all models, we test their performance on different dataset variants. In the first dataset variant, referred to as strong, we only include examples of strong cues for agency and communion. In the second variant, referred to as weak + strong, we include an equal proportion of weak and strong cues for agency and communion, representing a more inclusive (but potentially less clear-cut) definition of personality trait expressions. To address class imbalance, we assess performance on both a balanced dataset, where the three classes (“agency,” “communion,” and “none”) are equally represented, and an unbalanced dataset, where “none” appears twice as often as the other classes.
We compare three major strategies across multiple model architectures (see Figure 2) and assess performance on different dataset variants. All hyperparameters and computational resources used in our experiments are documented in Appendices E and F of the Supplementary Material.
For the supervised strategy (Strategy 1), we follow the approach by Ramey et al. (Reference Ramey, Klingler and Hollibaugh2019) and train a Support Vector Machine (SVM) as a baseline model on the human-annotated data. SVMs are fast and computationally efficient, as they are trained on a bag-of-words representation—a simple encoding of word frequency that lacks contextual information. This makes them particularly suited for detecting direct lexical cues (e.g., explicit words connected to a personality traits), but less effective at picking up ambiguous or implicit personality cues. We compute fivefold cross-validation, meaning the data are split into five parts and the model is trained and tested five times, each time holding out a different part for testing, using a train-test split of 80/20.
For fine-tuning a pretrained LLM (Strategy 2), we rely on a 279M parameters XLM-RoBERTa (base) model (Conneau et al. Reference Conneau2020) using common hyperparameters. As a transformer-based model, XLM-RoBERTa captures contextual relationships between words, and fine-tuning it on our annotated data allows it to adapt to domain-specific language, making it well-suited for detecting implicit or contextually embedded personality cues. We compute threefold cross-validation, using a train-test split of 80/20.
For the prompting approach (Strategy 3), we prompt three different generative instruction tuned LLM, following current research exploring generative models for classification (Gilardi, Alizadeh, and Kubli Reference Gilardi, Alizadeh and Kubli2023; Ziems et al. Reference Ziems, Held, Shaikh, Chen, Zhang and Yang2024). Generative models are particularly effective at detecting subtle or fuzzy personality signals without additional training, as they draw on broad pretraining and are optimized to interpret tasks from instructions alone. We rely on three models: a proprietary instruction tuned GPT-4o, and two open-weight models Llama 3-8B (Touvron et al. Reference Touvron2023) and DeepSeek-V3 (DeepSeek-AI et al. Reference DeepSeek-AI2024). For each model, we utilize different prompting templates (see Appendix G of the Supplementary Material) tailored to dataset variants featuring either exclusively strong or a combination of strong + weak cues, as outlined in our codebook. Additionally, our approach distinguishes between zero-shot and few-shot settings, with the latter including class-specific examples. Each prompt directs the respective model to classify politicians’ statements into one of three categories (“agency,” “communion,” and “none”) and to provide a brief, one-sentence explanation for its decision in a structured format.
To mitigate the effects of randomness in training and prompting, we run each of our models with three different random seeds and report the macroaveraged F1 performance across runs, which reflects the harmonic mean of precision and recall computed separately for each class and averaged to avoid bias toward dominant categories.
To validate our measures, we take a systematic approach to validation and rely on the ValiText framework (Birkenmaier et al., Reference Birkenmaier, Wagner and Lechner2024). Appendix H of the Supplementary Material outlines our validation strategy in detail, and Appendix I of the Supplementary Material contains the checklist templates documenting each step in the validation process.
4 Results
4.1 Substantive Validation
As outlined in our method section, we employ a domain-specific personality framework and attendant survey inventory as the conceptual basis for operationalizing personality cues. As detailed in Lechner et al. (Reference Lechner, Aichholzer, Birkenmaier and Bluemke2025), the substantive validity of this personality framework was ensured during its development process through multiple means, including an analysis of extant work on personality assessment in general and politicians’ personality traits more specifically; input and reviews by experts from psychometrics and political science; as well as extensive tests of structural and external validity of the resulting inventory in multiple samples of voters.
We then chose the prototypical traits of the framework to operationalize personality cues for agency and communion. To do so, we developed a codebook and conducted manual coding to evaluate whether human annotators could reliably identify corresponding personality cues. Our analysis demonstrated sufficient agreement, with Krippendorff’s alpha of .73 (strong) and .72 (weak + strong) for the overall identification of the presence of personality cues, and agreement ranging from .70 to .78 for “agency” and “communion” (see Table 1).
Table 1 Interrater agreement for human coding (n = 1,000).
Regarding label distribution, the human annotators classified most sentences as “none,” with 94% for strong cues and 89% for combined weak + strong cues. Among sentences containing personality cues, “communion” appeared more frequently (approximately 56% of strong cues and 63% of weak + strong cues) than “agency” (around 44% of strong cues and 38% of weak + strong cues).Footnote 7
4.2 Comparison of Computational Scores with Human-annotated Test Set (External Validation)
We next compared how the different computational text analysis models would compare to the human-annotated test set, using macro-F1 score as the performance measure. Figure 3 displays the performance of all models across balanced and unbalanced test and training datasets.Footnote 8 Red points indicate results on the strong datasets that only contain strong personality cues, and blue triangles indicate results on the combined weak + strong datasets with an equal proportion of weak and strong personality cues. Overall, our results show that DeepSeek-V3 and GPT-4o demonstrated the highest performance across dataset variants. As Table 2 shows, DeepSeek-V3 performed best on both balanced and unbalanced datasets, attaining a macro-F1 score of 0.77. GPT-4o performed similarly well, achieving a macro-F1 score of 0.76 on the strong-balanced dataset, and slightly worse performance on the balanced and unbalanced dataset. In contrast, the Llama 3-8B model showed significantly lower performance, with the highest F1 scores reaching approximately 0.60 for the balanced dataset and 0.55 for the unbalanced dataset, along with greater variability across training runs.
Figure 3 Comparison of classifier performance for balanced and unbalanced dataset (n = 600). The y-axis depicts the mean macro-F1 score across three random seeds. Error bars indicate standard deviation across random seeds.
Table 2 Overall best-performing models on the human-annotated test data across three random seeds.
Likewise, models using a semisupervised (XLM-RoBERTa) or supervised (SVM) strategy exhibited also lower macro-F1 scores. The SVM model performed reasonably well on the balanced dataset with strong labels (F1 = 0.63, but with great variability as indicated by the error bars), but performed worse on the weak + strong dataset (F1 = 0.59) and both unbalanced datasets (F1 = 0.55 and 0.61, respectively). The XLM-RoBERTa model showed only slight improvements over SVM for the balanced dataset, with its best performance recorded for the balanced strong-labeled dataset (F1 = 0.70).
4.3 Functional Tests (Structural Validation)
In addition to the human-label comparison, we conduct systematic tests of the different models’ performance on survey items drawn from Lechner et al.’s (Reference Lechner, Aichholzer, Birkenmaier and Bluemke2025) original personality inventory as well as on purposefully chosen sample sentences from the corpus to evaluate whether models can reliably predict explicitly defined test cases (units). Specifically, each model was tested with 18 test units: (a) the three self-report items designed to reflect each associated trait (for instance, “I show understanding for the needs of disadvantaged people” for communion or “I am a born leader” for agency); (b) the three opposing items per trait, expected to be classified as “other” (e.g., “I am hard, cold-hearted” as the opposite pole for communion or “I prefer to stay in the background” for the opposite pole for agency); and (c) six sentences from our original corpus, which we identified as unrelated to any personality cues and that also should be labeled “other” as well (for a full list of test units, see Appendix J of the Supplementary Material).
Our results are displayed in Figure 4. A well-performing model’s confusion matrix should ideally display three green predictions in the lower left corner (representing three correctly predicted “agency” items), three green predictions in the center (corresponding to three correctly predicted “communion” items), and twelve green predictions in the upper right corner (covering the six “other” sentences from the original corpus and the three opposing poles for both “agency” and “communion”, totaling twelve “other” predictions). For strong labels, only few-shot prompting with GPT-4o and DeepSeek-V3 correctly classified all test units, whereas zero-shot prompting for both models was still satisfactory with only minor prediction errors. For combined weak + strong labels, encompassing both weak and strong cues together, only DeepSeek-V3 was able to correctly classify all test units. For zero-shot prompting, the performance of GPT-4o and DeepSeek-V3 was also still satisfactory, again with only minor prediction errors.
Figure 4 Confusion matrices for the test units (n = 18). The y-axis corresponds to the true values, and the x-axis corresponds to the predicted values, with o = “other,” c = “communion,” and a = “agency.”
In contrast, SVM, XML-RoBERTa, and Llama 3-8B showed more overall wrong predictions for both strong and combined weak + strong labels, often failing to reach a passing rate of 50% of all units. Overall, we conclude that validation evidence for accurate and consistent classification was demonstrated exclusively by GPT-4o and DeepSeek-V3.
4.4 Exploring Sensitivity to Partisan Differences (External Validation)
Instead of focusing solely on convergence with human annotations, external validation can also assess whether a measure aligns with expected real-world relationships. We therefore conduct additional validation tests to examine the sensitivity of our measures to partisan variation in the data. Personality and ideology are known to intersect, and personality and value orientations show political correlations that are not always easily separable (Fatke Reference Fatke2017).Footnote 9
To test whether our measures are sensitive to partisan differences in our data, we examine partisan differences in public personality across two use cases: (1) aggregated personality scores of politicians across political parties in the German Bundestag for the full unlabeled corpus and (2) individual personality scores for the U.S. Presidential Debate between Donald Trump and Kamala Harris on September 10, 2024.Footnote 10 In both cases, we expect, in line with previous research, that liberal or left-leaning politicians generally display higher and more universal levels of communion compared to their conservative counterparts, who often tend to limit empathy to deliberatively chosen (in-)groups (Hasson et al. Reference Hasson, Tamir, Brahms, Cohrs and Halperin2018; McCue and Gopoian Reference McCue and Gopoian2000).
For clarity, we report results exclusively for the best-performing model that successfully passed all previous validation steps, namely DeepSeek-V3.
4.4.1 Left-Right Communion Distribution in the Bundestag
Figure 5 presents scatterplots showing the relationship between politicians’ shares of communion with mean politicians’ values aggregated at the party level (y-axis) and their parties’ expert-rated positions on the Chapel Hill Expert Survey (CHES, see Jolly et al. (Reference Jolly2022)) for two dimensions that make up the left–right continuum: the economic left–right (left panel) and the cultural libertarian–authoritarian (right panel) dimension. Generally, the highest shares of sentences containing communion cues can be found for the left party “LINKE” (19%), followed by the Greens (12%) and social–democrat SPD (10%), whereas politicians from more conservative parties like the CDU (8%) and right-wing AFD and liberal FDP (both around 6%) show overall lower communion values.
Figure 5 Correspondence of aggregated politicians’ shares for each party for “communion” (y-axis) and the respective CHES dimension (x-axis). Error bars indicate standard errors around each party mean.
As indicated by the regression lines, we see a clear and consistent negative relationship between the parties CHES score and the politicians’ share of communion. This association is particularly pronounced not only for the economic dimension (left panel), but also present in the cultural dimension (right panel). This pattern suggests that the signaling of communion is systematically more prevalent among politicians from economically and culturally left-leaning parties, in line with our prior assumption. These results support the validity of our agency–communion measures because they vividly demonstrate their correspondence with established ideological dimensions.Footnote 11 In Appendices M and N of the Supplementary Material, we further report descriptive analysis for the data at both the aggregated and individual levels, suggesting numerous avenues for inquiry that more in-depth analyses could explore.Footnote 12
4.4.2 U.S. Presidential Debate Corpus
Next, we measure the signaling of personality traits for the 2024 U.S. Presidential Debate between Donald Trump and Kamala Harris on September 10, 2024, to demonstrate that our measure is also sensitive to individual-level human variation. Transcripts for the debate were derived from the American Presidency Project (Woolley and Peters Reference Woolley and Peters1999), translated from English to German using the Deepl-API and subsequently processed using the same steps as in our previous analysis.
Our findings, visualized in Figure 6, confirm our expectations that Trump should signal more agency compared to Kamala Harris. On average, Trump’s sentences are notably shorter, leading to nearly twice as many utterances (n = 655) compared to Harris (n = 320). Notably, the overall share of personality cues is higher for Harris (approximately 23%) than for Trump (approximately 10%). However, examining the within-person distribution reveals a clear pattern: Trump displays a stronger relative emphasis on agency (63% agency vs. 37% communion), whereas Harris shows the reverse pattern, with a higher relative share of communion (59% vs. 41% agency). This means that the relative prominence of agency and communion is nearly reversed between the two candidates. These results show that our agency–communion measure was able to capture differences in the two candidates’ public personalities that were widely discussed in the media during the election campaign (Hyatt et al. Reference Hyatt, Campbell, Lynam and Miller2018).
Figure 6 Ratio of “agency” and “communion” for the presidential debate on September 10, 2024, between Donald Trump and Kamala Harris.
5 Summary and Conclusion
This study explored the feasibility of using computational text analysis to measure politicians’ publicly displayed personality traits from various textual sources (interviews, speeches, and social media posts) for two traits: agency and communion. Measuring personality traits from verbal behavior is highly challenging, not only because personality traits are inherently fuzzy constructs, making it difficult to identify valid linguistic cues, but also because their linguistic cues are often ambiguous and context dependent, making it difficult for computational models (as well as human raters) to correctly use the right cues to form accurate personality judgments. To address these challenges, we started from a validated personality framework to identify linguistic personality cues. We then compared multiple measurement strategies and models, varying dataset compositions (balanced vs. unbalanced) and personality cue conceptualizations (strong cues only vs. a combination of weak + strong cues). We conducted extensive validation using a systematic framework as outlined in Birkenmaier et al. (Reference Birkenmaier, Wagner and Lechner2024). Our findings suggest that prompt-based methods, particularly GPT-4o and DeepSeek-V3, allowed the most valid measurement of agency and communion from politicians’ public utterances.
Regarding the conceptualization of personality traits, models tasked to detect only strong cues for the traits in question performed better, as expected. This aligns with intuition, as detecting personality cues becomes more challenging when weak cues are included. The fact that personality traits are fuzzy and concepts with ambiguous verbal cues was reflected in the moderate agreement among human coders (between .70 and .78 Krippendorff’s alpha) and the fact that none of the computational models achieved a macro-F1 score above .80 (see Table 1).
Nevertheless, the results suggest that the best available computational methods can indeed capture meaningful personality-related cues in text. These findings provide the foundation for more substantive research in the field of political psychology. For instance, this could include the investigation of changes in politicians’ expressed personality across environmental factors that can indicate how political figures adapt their public personas in response to evolving circumstances. Potential explanatory factors include crisis events, such as economic downturns or global conflicts, which may lead politicians to fulfil the demand for dominant political leaders (Petersen and Laustsen Reference Petersen and Laustsen2020; Sprong et al. Reference Sprong2019). Additionally, individual role transitions—such as taking over the position of party secretary or becoming a lead candidate—could influence the way personality traits are displayed.
While this study demonstrates the potential of computational methods for inferring politicians’ public personality from verbal cues, several limitations remain. The analysis focused on only two personality traits—agency and communion—which should be expanded to other relevant traits. Next, keep in mind that personality cues can be expressed in multiple ways—such as paraverbal and nonverbal features like rhetoric, gestures, or interactions. Thus, future research should complement our approach by using multimodal approaches, incorporating video, audio, and behavioral data (Achmann-Denkler et al. Reference Achmann-Denkler, Fehle, Haim and Wolff2024). Connected to this, our study operated at the sentence level; extending it to paragraph-level or token-level analysis would provide deeper insights into personality expression in political speech. Finally, although our validation did not reveal systematic bias, we acknowledge that large language models can generally reflect biases from their pretraining data. In the case of DeepSeek-V3, recent reports have pointed to potential pro-Chinese bias stemming from data selection or censorship (Huang et al. Reference Huang, Lin, Imbot, Fu and Tu2025). While we found no indication of such bias in our study, we recommend additional validation in studies involving more politically sensitive content or contexts.
In conclusion, our study demonstrates that while measuring personality trait expressions from text remains a daunting task, it is not an impossible one. By combining robust validation strategies with recent advances in language models, we demonstrated that even abstract traits like agency and communion leave detectable traces in political discourse if the right operationalizations are chosen. This opens the door to a new generation of research: one where personality is not just surveyed or inferred, but measured at scale, over time, and in context—from speeches to debates, from parliaments to press conferences. If personality is indeed what politicians project to the public, then computational methods may soon become essential tools in decoding the psychological choreography of power.
Acknowledgments
We thank all those who contributed to this manuscript, especially the participants of the 6th ANNUAL COMPTEXT Conference 2024 in Amsterdam, as well as Brandon Stewart, Co-Editor-in-Chief, and the four outstanding anonymous reviewers, whose efforts went above and beyond to improve the manuscript.
Funding Statement
This research received no particular funding.
Data Availability Statement
Full replication materials for this article are available at: Birkenmaier, Lukas, and Lechner, Clemens. Reference Birkenmaier and Lechner2025, “Replication Data for: Measuring Politicians’ Public Personality Traits using Computational Text Analysis: A Multi-Method Feasibility Study for Agency and Communion”, https://doi.org/10.7910/DVN/MSY9L1, Harvard Dataverse.
Author Contributions
Both authors contributed equally to this work.
Competing Interests
The authors declare none.
Ethical Standards
No concerns regarding ethical standards are reported.
Supplementary material
To view supplementary material for this article, please visit http://doi.org/10.1017/pan.2025.10025.
Open access
