2.1. Connecting democratic resilience to framing analysis

What are the narrative aspects of democratic resilience? Here, we integrate widely used definitions of both democratic resilience and “framing” to narrow in on the focus of our study. First, according to Holloway and Manwaring’s definition of democratic resilience, a population riven by discordant narratives is a threat to democracy itself. For them, democratic resilience is that quality of a democratic system that can stand despite the narrowest intersection of agreement on the issues that matter. Namely, “democratic resilience” is (p.76),

As such, and integrating Snow and Benford’s decomposition of framing in social movements, a resilient democracy is one which can hold on to its foundational apparatus, even when the society at large is captured by highly opposing diagnoses, prognoses, or motivations regarding the issues of the day. Holloway and Manwaring (Reference Holloway and Manwaring2023)’s wide-lens review of literature on democratic resilience finds that (Table 2, p.72) “diversity (of groups, individuals),” “learning,” and “community participation and inclusion” are amongst the common characteristics of resilient systems. So democratic resilience is about finding ways for diverse narratives to co-exist—that is to challenge the status-quo, and/or seek inclusion—whilst at the same time, being wary of narratives which foment excessive polarisation, distrust of electoral institutions and representation, and undermine social cohesion. The most resilient democracy may be able to withstand the worst of these narratives, but recent international experience suggests that democratic systems can be surprisingly susceptible to hitherto marginal narratives.

Second, a common approach to classifying narratives for quantitative social science, is to draw on theories of issue-framing (or just “framing”). Here, one or more positions on any given topic or issue can be identified by examining how a communicator crafts their message on the topic, that is how they “frame” the topic. Entman (Reference Entman1993) defines framing thus (p.52) (emphasis added),

Key in Entman’s definition is selection and saliency, to serve the objective of promotion of the speaker’s particular messaging on the topic. As such, issue framing can be seen as a specific aspect of narrative. Where narrative involves characters, events, and often a temporal component, framing is concerned with the specific treatment of an issue, event, or topic by the speaker who wishes to convince the receiver of the same. Chong and Druckman (Reference Chong and Druckman2007) makes this connection in their more recent theorising (p.104) (emphasis added),

Critically then, issue-framing applies equally to the speaker and receiver. The speaker seeks to promote a particular framing of an issue (ala Entman), with the objective of bringing the receiver to the same perspective (ala Chong & Druckman).

But framing is not merely an individual consideration, in democracies, individuals freely join with others to form groups, groups join again to form movements. And here, at these higher orders of organisation, social theorists see that framing applies equally well. Indeed, there is an important interplay between the framing of the group or movement and the individuals who compose it. Snow and Benford (Reference Snow and Benford1988) write (p.198),

So together, we find that narratives matter to democratic resilience (at least) because of the power of framing in the hands of political leaders to coerce reality to their messages, because of the way that framing can form and locate political identities, and because framing can mobilise the masses. For these reasons, we should not have been surprised by our earlier observation that the majority of threats to democracy identified by the Australian Strengthening Democracy Taskforce have a narrative component.

2.2. So which frames matter to democratic resilience?

The literature offers up a number of recognisable narratives that any democratic society may wish to be wary of. We briefly consider three.

First, the “Us vs. Them’” narrative is the most transparently polarising, reducing complex, multi-dimensional issues down to a single dimension, and at its worst, encouraging a kind of “purity test” of belief for those who adhere to one “side” of the political landscape or the other (Hetherington and Rudolph, Reference Hetherington and Rudolph2015; McCoy et al., Reference McCoy, Rahman and Somer2018; Iyengar et al., Reference Iyengar, Lelkes, Levendusky, Malhotra and Westwood2019; McCoy and Somer, Reference McCoy and Somer2019). Despite wide variation in how a specific country becomes extremely polarised, authors contend that the subsequent undermining of democracy follows a very common path: political opponents dis-engage from meaningful communication and compromise; government reform agendas are gridlocked; each side dismantles the other’s legislative agenda immediately on gaining power; and in the end-state, democratic transfer of power falls under restrictions of civil liberties and sees the military’s entry to replace government.

Relatedly, narratives of nationalism can also work to undermine a plural, inclusive democratic society. As argued by Mylonas and Tudor (Reference Mylonas and Tudor2021), we should distinguish between nationalism which draws on ideals of freedom and justice (e.g. the American and French Revolutions), and (p.111), “exclusionary nationalism (also called ethnic or essentialist nationalism).” Exclusionary nationalism shares rhetorical approaches with the “us versus them” narrative in its reductionism, that is driving a hierarchical narrative along racial, ethnic or religious grounds, “othering” some group or groups in society to achieve some idealised, patriotic objective.

Third, as Algan et al. (Reference Algan, Guriev, Papaioannou and Passari2017) ominously begin, “a spectre is haunting Europe and the West—the spectre of populism.” Populism is perhaps the narrative which is most directly opposed to the democratic project, for its proponents set themselves against the very institutions of democracy itself—established political parties, established institutions of state, even the judiciary, are all said to be held captive by “elites,” set against the best interests of the common people. Here, former president Donald Trump’s dark framing of modern America, promising to “drain the swamp” (that is of elites), and “make America great again,” spoke to the felt experiences of the American people and dove-tailed with right-wing programming across print, radio, TV and online (Polletta and Callahan, Reference Polletta and Callahan2017; Norris and Inglehart, Reference Norris and Inglehart2019). The message was particularly appealing to those who felt like “outsiders” in their own country due to social and economic marginalisation (Gidron and Hall, Reference Gidron and Hall2020).

But how prevalent are these narratives in a modern democracy? Are narratives of populism or exclusionary nationalism already making themselves heard in the United Kingdom, Australia, or the United States? Has there been a change over time? Where? Is this only for the online “echo-chambers” or do we see growing evidence of these narratives in our public discourses arising in Parliament, print or on the air-waves?

The answer to all these questions requires empirical tools.

2.3. Aside: social listening and surveillance

Before moving to empirical methods, it is worth spending a moment to distinguish present considerations from “surveillance” orientated methods, or what is sometimes referred to as “surreptitious social listening.” Listening—the active or passive act of receiving/collecting information from people in a given context—can take a variety of forms, best considered as sitting on a spectrum (Martino, Reference Martino2020). According to such typologies, tracking of public narratives related to social cohesion and democratic resilience, sits in the “active listening” category, and, according to Martino seeks to enhance democratic and foreign policy objectives by promoting trust and understanding. In a word, the government or actor who actively listens to citizens and counterparts, including those who hold opposing views, demonstrates a long-term commitment to dialogue, relation-building, and engagement. Importantly, such active listening, or “public discourse tracking”as we consider here, is far away from surreptitious listening, or what is otherwise known as surveillance or spying. Here, there are spaces for legitimate uses of such methods in democracies, but such uses must carefully balance personal rights and freedoms with policing and counter-terrorism objectives (Moore, Reference Moore2011; Hagen and Lysne, Reference Hagen and Lysne2016). Importantly, it should be acknowledged that all surveillance carries harm (Richards, Reference Richards2013) either because of a chilling effect on civil liberties (Penney, Reference Penney2016), or by introducing unhealthy power dynamics due to information asymmetries. Nevertheless, tracking public narratives related to democracy, where it seeks to understand, characterise, build trust, and engage, aims to support democratic values, not undermine them, and should be kept distinct from states or actors who seek to infiltrate the private sphere to advance coercive, controlling, or chilling purposes.

4.1. Risks associated with using transformers

Using LLMs for framing analysis at scale is not without concerns. Generative LLMs present outstanding opportunities for quantified approaches to tracking narratives at scale, but their effective use requires a robust understanding of a suite of risks (Bommasani et al., Reference Bommasani, Hudson, Adeli, Altman, Arora, von Arx, Bernstein, Bohg, Bosselut, Brunskill, Brynjolfsson, Buch, Card, Castellon, Chatterji, Chen, Creel, Davis, Demszky, Donahue, Doumbouya, Durmus, Ermon, Etchemendy, Ethayarajh, Fei-Fei, Finn, Gale, Gillespie, Goel, Goodman, Grossman, Guha, Hashimoto, Henderson, Hewitt, Ho, Hong, Hsu, Huang, Icard, Jain, Jurafsky, Kalluri, Karamcheti, Keeling, Khani, Khattab, Kohd, Krass, Krishna, Kuditipudi, Kumar, Ladhak, Lee, Lee, Leskovec, Levent, Li, Li, Ma, Malik, Manning, Mirchandani, Mitchell, Munyikwa, Nair, Narayan, Narayanan, Newman, Nie, Niebles, Nilforoshan, Nyarko, Ogut, Orr, Papadimitriou, Park, Piech, Portelance, Potts, Raghunathan, Reich, Ren, Rong, Roohani, Ruiz, Ryan, Ré, Sadigh, Sagawa, Santhanam, Shih, Srinivasan, Tamkin, Taori, Thomas, Tramèr, Wang and Wang2021). Attention must be paid to the provenance of training data, bias in generated text, processing cost, and open or closed access, given that many of the most performant models are closed-source, run by large platforms. As mentioned, the training of LLMs (next word prediction) does not guarantee correctness or completeness in responses obtained, and even the most frontier LLMs can still be tripped up by seemingly trivial completion tasks (For example, if one asks the most performant language model to date, Anthropic’s Claude 3.5 Sonnet, “How many i’s in Kit-Kat?’, it answers, “The correct answer is that there are zero i’s in Kit Kat.’ This behaviour is common for all the major models alike.). In the computational social sciences recent reviews of LLM opportunities and risks catalogue these issues in more detail (Thapa et al., Reference Thapa, Shiwakoti, Shah, Adhikari, Veeramani, Nasim and Naseem2025), with some authors arguing that the impact of LLMs will go beyond efficiency and opportunity to start re-shaping the scientific method itself (Fecher et al., Reference Fecher, Hebing, Laufer, Pohle and Sofsky2023), and exacerbate existing resource and access inequalities in the political economy of ideas (Luitse and Denkena, Reference Luitse and Denkena2021). On the former, work has already been done to create entire ecosystems of LLMs (Agentic-AI systems) which operate as virtual, automated scientific laboratories reading each other’s (AI) lab’s work, and improving upon it (Schmidgall and Moor, Reference Schmidgall and Moor2025). For now, these studies are largely confined to computer science domains, where algorithmic optimisation is well handled by such systems; nonetheless, the question of how far science is willing to bring LLMs into the scientific method is open and dynamic.

Direct assessments of political bias, perhaps most relevant to the present study, in LLMs are important and require constant updating due to the rapid developments in model releases and capabilities. For instance, Choudhary (Reference Choudhary2024) put four recent, major commercial “models” (“ChatGPT-4, Perplexity, Google Gemini, and Claude”—unfortunately, we do not learn which exact model variant in the paper) through major political orientation questionnaires such as the Pew Research Centre’s Political Typology Quiz, PoliticalCompass.org, and ISideWith political party quiz, and find that all models lean left of centre, with “ChatGPT-4” and “Google Gemini” aligning with “establishment liberals” (three categories to the left, of four), whilst “Claude” and “Perplexity” aligned just left of centre, as “outsider left” (one category to the left, of four). Political Compass results were similar. These results appear robust to model choice and language (Pit et al., Reference Pit, Ma, Conway, Chen, Bailey, Pit, Keo, Diep and Jiang2024; Rettenberger et al., Reference Rettenberger, Reischl and Schutera2025). It is important to note that such outcomes are more likely when models are used in a “zero-shot” manner, that is, when they are asked, without examples or further definitions, to answer a direct question (see a demonstration of prompt variation below Section 5). Further, we note that human annotation in politically relevant domains is itself subject to annotator bias (Plisiecki et al., Reference Plisiecki, Lenartowicz, Flakus and Pokropek2025), as such, it is unlikely that LLMs (which rely on human guidance) will obtain entirely “bias-free” behaviours. For now, these considerations, like any research involving judgements (human or machine), require thoughtful design and interpretation decisions with replication across models/labs/contexts, as ever, the most likely protection against false-hood. In this light, AI documentation literacy is key for success in this direction (Smith et al., Reference Smith, Bello, Bialic-Murphy, Clark, Delavaux, de Lauriere, van den Hoogen, Lauber, Ma, Maynard, Mirman, Lidong, Rebindaine, Reek, Werden, Wu, Yang, Zhao, Zohner and Crowther2024).

5.1. Identifying populism with transformers

Populism is a common object of study for the political analysis of text, and as noted earlier, is one of the key pernicious narratives related to democratic resilience. That said, defining the populism narrative is the first methodological challenge. Early studies typically looked to Mudde (Reference Mudde2004)’s succinct rendering,

This definition focuses on the core idea of a binary struggle between (a monolithic) “the people” and “the elite.” However, more recent authors contend that populism is a latent construct that can only be truly identified via a multi-dimensional tool. Meijers and Zaslove (Reference Meijers and Zaslove2021) make this argument and develop a five-dimensional survey tool for validation with an expert judgement panel comprising 294 respondents “with expertise in party politics” (p.382) across 28 countries (Included countries: Austria, Belgium, Bulgaria, Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Ireland, Italy, Lithuania, Malta, The Netherlands, Norway, Poland, Portugal, Romania, Slovakia, Slovenia, Spain, Sweden, Switzerland, United Kingdom.). Using exploratory factor analysis (EFA) and item response theory (IRT), they conclude that their five-dimensions of populism are a valid operationalisation of the latent concept. The five dimensions are given as (p. 384),

1. 1. Manichean worldview: “Politics is a moral struggle between good and bad”

2. 2. Indivisible people: “The ordinary people to be indivisible (i.e., the people are seen as homogenous)”

3. 3. General will: “The ordinary people’s interests to be singular (i.e., a “general will’)”

4. 4. People-centrism: “Sovereignty should lie exclusively with the ordinary people (i.e., the ordinary people, not the elites, should have the final say in politics)”

5. 5. Anti-elitism: “Anti-elite dispositions”

Given the more robust validation approach of Meijers and Zaslove (Reference Meijers and Zaslove2021), in what follows, we shall adopt this definition in our second, expanded, prompt strategy.

Approach

To provide a concise demonstration of how generative transformers perform, we shall conduct a small text ( $ n=5 $ ) corpus evaluation, leveraging variation across commercial platforms (Anthropic, OpenAI, Google), model flavours (chat models vs. so-called “reasoning” models) from these providers, and prompt design. On the latter, we will demonstrate two prompting strategies, starting with a “zero-shot” style prompt, which leans on the particular model’s “world-knowledge understanding” (from training and alignment) of “populist narrative” compared to an extended prompt, in which we explicitly define the concept, and require the model to only label a text as exhibiting a populist narrative if it fulfils all five dimensions. Both prompts variants are shown in Figure 3 (Note: prior to the prompt, a general system prompt was provided the model and was constant across all calls, “You are a helpful, knowledgeable, expert assistant. You are proficient in many tasks, such as coding, math, science, humanities, and more.’). The extended definition flows from the “ideational approach” to defining populism as a latent concept taken from Meijers and Zaslove (Reference Meijers and Zaslove2021) introduced above. For the purposes of the demonstration, the extended prompt treatment represents a more controlled setting (since we explicitly define the labelling policy) whilst simultaneously raising the bar for the positive labelling of the text as populist since the prompt requires the text to “fulfil all five dimensions.”

Figure 3.

Two prompt variants were employed in the populism labelling demonstration. Both prompts contain the same task: to respond with a label from a five-item ordinal set (and brief justification), as to whether the TEXT exhibits a populist narrative (blue text in both prompts). The extended prompt (bottom) adds a five-dimensional definition (Meijers and Zaslove, Reference Meijers and Zaslove2021) to fix ideas and effectively raise the threshold for positive identification.

Corpus

The small text corpus was constructed to contain a core of three texts representing populist politicians or populist political parties (Trump, Vlaams Belang, Left Front), plus two strongly pro-democracy politicians (Albanese, Obama). Of the two pro-democracy politicians, one text (Obama) gives a full throated affirmation of democratic institutions, whilst the other text (Albanese) represents a tricky case, since the (then) Australian Opposition Labor Leader leader accuses the incumbent government with quasi-elitist language (“[they have] treated. public money as a political slush fund”), calling for a vote to oust them, yet steering well clear of explicit populist rhetoric. We include the text as a mid-point between the explicitly populist texts of Trump, Vlaams Belang and the Left Front on the one hand, and the explicitly anti-populist text of Obama on the other. The texts have the following notes:

• • TRUMP—Excerpt from US Presidential inauguration speech, 20 Jan 2017. In one sentence, perhaps the “textbook” populist position, juxtaposing the “small group” in the “nation’s Capital” (i.e. the elites) profiting whilst “the people” (homogenous, united) “bear the cost.” (See: https://trumpwhitehouse.archives.gov/briefings-statements/the-inaugural-address/, April 2025).

  • – Hypothesis: We expect direct (zero-shot) calls to identify the text as populist, but it is an open question whether models will find this single sentence ticks all five dimensions of the extended prompt requirement.

• • VLAAMS BELANG (VB)—Founded in 2004, Vlaams Belang is a Flemish nationalist party in Belgium. We select here an excerpt from the VB Membership Magazine, Oct 2008. Of interest for the demonstration, this text arises from Pauwels (Reference Pauwels2011)’s study of Belgian parties, wherein they find that not only is VB “often labelled as populist by both commentators and scholars” (p.98) but according to the study results, VB is “by far the most populist of all the examined parties” (p.101).

  • – Hypothesis: Strong anti-elite messaging should satisfy zero-shot prompt, satisfy three of five dimensions of the extended prompt, but is not explicit about the indivisible people and their will.

• • LEFT FRONT (“Front de Gauche”)—“Front de gauche” (FG) was founded in 2009 as an alliance between the French Communist Party and the Left Party, with the excerpt taken from the party manifesto, Jun 2012. The manifesto was extracted from the Manifesto Project (See (requires login): https://manifesto-project.wzb.eu//down/originals/31021_2012.pdf, April 2025) and has been translated from the original French to English using Anthropic’s Claude 3.7 Sonnet model (Specifically: claude-3-7-sonnet-20,250,219, prompt: “Translate to English: {text}’. It is worth noting that each model tested in this demonstration could have handled undertaking the task in the original language (French), however, to reduce contamination due to their respective strengths at translation, pre-translation to each model’s main training language (EN) was undertaken.). The Left Front represents one of the most prominent left-leaning populist movements today, scoring−47.925 on the rile score (extreme left) per the Manifest Project’s own data. As such, this text provides a useful test of populist language from the left, as counter point to the VB text, from the right.

  • – Hypothesis: This extended text encapsulates well the multi-dimensional populism concept, including strong phrasing around the “the people,” their will, their return to power through a “citizen revolution” of the incumbent “greed of a few.” Likely to pass both the zero-shot and extended prompt version.

• • ALBANESE—Excerpt from campaign speech delivered 1st May, 2022 in Perth, Australia (See: https://electionspeeches.moadoph.gov.au/speeches/2022-anthony-albanese, April 2025). Anthony Albanese was at that time the leader of the federal opposition Labor Party of Australia. The text was selected as a potentially challenging case for automated methods since in isolation, it has some flavour of anti-elite sentiment—accusing the incumbent leadership as shirking responsibility, and mis-using public money for their own interests—but a human expert would be unlikely to strongly identify this text as populist since there is little contrast of “the people” with the “elites” and his call for an anti-corruption commission supports the rule of law and democratic institutions in general.

  • – Hypothesis: Mixed outcomes. Zero-shot outputs could fall either side of populist or not, but unlikely strongly so. However, it should fail the extended, multi-dimensional prompt, with models unable to find convincing evidence across all aspects of populism required.

• • OBAMA—Excerpt from US Presidential inauguration speech, 21 Jan 2013 (See: https://obamawhitehouse.archives.gov/the-press-office/2013/01/21/inaugural-address-president-barack-obama, April 2025). This text was delivered by Obama near the top of his speech. It affirms the founding democratic principles of the United States Constitution strongly, it uses inclusive language for the peoples of the country and notes the protections provided each of them under the Constitution and the rule of law.

  • – Hypothesis: Neither the zero-shot nor the extended prompts should return any positive identification of populism. The text is anti-populist from start to finish.

Models

Models were included to present a spectrum of capabilities and varieties. From each provider, a strong “reasoning” model was included, that is one which undertakes chain-of-thought reasoning by design (Wei et al., Reference Wei, Wang, Schuurmans, Bosma, Ichter, Xia, Quoc, Le and Zhou2022) with reasoning effort or length set to “high,” a small/fast model, that is one which is intended for simpler tasks emphasising low latency and low cost, and an intermediate model, typically the strong model without reasoning, or an equivalent recommended model from the provider (Exact model details (reading left to right of Fig. 4): Anthropic – “claude-3-7-sonnet-20,250,219-high’ (thinking enabled, budget tokens 32,768, cut-off Oct 2024), “claude-3-7-sonnet-20250219’ (LLM mode, no thinking enabled, cut-off Oct 2024), “claude-3-5-sonnet-20241022’ (cut-off Apr 2024); OpenAI – “o3-high’ (“o3–2025-04-16’, reasoning effort: “high’, cut-off 1 Jun 2024), “gpt-4.1’ (“gpt-4.1-2025-04-14’, cut-off 1 Jun 2024), “gpt-4o’ (“gpt-4o-2024-08-06’, cut-off 1 Oct 2023); Google/Gemini—“gemini-2.5-flash-preview-04-17’ (thinking enabled, budget tokens 24 k, cutoff Jan 2025), “gemini-2.5-flash-preview-04-17’ (LLM mode, no thinking, cutoff Jan 2025), “gemini-2.0-flash’ (cutoff June 2024).). All calls were made via API with default API parameter settings. It is important to note that models are trained only up to a certain cut-off date. These dates ranged from the earliest in Oct 2023 (OpenAI’s gpt-4o) to the most recent in Jan 2025 (Google’s Gemini 2.5). As such, all texts were within the training context window of each model, bringing in the potential (or advantage), that the model may bring prior training world knowledge to bear on the task at hand (e.g. the model may recognise an excerpt is from a broadly known populist party or figure and so, be biased by the “reputation” of the source in its adjudication of the given text). Whilst the researcher could attempt to redact identifying characteristics from the text, the reality is that today’s models will likely recognise the surrounding text anyway (see below), hence, this consideration is more of a source of potential bias to be assessed through validation against ground-truth rather than something that can be completely avoided.

Figure 4.

Results for the labelling of populism in the small political texts corpus. Each column indicates the model response, colour-coded according to the Key (bottom): more red colours indicate a finding of populism, more green colours indicate a finding against populism. Results are grouped according to model family (Anthropic, OpenAI, Google) with stronger to weaker variants of each family presented left to right within a family, and prompt in use (Zero-shot prompt: first 9 cols; Extended prompt: last 9 cols). The final two cols give a majority rating under each prompt.

Findings

Figure 4 presents results across the 90 model calls, noting that these are strictly for demonstration purposes only; a full study would incorporate a human validation and sensitivity strategy. Results are presented to facilitate ease of comparison: across prompt strategies (zero-shot, then extended prompt: main panels), across model families (groupings within panels), and across model strengths within each family (stronger to weaker models are presented left to right within each family). We also provide a majority outcome for the zero-shot and extended prompt variants, weighting models equally.

From this exercise, we draw the following insights:

1. 1. Labelling with transformers is very flexible. It is hard not to overstress this point, though many practitioners may take for granted the new “chat-based” world of AI, the ability to take the full text, of varying lengths, and even languages (all models considered in this demonstration are multi-lingual), and obtain valid labels in fractions of a second demonstrates much of the remarkable opportunity these methods bring to the table.

2. 2. Labelling with transformers is very cost-effective. Model calls were of the order of USD 0.01–0.02 for the most expensive (strongest) models (e.g. Claude 3.7 Sonnet, OpenAI’s o3(high)) and fractions of a cent for the weaker models. Together, doing an ensemble call over the nine models cost on the range USD 0.05 to USD 0.10. Human labelling would take around 30s of reading time (e.g. Left Front, Obama), and perhaps 30s of judgement time, in all around 60s, at a typical data labelling rate of USD 25/hour, or USD 0.40/minute, we have an approx. 4 to 8x cost saving. However, this is a lower bound, given that one would rarely call three “reasoning” models in practice, so the real cost saving is likely on the order of 12 to 24x (Noting that this simple analysis equally does not price development time for the model pipelines, nor equivalent time spent setting up human labellers.).

3. 3. Zero-shot prompting comes with model risks. The most striking pattern across the two panels is the uniformity of findings under the zero-shot scenario. Here, the three populist texts (Trump, VB, Left Front) were universally found to be populist by each of the nine models, Albanese’s tricky case leaned populist, whilst Obama’s anti-populist case leaned strongly anti-populist. Whilst on face value this seems reasonable, the Albanese results are troubling given our prior hypothesis that Albanese’s text should not be confused as populist despite the inherent government critique. Without defining terms, the models must leverage their own (hidden, “black-box”) understanding of the term “populist,” which poses a control risk to any study design.

4. 4. Extended prompting is more robust, but results need careful interpretation. Extended prompting has confirmed Obama’s anti-populism, and moved the dial towards the same for Albanese, especially so in five of the six stronger model calls. However, we also see that positive populism labels are harder to obtain in general, with only the Left Front universally receiving the populist tag. Aligning with our hypotheses, Trump’s short, sharp text, and VB’s less explicitly five-dimensional excerpt both receive mixed reviews. We return to this point below.

5. 5. Ensembling models can recover signal from noise. The pattern of the final columns in the figure, where we take the majority view from across the nine models, delivers an almost exact imprint of our grounded pre-trial expectations for each text. Ensembling is well known in machine learning to be a helpful technique for distinguishing signal from noise, and the same appears true from our demonstration in the world of transformer-based labelling.

We close this exercise with one final observation around transformer behaviour. Recall that our two prompts (Figure 3) both asked the model to provide a label and a brief justification for the model response. These justifications are instructive but need to be taken with a grain of salt (Recent work by Anthropic under the heading of model “biology’ has been able to show strong evidence that model explanations, and even internal chains of thought, do not always align with model outputs. See: “On the Biology of a Large Language Model’, published at https://transformer-circuits.pub/2025/attribution-graphs/biology.html#dives-cot). We focus on the anomalous cases, specifically the differential ways that models handled the VB text under the extended prompt (where most variation in outcome was observed). The three strongest models, across each provider, gave the following:

• • Anthropic:: Claude 3.7 Sonnet (High): “weak yes” (“Only demonstrates anti-elitism, missing other required populist dimensions.”)

• • OpenAI:: o3 (High): “strong no” (“Only anti-elite language; no unified people or general will.”)

• • Google:: Gemini 2.5 Flash (High): “strong no” (“Lacks explicit indivisible people, general will, and people-centrism.”)

As expected, we can see that with the extended definition, each model struggled to endorse populism due to the requirement that all five dimensions should be present. It would be wrong, therefore, to conclude that the stronger models have made a “mistake” here; instead, it is crucial to understand that as models have grown in capability, they have also grown in their ability to more closely follow the task given to them. We can contrast these explanations with those of the weaker models. Google’s Gemini 2.0 Flash commented, “Anti-establishment, anti-elite sentiment is very strong” on the way to a “strong yes” to populism. Whilst OpenAI’s GPT-4o labelled “weak yes,” saying, “Anti-elitism and people-centrism are evident,” focussing on what was evident, not what was missing. A deeper exploration of model variability and human validation along these lines would be a fruitful line of further inquiry for this method.

5.2. Tracking narratives of immigration with transformers

In our second demonstration, we use the novel paired-completion approach detailed in Angus and O’Neill, Reference Angus and O’Neill2024 to analyse texts related to immigration. Our aim here is to provide a demonstration of what can be done with paired-completion by carrying out a small (non-causal) longitudinal study on Australian Parliamentary Hansard during a particularly tumultuous time for the issue of asylum seeker policy in Australia. While robustness, model-choice, and validation are important considerations, these have been addressed in prior work (Angus and O’Neill (Reference Angus and O’Neill2024)). The present study’s purpose is to demonstrate the feasibility and policy relevance of transformer-based narrative tracking, rather than to conduct a full empirical evaluation. To this end, we provide and critically discuss below several examples of extant speeches and their scores via the method. Any full, stand-alone, empirical study would be expected to conduct a grounded validation study with human-expert evaluation.

Approach

We follow the method of paired-completion as provided in the reference. Paired-completion requires two key inputs: 1) a set of opposing “seed” statements which define the opposing framings on the topic under study; and 2) a corpus of texts to apply the method to. On the first, OpenAI’s GPT-4 (gpt-4-turbo) model was used to generate five statements to represent two perspectives on asylum seekers in the Australian context (Prompt 1: “Provide two perspectives on asylum seekers in the Australian context. Name each perspective. For each perspective, provide five sentences that together represent that perspective.’, followed by Prompt 2: “For each perspective, turn the sentences into example statements that represent these views.’). This produced the following seed statements for use as priming texts in the paired-completion method:

Perspective 1: Humanitarian Approach:

1. 1. Human Rights Focus: “We must remember that asylum seekers are escaping dire situations where their basic human rights are often violated; it’s our duty to offer them sanctuary.”

2. 2. International Obligations: “As a signatory to the UN Refugee Convention, Australia has a legal and ethical obligation to protect those who arrive on our shores seeking refuge from persecution.”

3. 3. Moral Duty: “It is our moral obligation as a prosperous and stable nation to extend a helping hand to those in need, showing compassion beyond our borders.”

4. 4. Community Benefits: “Asylum seekers bring a wealth of diverse cultural perspectives and skills that can greatly benefit our society and economy, particularly in sectors with labor shortages.”

5. 5. Legal and Fair Processing: “We must ensure our asylum processing system is fair and transparent, adhering to international laws and respecting the dignity of each individual.”

Perspective 2: Border Security Approach:

1. 1. National Security: “It is crucial to maintain stringent border controls to prevent the entry of potential threats to our national security, ensuring the safety of all Australians.”

2. 2. Economic Concerns: “The economic burden of supporting a large number of asylum seekers could strain our public services and welfare systems, impacting the financial stability of our country.”

3. 3. Legal Immigration Channels: “We need to uphold the integrity of our legal immigration system; allowing people to bypass it by arriving irregularly sets a precedent that could undermine the entire framework.”

4. 4. Sovereignty and Control: “Australia must have the sovereign right to control its borders and decide who can enter and stay in our country, to preserve the order and effectiveness of our immigration policies.”

5. 5. Deterring Smuggling Operations: “Implementing strict asylum policies can act as a deterrent to human trafficking and smuggling operations, which often exploit vulnerable individuals seeking refuge.”

To proceed, the method requires interaction with a chat-completion (generative) LLM from which completion log-probabilities are available (That is Typically via the “echo’ switch.). Since many commercial models do not support this feature, the open-source model “llama-2” from Meta was employed (Touvron et al., Reference Touvron, Martin, Stone, Albert, Almahairi, Babaei, Bashlykov, Batra, Bhargava, Bhosale, Bikel, Blecher, Ferrer, Chen, Cucurull, Esiobu, Fernandes, Fu, Fu, Fuller, Gao, Goswami, Goyal, Hartshorn, Hosseini, Hou, Inan, Kardas, Kerkez, Khabsa, Kloumann, Korenev, Koura, Lachaux, Lavril, Lee, Liskovich, Lu, Mao, Martinet, Mihaylov, Mishra, Molybog, Nie, Poulton, Reizenstein, Rungta, Saladi, Schelten, Silva, Smith, Subramanian, Tan, Tang, Taylor, Williams, Kuan, Xu, Yan, Zarov, Zhang, Fan, Kambadur, Narang, Rodriguez, Stojnic, Edunov and Scialom2023), using the $ k=2 $ variant of the paired-completion approach (Specifically, the 70B parameter variant, having a context length of 4096 chars. Meta does not give a precise endpoint for the training window, but notes training on “publicly available data”.). This means providing the model with two randomly selected seed sentences on a given perspective, concatenated together, as a primer, and then appending the text under study “as if” the model had generated the whole sequence itself (primer(s) plus completion). One can read off the log-probabilities (i.e. the likelihoods) of the completion text, given the priming text(s) as explained earlier. The “diff” metric is then computed for each text, that is the difference in likelihoods between the text being a completion of perspective 1 versus a completion of perspective 2. Each text thus receives a textual alignment score on the humanitarian—border-security dimension, with higher (positive) scores associating most strongly with the humanitarian framing, and lower (negative) scores associating with the border-security framing.

Corpus

To illustrate the method, we emphasise the longitudinal dimension whilst reducing complexity by focussing only on 7 Prime Ministers along with four minor party or independent prominent speakers on the topic over this period. In alphabetical order the selected speakers were:

• • Andrew Wilkie, Independent (Prominent speaker);

• • Anthony Norman Albanese, Australian Labor Party (PM 2022-);

• • John Winston Howard, Liberal Party of Australia (PM 1996–2007);

• • Julia Eileen Gillard, Australian Labor Party (PM 2010–2013);

• • Kevin Michael Rudd, Australian Labor Party (PM 2007–2010, 2013);

• • Malcolm Bligh Turnbull, Liberal Party of Australia (PM 2015–2018);

• • Nick McKim, Australian Greens (Prominent speaker);

• • Richard Di Natale, Australian Greens (Prominent speaker);

• • Sarah Hanson-Young, Australian Greens (Prominent speaker);

• • Scott John Morrison, Liberal Party of Australia (PM 2018–2022); and

• • Tony John Abbott, Liberal Party of Australia (PM 2013–2015)

In terms of hypotheses, the debate in Australia during the boat arrival period was highly polarised between the coalition (liberal/national) “stop the boats” policy framing and the greens and independents who articulated a strongly humanitarian critique of the coalition’s approach. Labor notoriously tried to pursue a more humanitarian approach under Rudd/Gillard, seeking to accommodate and/or process asylum seekers in a mix of onshore and (eventually wholly) offshore detention centres, which in the eyes of the Greens and independents created more humanitarian disasters. Ultimately, with the “success” of the strengthened (military run) turn-back policy of the coalition, Labor in recent years has been accused of effectively adopting the identical policy formulation they previously rejected in opposition. Taken together, we might expect to see:

• • Hypothesis 1: Polarisation between the Greens/Independents and the Liberal party on this topic;

• • Hypothesis 2: Labor’s policy position shifting from somewhat humanitarian to strongly border security based in recent years;

Hansard speeches for the Australian House of Representatives (Australia’s lower, legislative, house in its bicameral system) were obtained from the Open Australia project API (See: http://data.openaustralia.org.au/) covering Jan 2006 to Mar 2023 (At the time of ingestion, these were the extent of data available through the project, as the engineering team were having trouble fixing their methods. See comments below on data availability), a period of particular interest in Australian politics due to the outsized increase in asylum seeker boat arrivals (typically from waters North of Australia) over this period. By illustration, just one boat arrived in 2003, carrying 53 asylum seekers, by 2008 these numbers rose to 161 people on seven boats, increasing rapidly to 6555 arrivals on 134 boats in 2011 and an astonishing 20,587 people on 300 boats by 2013 (Figures from the Refugee Council of Australia, https://www.refugeecouncil.org.au/asylum-boats-statistics/, April 2025), Asylum seeker policy of the major parties was a key concern during this period with the tragic loss of life at sea from failed crossings, coupled with images and stories of immigration detention (including children), intermingling with the wars and political hardships being fled by the occupants of these boats, weighing heavily on the Australian psyche.

Following the paired-completion method, speeches (texts) were first selected into the study through classical, strict (literal) keyword matching (Terms used for literal matches: [asylum seeker, seeking asylum, border protection, sovereign border, people smuggling, mandatory detention, refugee status, migration policy, stop the boat, turn-back, detention centre, maritime arrival, unauthorised boat, boat arrival, Australian border force, offshore processing, offshore detention, regional processing.)). In addition, speeches were restricted to those having lengths of between 150 and 1200 characters to ensure speeches were long enough for accurate analysis (i.e. not interjections), and not too long that the speech may cover multiple topics. Together, these steps resulted in a corpus comprising 2856 speeches related to the asylum seeker issue from the 11 party leaders.

Findings

Figure 5 presents longitudinal results of the diff scores arising for the 11 speakers in the corpus. Time was dynamically binned based on the pooled count of speeches in the corpus to adaptively focus on the rhythm of the intensity of debate. Markers are included for a speaker if, for that time period, they had at least three speeches from which to draw an inference. Speaker diff average scores are plotted, with 90% confidence intervals (bootstrapped) provided for reference.

Figure 5.

Framing analysis with LLMs: “paired-completion” (Angus and O’Neill, Reference Angus and O’Neill2024) applied to asylum seeker debate framing in Australian Federal Parliament (2006–2023). Each line represents paired completion scores on a scale from negative (pro Border Security Approach) to positive (pro Humanitarian Approach) for the speaker shown. Rings around markers indicate that the speaker was the Prime Minister at the time. Federal election dates are indicated with vertical grey lines for context.

We draw the following insights with respect to the paired completion (P-C) analysis:

1. 1. P-C is well suited to identifying debate polarisation—as a method which, by construction, exploits the differential textual alignment between one framing and another, by placing speakers on a Humanitarian—Border Security axis, the method is well suited to debate polarisation analysis. In this case, P-C strongly finds two “camps” emerge in the debate, with all PMs drawn from the lower score, “border-security” camp, with clear day-light to the higher score, “humanitarian” camp of the independents and greens. Thus, confirming Hypothesis 1.

2. 2. P-C can pick up significant shifts in framing—since P-C assigns a real-valued score to every text, it is possible to bring statistical machinery to the results. Here, bootstrapped confidence intervals reveal that the gap between camps, or even speakers are significant from a statistical standpoint, thus P-C lends itself to stronger forms of hypothesis testing. Here, we can see that Albanese’s trajectory charts a course which begins within the distribution of Independent/Greens speakers, but ends statistically different from them, and thus indistinguishable from the Liberal position of the Abbott/Turnbull years, thus supporting Hypothesis 2;

3. 3. P-C reveals interesting cyclical dynamics—an unexpected result that the method reveals is the way that the independent/greens show cyclical framing dynamics around elections. We can see that in the case of Hanson-Young, Wilkie, and Natale, speeches just after an election are found to be most aligned with the humanitarian framing, but as the next election approaches, their speeches shift towards the centre. The same dynamics (but from the opposite direction) for leaders such as Rudd, Abbott, Turnbull, and Morrison is evident; further analysis would need to be done to explore these dynamics causally, but centre-shifts in political framing prior to elections are unsurprising.

As a complement to the time-series analysis, we can also inspect the most “extreme” speeches in the corpus, as scored by paired-completion on this topic. The highest three scored speeches (i.e. most aligned with the Humanitarian Approach) unsurprisingly come from the Green/Independent speakers (Note: some speeches end with a hyphen, indicating an interjection.):

Speaker: Sarah Hanson-Young (19 June 2014) (diff: 14.49).

Speaker: Andrew Wilkie (21 Oct 2019) (diff: 14.42).

Speaker: Sarah Hanson-Young (14 July 2014) (diff: 14.36).

These speeches are clearly in the Humanitarian camp, with each text arguing strongly against the “unconscionable,” “ridiculous,” “fearmongering” stance of the government of the day. As with prompt-based methods in our earlier demonstration, paired-completion can flexibly handle text of any length.

Turning to the other framing, the lowest three (i.e. most negative, most aligned with the Border Security Approach) scored speeches all come from the main party leaders:

Speaker: Scott John Morrison (2 Nov 2011) (diff: −23.42).

Speaker: Kevin Michael Rudd (26 Oct 2009) (diff: −19.99).

Malcolm Bligh Turnbull (9 Nov 2016) (diff: −18.50).

Of interest, and showing the strength of recent transformer based models, whilst the first text (Morrison) argues strongly for a hardened, offshore, processing amendment to the incumbent’s policy, the second and third texts (Rudd, Turnbull) both include language pointing to the humanitarian consequences (Rudd: “the scourge of people smugglers and the humanitarian plight of asylum seekers”; Turnbull: “most tragically of all, 1200 deaths at sea of which we know”) yet rightly the model understands that these notes sit within a wider semantic posture supporting tougher border policies in respect of asylum seekers. This kind of nuanced understanding of human language is both critical for political narrative analysis and unachievable in the pre-transformer technology era.

6.1. Recommendations for data: removing blockers to policy-relevant discourse analysis at scale

Policymakers who wish to analyse the strength or fragility of our democracy by tracking narratives of democratic resilience may rightly consider starting with the narratives spun by our own elected ministers and the election platforms that would-be ministers are touting. Together, these would provide a key input to narrative analysis for policy-relevant questions such as: is populist language on the rise? What fraction of election platforms in the recent election touted exclusionary nationalism? Which jurisdictions are on the receiving end of political platforms that portray an “Us vs. Them” narrative?

Unfortunately, through an analysis of the situation in several apparently healthy democracies, e.g. the United Kingdom (scoring 91 on Freedom House’s “Global Freedom Scores” [Freedom House, https://freedomhouse.org/countries/freedom-world/scores]), Australia (95), and Finland (100), we find that, despite the promise of NLP/AI technology, answering these questions is still frustrated by failures in rather more basic requirements.

6.2. Recommendations for future research: towards real-time tracking of policy-relevant democratic discourse

We finish by sketching a potential agenda for further democratic narrative research that would support policymakers, academic researchers, and the wider community alike. Granted, significant data challenges exist, but laying these to one side, we consider a world where data were more available, enabling enriched analysis with AI-Transformer backed technologies already in use in other disciplines and fields as described in Section 4.

6.3. Research recommendation 3: supporting accountability in our Parliamentary discourses: a public monitor of democratic discourse

One of the major challenges facing the public, journalists, and academics alike is the challenge of identifying objectively that a speaker or outlet is saying something unusual (for them) or fitting a particular narrative of democracy. Without accurate, quantitative tools, we are left relying on “opinion” pieces and expert talking heads to opine on whether a particular speaker’s comments really were anti-democratic, populist, or divisive. Whilst expert commentary will always be important, what we lack is a way for the public to see insightful visualisations, driven by accurate discourse measurement, of how a particular discourse fits in to a spectrum of discourse at a point in time, or across time. Such a “Narrative Observatory” like tool would be a major benefit to the democratic project. Speakers would be held immediately accountable for their words, policymakers, journalists, and analysts would more easily be able to draw comparisons and make data-driven insights about the positions being taken on the floors of our Parliaments nationally. Fact-checking initiatives exemplify this kind of public good provision; a narrative observatory would complement and empower our society to better understand and hold to account those who represent us.