Introduction

In recent years, significant concern has emerged regarding the potential threat that large language models (LLMs) pose to democratic societies through their persuasive capabilities. Numerous scholars, governments, and industry leaders have warned that these systems could undermine democratic processes by manipulating public opinion, amplifying political polarization, and enabling mass-scale computational propaganda (Bengio et al., Reference Bengio, Mindermann, Privitera, Besiroglu, Bommasani, Casper, Choi, Fox, Garfinkel and Goldfarb2025; Hackenburg and Margetts, Reference Hackenburg and Margetts2024; Hackenburg et al., Reference Hackenburg, Tappin, Röttger, Hale, Bright and Margetts2025; Goldstein and Sastry, Reference Goldstein and Sastry2023; Goldstein et al., Reference Goldstein, Sastry, Musser, DiResta, Gentzel and Sedova2023; Shevlane et al., Reference Shevlane, Farquhar, Garfinkel, Phuong, Whittlestone, Leung, Kokotajlo, Marchal, Anderljung and Noam2023). For example, a malicious foreign actor could use LLMs to generate personalized political content at an unprecedented scale, leading to crises in democratic confidence and civil unrest (Karnofsky, Reference Karnofsky2024).

In this paper, we propose a framework to assess the persuasive risks that LLM chatbots may pose. Existing frameworks for evaluating the political persuasion impact of LLMs fail to incorporate the real-world challenge of getting potential voters to interact with LLM-produced content. In the real world, political persuasion is a two-step process in which a voter must first receive a political message before they can choose whether or not to accept that message (Zaller, Reference Zaller1992). There is a well-developed literature on incorporating this stage in assessments of persuasion by campaign messages (Arceneaux and Johnson, Reference Arceneaux and Johnson2010; de Benedictis-Kessner et al., Reference de Benedictis-Kessner, Baum, Berinsky and Yamamoto2019) by examining how selective exposure operates in partisan media environments. Existing academic and industry research on LLMs has focused primarily on the second step, answering the question: conditional on being exposed to LLM-produced content in a controlled laboratory setting, how persuasive is an LLM?

We expand upon existing research by conducting two survey experiments and a real-world simulation exercise to determine whether it is more cost-effective to persuade a large number of voters using LLMs compared to standard political campaign practice, taking into account both the receive and accept steps in the persuasion process. We find that, conditional on being exposed to LLM-produced content, LLMs are about as persuasive as actual political campaign TV and digital ads. After taking into account the real-world costs of exposure, we estimate that LLM-based persuasion costs between $48 and $75 per persuaded voter compared to $100 for traditional campaign methods. However, given the difficulties in scalable exposure to LLM-based persuasion, LLM chatbots today likely do not pose a substantial persuasion threat to democratic societies, although this risk is likely to increase with the growing usage of LLMs.

LLMs and political persuasion

Research on LLMs as political persuasion tools has expanded rapidly, yet current approaches might not capture their full real-world impact. First, many existing studies from both academic and industry researchers evaluate LLM persuasiveness using single-message interactions rather than extended conversations with chatbots. For example, prior work has compared LLM-generated persuasive messages to those written by humans but limited their evaluation to single messages (Bai et al., Reference Bai, Voelkel, Muldowney, Johannes and Willer2025; Hackenburg et al., Reference Hackenburg, Tappin, Röttger, Hale, Bright and Margetts2025). This may underestimate LLMs’ persuasiveness, as research often finds that extended conversations are particularly persuasive (Broockman and Kalla, Reference Broockman and Kalla2016; Green and Gerber, Reference Green and Gerber2019; Kalla and Broockman, Reference Kalla and Broockman2020). LLMs engaging in conversation might be substantially more persuasive as they can promote more active engagement, dynamically respond to voter concerns, and tailor arguments to individual predispositions (Matz et al., Reference Matz, Teeny, Vaid, Peters, Harari and Cerf2024; Simchon, Edwards, and Lewandowsky, Reference Simchon, Edwards and Lewandowsky2024; Teeny et al., Reference Teeny, Siev, Briñol and Petty2021).Footnote ¹

Second, when researchers have examined the persuasiveness of LLMs engaged in conversations, they typically do not include a human or campaign standard practice comparison (Costello, Pennycook, and Rand, Reference Costello, Pennycook and Rand2024, Reference Costello, Pennycook and Rand2025; Crabtree et al., Reference Crabtree, Holbein, Bosley and Sevi2024).Footnote ² Without direct comparisons to human campaign tactics, it remains unclear whether LLMs represent a novel threat.

Third, industry evaluations of LLM persuasive capabilities suffer from methodological flaws that conflate perceived and actual effectiveness. For instance, OpenAI’s o1 risk assessment framework classifies political persuasion as a “medium risk” application of their models based primarily on raters’ perceptions of a message’s persuasiveness rather than experimental evidence of attitude change (OpenAI, 2024). This approach mirrors a common error in political communication research where message effectiveness is judged by perceived persuasiveness rather than demonstrated impact on attitudes or behaviors (O’Keefe, Reference O’Keefe2018). Given that perceived message effectiveness is rarely correlated with actual message effectiveness (Broockman et al., Reference Broockman, Kalla, Caballero and Easton2024; O’Keefe, Reference O’Keefe2018), such evaluations may dramatically over- or under-estimate LLMs’ real-world persuasive potential.

Lastly, almost all existing research on LLMs and political persuasion overlooks the first step in the persuasion process: the cost of voter engagement with the persuasive material. Existing studies measure the persuasiveness of LLMs conditional on exposure. However, political campaigns expend substantial resources in the first place simply to get voters to pay attention to their messages through paid advertising, door-to-door canvassing, phone banks, and digital outreach (Limbocker and You, Reference Limbocker and You2020). The challenge of competing for voters’ limited attention represents a significant barrier to real-world persuasion that laboratory settings bypass (Barabas and Jerit, Reference Barabas and Jerit2010; Coppock and Green, Reference Coppock and Green2015; Jerit, Barabas, and Clifford, Reference Jerit, Barabas and Clifford2013). Ignoring this step may dramatically overestimate LLMs’ threat to democratic processes by implicitly assuming similar exposure costs between human campaign persuasion content and LLMs. In reality, even if LLMs prove exceptionally persuasive when audiences engage with them, their real-world impact might be severely constrained if the costs of achieving widespread engagement remain prohibitively high or if engagement rates with LLM-generated content are substantially lower than with traditional campaign tactics.

Our research addresses each of these limitations. First, we build upon prior research on door-to-door canvassing to test the persuasive impact of LLM chatbots (Broockman and Kalla, Reference Broockman and Kalla2016; Kalla and Broockman, Reference Kalla and Broockman2020; Kalla and Broockman, Reference Kalla and Broockman2023; Kalla, Levine, and Broockman, Reference Kalla, Levine and Broockman2022). Second, we benchmark the persuasiveness of these chatbots against current campaign practices using actual campaign advertisements, providing a meaningful comparison to existing persuasion approaches. Third, we measure both immediate and long-term attitudinal shifts rather than measures of perceived persuasiveness. Finally, we examine not only the second step of the persuasion process (acceptance of the message once exposed) but also the first step (securing voter engagement with LLM content) – allowing us to estimate the real-world threat that LLMs might pose to democratic processes.

Study 1

We conducted a randomized controlled experiment that compares LLM-driven and traditional human persuasion methods on attitudes toward immigration policy.Footnote ³ Participants (N = 5, 150) recruited from Prolific were asked to complete a survey on the Qualtrics survey platform. They were randomly assigned to one of four experimental conditions:

1. 1. Placebo condition: Participants watched a video unrelated to immigration.

2. 2. Human persuasion: Participants viewed a video featuring a human advocate presenting pro-immigration arguments.

3. 3. AI chatbot (as human): Participants engaged in interactive conversations with an AI chatbot (using Claude 3.5 Sonnet (20241022-v2)) that presented pro-immigration arguments while identifying itself as a human.

4. 4. AI chatbot: Participants engaged in interactive conversations with an AI chatbot (using Claude 3.5 Sonnet (20241022-v2)) that presented pro-immigration arguments while identifying itself as an AI.

For the last two conditions, conversations took place within Qualtrics using a JavaScript-powered chatbox (Molnar, Reference Molnar2019), with the LLM called by the AWS Bedrock API. To maximize the persuasive impact, the prompts relied heavily on the training materials used in prior experiments on the persuasive impact of conversations (Broockman and Kalla, Reference Broockman and Kalla2016; Kalla and Broockman, Reference Kalla and Broockman2020; Kalla and Broockman, Reference Kalla and Broockman2023; Kalla, Levine, and Broockman, Reference Kalla, Levine and Broockman2022; Santoro et al., Reference Santoro, Broockman, Kalla and Porat2025).Footnote ⁴ The AI was instructed to persuade participants to support the policy position that “illegal immigrants should be eligible for in-state college tuition at state colleges.” The prompt engineering included specific guidelines intended to increase its persuasiveness using findings from prior research on persuasive conversations: use an 8th-grade reading level; engage participants in back-and-forth conversation; listen actively and show understanding; build rapport through vulnerability; demonstrate compassion by acknowledging emotions; anticipate and address counter-arguments; and adapt persuasive approaches based on participant responses. The full prompts are presented in Online Appendix Section 7. Median conversations involved 4 participant turns (98.5 words), and 5 chatbot turns (258 words).

Similarly, in an attempt to maximize the persuasive impact of the human persuasion condition, participants watched a video of a teacher sharing their personal reason for supporting this immigration policy. This video was selected due to its large persuasive effect in prior research (Santoro et al., Reference Santoro, Broockman, Kalla and Porat2025). In this 3-minute video, the confederate shares that he supports the immigration policy because, as a teacher, he had first-hand experience with undocumented students who, despite their academic excellence, were unable to afford to attend college due to their immigration status.

We measured outcomes immediately posttreatment and five weeks later using three policy items, rated on five-point Likert agreement scales: whether undocumented immigrants should be eligible for in-state college tuition at state colleges, allowed to receive government help to pay for college, and receive the same government benefits as American citizens. To reduce measurement error, we compute an immigration support index by averaging the three policy items. We also re-code the individual items as binary measures, where 1 is any support for the pro-immigration policy, and 0 is any opposition or indifference. N = 3, 412 participants completed the follow-up survey five weeks later. Details on tests of covariate balance and differential attrition are in Tables A.2–A.4. To estimate average treatment effects, we use linear regression with preregistered pretreatment covariates to increase precision.

The preregistration document is available at https://osf.io/s976d/files/4vrpb?view_only=1b4aff47cf744391beafe3ba3dccdf78. See Additional Supplementary Materials for information about the correspondence between the manuscript and the preregistration document. This study was conducted in accordance with the APSA Principles and Guidance for Human Subjects Research.

Figure 1 shows our main results. We combine the two AI conditions into a single group. Across both our additive index and each outcome measure, we find that both the chatbot and human persuasion conditions are persuasive relative to the placebo immediately posttreatment and in our five-week follow-up survey. Relative to the placebo, the chatbot shows an effect of 0.363 scale points (SE = 0.027) immediately and 0.206 scale points (SE = 0.032) after five weeks. The corresponding effects for human conditions are 0.349 scale points (SE = 0.031) and 0.196 scale points (SE = 0.038). However, there is no distinguishable difference between the chatbot conditions and the human persuasion condition either immediately posttreatment (p = 0.601) or in the five-week follow-up survey (p = 0.758). Figure 1 presents additional results separately by each policy item in the additive index, similarly finding no distinguishable difference. Table A.5 contains full numerical point estimates, standard errors, and p-values. Table A.6 presents the estimated average treatment effects comparing AIs that identify as human versus those that identify as AI. The results show no significant difference in persuasive effectiveness between the two AI conditions, consistent with other research findings that authorship labels have minimal effect on the persuasiveness of AI-generated content (Boissin et al., Reference Boissin, Costello, Alonso, Rand and Pennycook2025; Gallegos et al., Reference Gallegos, Shani, Shi, Bianchi, Gainsburg, Jurafsky and Willer2025; Goel et al., Reference Goel, Bergeron, Lee-Whiting, Bohonos, Islam, Lachance, Savolainen, Treger and Merkley2025). The estimated effect of AI identifying as human, relative to AI identifying as AI, is 0.00 scale points (SE = 0.033) immediately and 0.018 scale points (SE = 0.038) after five weeks. Online Appendix Section 1.9 examines the association between the number of conversation turns and the persuasive effect. We find a positive association between the number of turns and the persuasive outcome. Table A.7 reports a non-preregistered equivalence test (Rainey, Reference Rainey2014) of AI versus Human.

Figure 1.

95% confidence interval on the average treatment effect compared to placebo in the short term (post-survey) and long term (5 weeks after the survey). See Table A.5 for full results.

Figure 1. Long description

The line graph presents treatment effect estimates for four immigration support metrics: Immigration Support Index, Same Benefits as Citizens, Allowed to Receive Government Help, and In-State Tuition. The x-axis represents the treatment effect estimate, ranging from 0.0 to 0.4, while the y-axis lists the metrics. Data points are color-coded: green for chatbot treatment and orange for human treatment. Immediate effects are marked with circles, and long-term effects with triangles. The Immigration Support Index shows higher estimates for human treatment in the long term. Same Benefits as Citizens and Allowed to Receive Government Help metrics indicate varying effects with overlapping confidence intervals. In-State Tuition shows minimal treatment effects. All values are approximated.

Taken together, these findings suggest that AI chatbots can be as persuasive as human persuaders, both in the short-term and up to five weeks later. When it comes to the second step of persuasion, there is no significant difference between the two. Before comparing cost-effectiveness, we note several Study 1 limitations that we address in a second study. First, the human persuasion condition was a 3-minute video, substantially longer than the typical 30 or 60 second TV and digital ads used by political campaigns. This longer video could overstate the effectiveness of human persuasion. Second, this study only investigated one political issue on immigration. The relative effectiveness of AI chatbots compared to human persuasion might vary across different domains. Third, LLMs continue to advance; more recent models might be more persuasive.

Study 2

We conducted a second survey experiment on Prolific (N = 5, 267) on three issues: immigration, transgender rights, and minimum wage. The persuasion treatments were designed to encourage participants to support the following policy positions:

1. 1. Illegal immigrants should be eligible for in-state college tuition at state colleges.

2. 2. Transgender people should be allowed to use the restroom that matches the gender they live every day.

3. 3. The federal minimum wage should not be increased from the current $7.25 per hour to $15 per hour.

The third policy ensures testing persuasion in both liberal and conservative directions. This study was conducted on the Qualtrics survey platform in a manner similar to Study 1. Participants were randomly assigned to a policy, and then within each policy, participants were randomly assigned to a placebo, a human persuasion, or an AI chatbot condition. The human persuasion condition involved watching a 30–60 second campaign advertisement, a scenario closely aligned with real-world political ads. The AI chatbot condition used in this study employed similar prompts to those in Study 1 and was powered by Claude 3.7 Sonnet, which outperforms Claude 3.5 Sonnet in advanced reasoning (Anthropic, 2025). The full prompts are presented in Online Appendix Section 7. Median conversation involved 4 participant turns (89 words), and 5 chatbot turns (281 words). Tables A.12 and A.13 present tests of covariate balance and differential attrition.

We measured outcomes immediately posttreatment using a single issue-specific policy item rated on five-point Likert agreement scale: whether illegal immigrants should be eligible for in-state college tuition at state colleges; whether transgender people should be allowed to use the restroom that matches the gender they live every day; or whether the federal minimum wage should be increased from the current $7.25 per hour to $15 per hour. We also re-code the individual items as binary measures, where 1 indicates any support for the policy position advocated in the treatment, and 0 represents either opposition or indifference. To estimate average treatment effects, we use linear regression with preregistered pretreatment covariates.

The preregistration document is available at https://osf.io/s976d/files/hd43t?view_only=1b4aff47cf744391beafe3ba3dccdf78. See Additional Supplementary Materials for information about the correspondence between the manuscript and the preregistration document. This study was conducted in accordance with the APSA Principles and Guidance for Human Subjects Research.

Figure 2 shows the results. Table A.14 contains full numerical point estimates and standard errors for each outcome. Table A.15 presents the point estimates, standard errors, and p-values comparing the AI persuasion and human persuasion, pooled across the three issues. Across all issues, we find no consistent evidence that the AI chatbot is more persuasive than human persuasion. If anything, when pooling across the three issues, the human persuasion condition shows slightly larger persuasive effects than the AI chatbot. We estimate that the human persuasion is 0.001 percentage points more persuasive than the chatbot (SE = 0.010; p = 0.896) for the pooled binary outcome and 0.059 scale points more persuasive (0.025; p = 0.018) for the pooled continuous outcome. Robustness check among participants who, in the pretreatment survey, opposed the position advocated in the treatment condition, yields similar effects (Tables A.17–A.18). Online Appendix Section 2.10 examines the association between the number of conversation turns and the persuasive effect for each issue. We find a positive association between the number of turns and the persuasive outcome. Table A.16 reports a non-preregistered equivalence test (Rainey, Reference Rainey2014) of AI versus Human. Online Appendix Section 8 presents a non-preregistered investigation of the on-topic engagement rates and AI persuasion strategies in Studies 1 and 2.

To conclude, even when (1) the human persuasion condition is limited to a 30–60 second campaign video, (2) a more powerful language model – Claude 3.7 Sonnet – is used, and (3) the persuasion topics span both sides of partisan issues, we find no evidence that AI chatbots outperform human persuasion.

Figure 2.

95% confidence interval on the average treatment effect of AI versus human and placebo on three different topics. Tables A.14–15 contain full results.

Figure 2. Long description

The image contains two side-by-side graphs comparing treatment effect estimates for three topics: undocumented immigrants, transgender rights, and minimum wage. The left graph represents binary outcomes, while the right graph represents continuous outcomes on a scale from one to five. Each graph features three sets of data points with error bars, color-coded to indicate different treatment effects: red for chatbot versus human, blue for human versus placebo, and green for chatbot versus placebo. The x-axis shows the treatment effect estimate, with the left graph ranging from zero to 0.2 and the right graph ranging from negative 0.2 to 0.6. The y-axis lists the three topics. The error bars indicate the 95% confidence interval for each treatment effect estimate. The red data points generally show smaller treatment effects compared to the blue and green data points across all topics and outcomes.

Real-world cost-efficacy analysis

While both AI-based and standard campaign persuasion efforts were comparably effective in shifting policy attitudes under conditions of forced exposure – for example, in Study 1, both conditions led to a ≈ 13-percentage-point increase in support for the immigration policy, and a ≈ 5-point increase one month later – real-world persuasion depends on securing exposure, a key often overlooked cost. In this section, we present the results of a bounding exercise to estimate the relative cost efficacy in the real world.

First, our human persuasion estimates from the survey experiment need to be adjusted. In our survey experiments, subjects were forced to watch the human persuasion video. They were also instructed to pay careful attention to the video. Both of these features likely overstate the real-world efficacy of our human persuasion. To adjust for this, we conducted a survey of 12 academics and campaign practitioners who are experts in survey experiments and their implications for real-world campaign effects (17 were invited, for a 71% response rate). We asked these experts their priors for how our survey experimental evidence would generalize to the real-world, asking specifically about non-skippable YouTube video ads (see Online Appendix Section 4 for full question wording).Footnote ⁵ The median expert estimated that the real-world treatment effect would be roughly 14% the size of the survey experiment treatment effect, with a minimum estimate of 1% and a maximum estimate of 60%. Given the often minimal effects of real-world persuasion (Kalla and Broockman, Reference Kalla and Broockman2018) and experts’ overconfidence in estimating persuasiveness (Broockman et al., Reference Broockman, Kalla, Caballero and Easton2024), for the purposes of this exercise, we conservatively assume that the real-world efficacy would be 1% that of our survey experiment estimate, using the smallest deflator selected by an expert. As a result, our best guess is that over the long term, the human persuasion would produce a 0.05 percentage-point treatment effect (5 * 0.01).

To estimate the cost-effectiveness of such persuasion at scale, we draw on two independent cost estimates. Based on data from Google, non-skippable YouTube ads cost between $2.70 and $8.10 per 1,000 impressions. Political practitioners we consulted estimated a substantially higher real-world average of $50 per 1,000 impression given recent costs from the 2024 election cycle.Footnote ⁶ As a conservative estimate of the cost-effectiveness of human persuasion, using the higher cost estimate from the 2024 election cycle, we estimate a cost per persuaded voter of $100 (50/1000/.0005). Additionally, YouTube has a large inventory of non-skippable advertisements. For example, YouTube estimates that a month-long advertising campaign would reach an estimated 42 million unique viewers, yielding an estimated 21,000 net new votes.

Whereas human persuasion is quite scalable and cost-effective, AI persuasion faces a fundamental difficulty: the requirement for voluntary opt-in by the target audience to speak with a chatbot. To test the scalability and expense required to persuade a large number of people, we experimented with three different recruitment methods for AI interaction.

First, we launched an advertising campaign on Meta encouraging people to have a conversation with a chatbot (see Figure 3a). We spent $199.74 on ads, which produced 53 conversations where the user made at least one statement to the AI chatbot. Inference cost around $0.05 per conversation. If we assume the treatment effect from these conversations is as large as those in our survey experiments, this implies a cost per conversation incorporating advertising and inference costs of $3.77 and a cost per vote of $75.

Second, we launched a follow-up advertising campaign on Meta where we offered to pay participants $1 for having a conversation with a chatbot (see Figure 3b). We spent $96.35 on ads and $168 on gift cards to produce 102 conversations where the user made at least one statement to the LLM chatbot and 73 where the user made at least two. If we assume the treatment effect from these conversations is as large as those in our survey experiments, this implies a cost per one-statement conversation of $2.59 and a cost per vote of $52.

Lastly, on Prolific, a campaign can get voters to have a conversation with an AI chatbot for around $2.39, between participant payment, Prolific’s fees, and inference costs. Assuming a 5 percentage-point treatment effect, this implies a cost per net vote of $48.

However, it could be difficult to scale these outreach methods. Prolific has around 110,000 active users in the United States. Reaching all of these users with an LLM chatbot would yield an estimated 5,500 net new votes. On Meta, 8,253 users saw the paid advertisement, which produced 73 conversations where the user made at least two statements to the LLM chatbot. Reaching 130 million American adults (roughly half of all American adults) with the Meta ad would yield an estimated 1.1 million two-statement conversations and 57,000 votes.Footnote ⁷

Figure 3.

Meta advertisements.

Figure 3. Long description

The image contains two Meta advertisements side by side. The left advertisement, labeled as an example uncompensated Meta ad, features a man in a suit shaking hands with a robot. The text above the image reads ‘Yale Media Opinion Survey’ and ‘Share Your Views with an AI’. Below the image, the text reads ‘Debate a Smart AI-See How Your Ideas Stack Up!’ with a ‘Learn more’ button. The right advertisement, labeled as an example compensated Meta ad, features the Yale University logo with the text ‘Earn one dollar-Share Your Views with an AI’ above it. Below the logo, the text reads ‘Debate a Smart AI-See How Your Ideas Stack Up!’ with a ‘Learn more’ button. Both advertisements have engagement metrics such as likes, comments, and shares displayed below the main content.

Across these three different recruitment modes, we estimate that AI-based persuasion costs between $48 and $75 per vote, potentially less expensive than the $100 per vote we estimate for human persuasion. However, it could be challenging to scale the LLM-based persuasion given the limited size of the Prolific audience and the low conversation conversion rate on Meta. These results suggest that while AI-based persuasion can match human performance on a per-person basis under ideal conditions of forced exposure, its real-world deployment is currently constrained by exposure costs and audience sizes.

Importantly, this exercise has several important limitations. We assume a conservative estimate of the real-world efficacy of human persuasion, taking the minimum estimate from our expert survey. On the other hand, we assume the real-world efficacy of our chatbot would be 100% that of our survey experiment. If the subjects in our survey experiment are more persuadable or more willing to engage with the chatbot than the typical American (which we find to be the case when comparing the transcripts of chats collected via Prolific compared to Meta), we could be overstating the efficacy of chatbot-based persuasion. Furthermore, our framework treats the receive and accept stages as independent events. But in reality, a voter’s susceptibility to persuasion is almost certainly correlated with their willingness to engage with the message. Future research should consider better real-world estimates of these treatment effects.