Introduction
In recent years, scholars of political inequality have turned to field experiments for solutions to disparities of representation and influence (e.g., Foos and Gilardi Reference Foos and Gilardi2020; Garcia Bedolla and Michelson Reference Garcia Bedolla and Michelson2012; Kalla, Rosenbluth and Teele Reference Kalla, Rosenbluth and Teele2018). Many of these experiments use interventions to induce behavioral changes in subjects; however, questions remain about the extent to which experimentally induced changes are durable over longer periods of time. Understanding the continued efficacy of field experimental interventions designed to reduce inequality is an important scientific aim because in the limited cases when scholars engaged this question, they have made substantial theoretical contributions (Broockman and Kalla Reference Broockman and Kalla2016; Garcia Bedolla and Michelson Reference Garcia Bedolla and Michelson2012).
We focus on the long-term effect of interventions by party leaders to increase women’s representation. In the context of a 2014 state Republican caucus meeting, Karpowitz, Monson and Preece (Reference Karpowitz, Monson and Preece2017) found that the combination of stoking both the supply of and demand for women delegates via messages from party leaders significantly increased women’s representation. In this article, we ask whether the experimental treatment effects initially seen in the 2014 precinct elections endured through the 2016 caucus meetings.
We find no evidence that experimental effects persisted, but importantly, this result is not a function of treated precincts regressing to prior lower levels of women’s representation. Precincts originally assigned to the highly effective Supply + Demand condition maintain their higher levels of women’s representation two years later. However, the size of the 2016 treatment effect is smaller and no longer statistically significant because of increases in women’s representation in the control condition.
In light of these results, we ask two additional questions: first, why did women in the Supply + Demand condition retain their experimentally induced higher levels of representation? Second, what caused women’s representation in the control condition to increase over time? Examining multiple possible explanations, we find considerable evidence of a distinct and durable incumbency effect among women in the Supply + Demand condition, and increases in women’s representation in the control condition appear to be related in part to larger turnout during the 2016 election cycle. By contrast, we find little evidence of lasting changes in voter attitudes about women’s representation across any of the treatment precincts and few traces of post-experimental spillover across treatments. These findings not only emphasize the importance of continued attentiveness to field experimental interventions over time but also highlight the staying power of institutional factors in addressing gender inequality.
The Republican Caucus Experiment
Women’s underrepresentation in American politics is driven largely by the Republican Party.Footnote 1 Research on why women’s representation in the Republican Party is so low has increased in recent years (e.g., Carroll and Sanbonmatsu Reference Carroll and Sanbonmatsu2013; Elder Reference Elder2021; Karpowitz et al. Reference Karpowitz, Monson, Preece and Aldridge2024; Och and Shames Reference Och and Shames2018; Thomsen Reference Thomsen2015; Wineinger Reference Wineinger2022), but research on effective interventions to decrease the gender representation gap in the GOP remains less common.
However, Karpowitz, Monson and Preece (Reference Karpowitz, Monson and Preece2017) found that simple messages from party leaders to party activists were able to substantially increase women’s representation among delegates to the state convention. The researchers worked with the state Republican Party in a Republican-dominated state to exogenously shock the supply of and demand for female candidates vying for the highly desirable position of delegate to the state nominating convention. Prior to the party caucus meetings in March of 2014, precinct chairs were randomly assigned to receive one of four letters from Republican Party leadership in a 2 × 2 factorial design. The letters contained one of the following treatment conditions:
Placebo Control Precinct chairs were encouraged to foster a good environment at the caucus. This condition included no mention of gender.
Supply Precinct chairs were encouraged “to reach out and encourage two or three women in your precinct to run for positions as precinct leaders or delegates this year.”
Demand Precinct chairs were encouraged to read out loud a message about women’s underrepresentation from the state party chair at the beginning of the caucus meeting that included a plea to “consider this as you nominate and vote for delegates. We would like to see more women at our state convention.”
Supply + Demand Precinct chairs were encouraged to reach out to two or three women and read the paragraph.
The researchers found that party leaders’ efforts to stoke both supply and demand together increased the number of women elected as delegates by about six percentage points.
Did the effects of these experimental treatments endure into the 2016 precinct elections? On the one hand, regression to the mean is a likely outcome, especially given the high year-to-year turnover in attendance at caucus meetings. On the other hand, treatment effects might be expected to persist, either through norm changes or through other pathways such as incumbency advantage. For example, the experience of being a state delegate may make incumbents stronger candidates in subsequent years.
Data
We obtained information from the Republican Party about participation in the party’s neighborhood caucuses in 2012, 2014, and 2016. We include the 2012 data, collected prior to the experimental interventions, as a point of comparison. However, many new precincts were created between 2012 and 2014, so 2012 is not a perfect baseline, and comparisons with that year should be treated with caution. We also restrict our analysis to the precincts that were sent a letter as part of the original experiment and thus had some opportunity to receive the treatments.Footnote 2 In addition, party records for each year contain some gaps: some precincts did not report either attendance or the identities of participants who were elected as delegates or both.Footnote 3 Because we are working from the same dataset used by party leaders, however, the precinct- and individual-level data we analyze here include the best available data about both participation and the election of state delegates.
Table 1 shows that as random assignment would predict, precincts are roughly evenly spread across the experimental conditions.Footnote 4 A slightly larger proportion of 2016 meeting attenders in control condition precincts were women compared to 2014 (0.46 compared to 0.45), but this difference is not statistically significant (p = 0.20), and there are no other statistically significant differences across the conditions with respect to any of the precinct-level variables shown in Table 1.
Summary data
Table 1 Long description
The table presents data on precinct conditions and various metrics for the years 2012, 2014, and 2016. It includes the total number of precincts, control condition precincts, supply condition precincts, demand condition precincts, and supply plus demand precincts. Additionally, it shows the mean number of delegate positions available, the mean age of caucus attendees, the mean number of caucus attendees, and the mean proportion of attendees who are women. The table has 11 rows and 4 columns. Row 1: Total number of precincts, 1580, 1688, 1671. Row 2: Control condition precincts, 400, 426, 417. Row 3: Supply condition precincts, 397, 429, 412. Row 4: Demand condition precincts, 400, 417, 429. Row 5: Supply plus Demand precincts, 383, 416, 413. Row 6: Mean number of delegate positions available, 2.00, 2.01, 1.99. Row 7: Mean age of caucus attendees, 55.21, 54.19, 51.91. Row 8: Mean number of caucus attendees, 56.61, 25.55, 38.53. Row 9: Mean proportion of attendees who are women, 0.48, 0.44, 0.44.
Summary statistics of caucus precincts for each year, limited to precincts included in the original experiment.
One important difference between 2014 and 2016 is that overall attendance was much higher in the caucuses held during the presidential nominating process. Average caucus attendance at the precincts increased by more than 50% (p < 0.0001). The 2016 caucuses thus took place in a context of substantially higher levels of political engagement, and the set of meeting attenders may have somewhat different characteristics than those who voted in the 2014 meetings (Karpowitz and Pope Reference Karpowitz and Pope2015; Panagapoulos Reference Panagapoulos2010). Changes over time should also be considered in light of this different context.
Results
What happened in 2016
To evaluate the persistence of treatment effects over time, we focus on the proportion of the precinct’s elected state delegates who are women. Panel A of Figure 1 shows women’s share of the proportion of state delegates across the experimental conditions in 2012, 2014, and 2016. Regression analyses used to produce these results are available in Appendix Table A1.Footnote 5 Results are robust to the inclusion of controls for the number of delegate positions elected from the precinct, the mean age and number of caucus attendees, the gender balance of the caucus attendees, and the precinct location.
Effects of experimental treatments on proportion of state delegates who are women.
Figure 1. Long description
Panel A: Three scatter plots depict the proportion of delegates who are women across different experimental conditions for the years 2012, 2014, and 2016. The x-axis represents the experimental condition (Control, Supply, Demand, Supply+Demand) and the y-axis represents the proportion of delegates who are women. Each plot includes two data series: No Controls (circles) and Controls (triangles). Panel B: Three scatter plots depict the average marginal effect of experimental conditions for the years 2012, 2014, and 2016. The x-axis represents the experimental condition (Supply - Control, Demand - Control, Supply+Demand - Control) and the y-axis represents the average marginal effect. Each plot includes two data series: No Controls (circles) and Controls (triangles). The red dashed line indicates a zero effect.
The 2012 proportion of state delegates who were women hovered at approximately 25% regardless of the treatment condition, as expected given the absence of experimental interventions. In 2014, substantial differences emerged, replicating the result from Karpowitz, Monson and Preece (Reference Karpowitz, Monson and Preece2017). More precisely, after controlling for precinct-level characteristics, nearly 30% of state delegates in the Supply + Demand condition were women, compared to less than 24% in the control condition (p = 0.003).
These differences between treatment and control conditions did not persist in 2016 (p = 0.31), but this null effect was not the result of a large regression to pre-experimental rates of women’s electoral success in treated precincts. The raw proportion of women elected as state delegates in Supply + Demand precincts remained constant at 28.9 in 2014 and 2016 (p = 0.99, two-tailed difference of means test).Footnote 6 Comparing across the experimental conditions, the estimated proportion of women elected delegates was again highest in the Supply + Demand treatment. At the same time, the experimental effects dissipated in 2016 primarily because women’s representation in the control condition increased by nearly 4 percentage points, from 23.6% to 27.5% (p = 0.12). Thus, the ATE in 2016 was about 1/3 the size of the 2014 effect in models with controls and no longer statistically significant. Panel B of Figure 1 shows the average marginal effect of each of the treatments relative to the control in each year, along with 95% confidence intervals. As seen in Section 2 of the Supporting Information Appendix, all of these patterns were nearly identical for a second dependent variable: the proportion of precincts electing at least one woman.
What happened in Supply + Demand
As previously mentioned, empirically, the effects of 2014 did not persist because the number of women elected as state delegates rose in the control condition relative to the levels in Supply + Demand precincts. However, this does not erase the fact that the effects of 2014 did persist in 2016 in the Supply + Demand condition. Normatively, this presents a positive outcome: an intervention in 2014 had persistent effects among the treated units such that more women were elected for office in 2016. For this reason, we explore a potential mechanism behind the persistent representation levels found in the Supply + Demand treatment.
Across a variety of electoral settings, incumbency is strongly related to electoral success. In the neighborhood setting that is the focus of this study, turnover tends to be high: party administrative data shows only about 20% of 2014 state delegates were elected again in 2016. Nevertheless, across all experimental conditions and all participants, evidence for a modest incumbency effect is strong: delegates who were elected in 2014 were, on average, about 15 percentage points more likely than non-incumbents to be elected in 2016 (p < 0.01).
How did the experimental interventions influence men and women’s probability of being re-elected in 2016? To answer this question, we turn to individual-level party administrative data about all caucus attenders, including which attenders were elected as state delegates. Figure 2 shows the predicted probability of being re-elected in 2016 among men and women who were elected as state delegates in 2014, controlling for whether or not the participant attended meetings in both years.Footnote 7 Among men, there are no statistically significant differences across the conditions that meet the 95% confidence threshold, though delegates in the supply condition had the most success (b = 0.38, p = 0.081, compared to the control condition).Footnote 8
Effects of experimental treatments on re-election.
Among women, however, the pattern is very different and parallels the results from the initial experiment. Incumbents in the Supply + Demand condition were more than 7 percentage points more successful than those in the control condition (p = 0.040). In other words, in the condition where the initial experiment produced the biggest electoral effect, women experienced rates of subsequent re-election that were much higher than women in the control condition. In the supply and demand conditions, the incumbency effect was positive but not statistically significant relative to the control. In a difference-in-differences analysis comparing the gender gap between incumbents and non-incumbents, incumbent women in the Supply + Demand condition were the only group that experienced a significant advantage over men (see Appendix Table A3). Thus, women’s representation did not revert to pre-treatment levels in the Supply + Demand condition in part because incumbent women in that condition’s precincts were unusually successful in retaining their elected offices.
Reasoning about the Null
What explains the increases in women’s representation in the control condition between 2014 and 2016 and the modest but persistent durability of women’s increased representation over the years in the Supply + Demand condition? We examine three potential explanations: spillover, durable attitudinal changes about gender representation, and the effects of increased political participation.
Spillover
Is it possible that the changes in the control condition are the result of spillover from the treatment conditions between 2014 and 2016? We cannot entirely rule out that possibility, but the evidence is thin. We contacted party leaders to ask if they had undertaken a campaign to spread those messages leading up to the 2016 caucus meetings. The party’s executive director sent us the materials used to train precinct chairs prior to the 2016 caucus meetings, and these made no mention of electing more women. Furthermore, the executive director could recall no discussion of the topic among state party leadership, and a search of his email turned up no formal discussion of the issue. Moreover, the president of the statewide federation of Republican women revealed that she had not heard of any organized effort either.
However, spillover could have happened informally. Many precinct caucuses are conducted in public schools or other large public buildings that host multiple precincts. Newly elected precinct officers and delegates often linger after the caucus concludes to exchange contact information, fill out paperwork, and receive information from legislative district leaders or other party officials. Precinct chairs and delegates also had several opportunities to interact as preparations and training for 2016 occurred. Still, we have no direct evidence for this mechanism.
Durable Attitudinal Changes
What about the possibility of a wider culture change in gender attitudes among Republicans in the state? To examine attitudes among caucus-goers about gender representation, we conducted an online survey of 2016 Republican caucus participants. We obtained an email list of caucus participants and administered the survey just a few days after the caucus meetings held in the spring of 2016.Footnote 9 We had unexpectedly high cooperation rates and received more than 10,000 responses.Footnote 10 In addition, we find no evidence of differential participation in the survey across 2014 treatment assignment.Footnote 11
We asked caucus-goers their level of agreement with the statement, “More women are needed as convention delegates.” Figure 3 presents results for the 1,593 respondents (925 men and 668 women) who reported attending the 2014 caucus.Footnote 12 Attitudes were virtually the same across the treatment conditions, and in fact, the point estimates were lowest among both men and women in the Supply + Demand condition. Thus, we find no evidence of a substantially elevated concern about the need for greater representation of women in any experimental condition. To the extent that there is any pattern at all, women in the control condition were most likely to emphasize the need for more women as convention delegates. Perhaps this suggests that women who attended the caucus in 2014 were sensitive to the extremely low levels of women’s representation in those precincts.
2016 agreement that “More women are needed as convention delegates” self-reported 2014 attendees only.
Next, we presented a random subset of survey respondents with statements from Republican officials encouraging greater women’s participation, then asked them if they had ever heard Republican officials make such statements. In the 2014 experiment, half of precinct chairs were asked to read a statement from Republican officials encouraging greater women’s participation (Demand or Demand + Supply). Thus, survey respondents who were assigned to precincts in one of those two conditions should be more likely to say such statements were familiar. Compared to non-attenders, 2014 caucus attenders were more likely to report that they had heard party leaders emphasize the importance of women as state delegates: 36% of attenders said they had heard the messages before, compared to 23% of non-attenders (p < 0.001).
However, among attenders, we find no evidence of differences across the experimental conditions: 36% of respondents in Demand and Supply + Demand conditions said they had heard party leaders make statements about this issue, compared to 35% of those in the control or supply condition (p = 0.79). When combined with the results that show no attitudinal differences across precincts, this casts significant doubt on the likelihood that norm changes spurred by the treatments account for the pattern of outcomes.Footnote 13
Effects of Changing Levels of Caucus Attendance
Finally, we can use our dataset of all caucus attenders for one additional purpose: to explore changes in the likelihood of being elected as a state delegate between 2014 and 2016. The dataset includes 91,258 individuals, 44,276 of whom attended caucus meetings in 2014 and 66,736 of whom attended in 2016. These unique data allow us to compute the gender gap in the likelihood of being elected each year. For example, in the control condition, the dataset includes nearly 11,000 attendees in 2014 – more than 6,000 men and nearly 5,000 women. That year, a little more than 10% of men who attended the control condition were elected as state delegates, compared to 4% of women – a gap of about 6 percentage points. In 2016, the dataset includes nearly 9,000 men who attended in control condition precincts and over 8,000 women. With the higher attendance, the likelihood of being elected a state delegate dropped for both sexes. Approximately 5.6% of male attenders were elected as state delegates, compared to 2.1% of women. The 2016 gap (3.5 percentage points) was thus a little less than half the size of the 2014 gap.Footnote 14
Change in gender gap between 2014 and 2016, by experimental condition.
Figure 4 shows how the gender disparity in the likelihood of being elected changed between 2014 and 2016 across the experimental conditions. Higher scores indicate a greater reduction in the gender gap. As the figure clearly shows, the largest movement toward gender parity occurred in the control condition, and the smallest occurred in the Supply + Demand condition, where the likelihood of men and women being elected was already closer to equality in 2014 because of the experimental treatments. To be clear, across all conditions and both years, men were still more likely than women to be elected as state delegates, so gender equality remains an elusive goal. But as overall levels of participation increased in 2016, the gap between men and women in the likelihood of being elected as a state delegate also shrank the most in the precincts that were most unequal in 2014: those randomly assigned to the control condition.
Of course, 2016 was an unusual election year, one in which the Democratic candidate was a woman and the candidate who ultimately won the Republican nomination was accused of having a history of racist and misogynist attitudes and other controversial behavior toward women. At the time the 2016 caucus meetings were held, Donald Trump was not the preferred choice of most caucus attenders, coming in third behind Ted Cruz and John Kasich. We cannot exclude the idea that increases in women’s representation were at least partly related to this unique electoral context, though as we have shown, survey data revealed no evidence of robust differences in a commitment to gender equality across the experimental conditions.
Conclusion
We have assessed whether or not the treatment effects from a field experiment meant to increase women’s representation at a Republican state party convention persisted two years later. Overall, women were still substantially under-represented relative to both their proportion of the population and their attendance at caucus meetings, and our re-examination showed no enduring treatment effects. However, this did not mean that precincts in the study merely reverted to pre-treatment levels. Precisely the opposite is true: the experimental conditions with the largest effects in 2014 remained at relatively high levels in 2016, supported in part by a substantially larger incumbency effect in precincts assigned to that condition. At the same time, the control condition moved appreciably in the direction of increased representation for women.
While this incumbency effect is important and intriguing, equally important is the question of what prompted change in the control condition. The evidence is circumstantial, but the most substantial difference between 2014 and 2016 was a dramatic increase in overall levels of political participation during the presidential nominating process, compared to patterns of off-year engagement. One consequence of the increased participation was that the gap between men and women in the likelihood of being elected as a state delegate declined, with the largest reductions occurring in the control condition. While the available evidence does not offer a definitive explanation, the effects of increased participation from a larger and more ideologically moderate set of caucus attendees is one plausible possibility. At the same time, we find no direct evidence that these dynamics were the result of spillover across conditions, and we find little evidence of meaningful change in public norms or attitudes.
The original study involved a unique encouragement design undertaken in partnership with the state Republican Party and showed that party leaders could take some simple steps to promote women’s electoral success. Understanding whether these effects remained two years later has both scientific and normative value. Despite the proliferation of important field experiment designs, researchers know less about the long-term effects of their interventions. In this case, the original interventions led to a meaningful increase in women’s presence in important state party decision-making. Documenting whether those interventions have continued power over time is thus also an issue of normative importance.
Together, these results highlight the importance of understanding experimental effects over the long term. Normatively, they also point to an important consideration as those interested in durable social change decide how to invest their efforts. Attitude change about gender is difficult to induce, and efforts to do so are sometimes counterproductive. Institutional changes such as a focus on incumbency or on factors that are likely to boost overall levels of civic engagement may be more effective strategies. Outside of the robust work on gender quotas, however, political scientists have spent significantly less time studying gender and institutions than gendered attitudes. We hope this unique re-examination of field experimental data two years later provides not only helpful evidence about the persistence of experimental effects (or lack of it) but also further motivation to focus on structural ways to pursue gender equality in political representation.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/XPS.2026.10036.
Data availability statement
Financial support for this research was provided by Brigham Young University’s Center for the Study of Elections and Democracy, College of Family, Home, and Social Sciences, and Department of Political Science. The data, code, and any additional materials required to replicate all analyses in this article are available at the Journal of Experimental Political Science Dataverse within the Harvard Dataverse Network (Aldridge et al. Reference Aldridge, Karpowitz, Monson and Preece2026), at: https://doi.org/10.7910/DVN/YQ3J3W.
Competing interests
One of the authors previously consulted for the party under the administration of a different party chair. There were no ongoing ties during this project. We declare no conflicts of interest that would compromise or appear to compromise professional judgment or integrity in the conduct or reporting of research. No author has received financial support from any interested party related to this article. No author holds a paid or unpaid position as officer, director, or board member of a non-profit or profit-making organization relevant to this article. No party had the right to review this manuscript prior to its circulation.
Ethics statement
This project was approved by Brigham Young University’s Institutional Review Board (Protocol X130426 and subsequent amendments). The studies analyzed in this publication adhere to APSA’s Principles and Guidance for Human Subjects Research. Participants were recruited via email and given the opportunity to electronically consent or not on the first page of the Qualtrics survey. Risks of participation were minimal. Participants were not deceived and were not compensated.
Open access
