2.1 The Ethical Objective

Following the American Political Science Association (2020) principles, I consider an ethical objective that aims to avoid changing who ultimately wins office. An experimentalist with this objective seeks to minimize the probability that their interventions change the ex post officeholders. This is clearly not the only relevant ethical consideration in the design of experiments, or even the only ethical goal with respect to aggregate outcomes.Footnote ²

For example, the present objective assumes that the primary electoral consequences on policymaking or governance occur because candidate A wins office, not because candidate A won office with 60% instead of 51% of the vote. This objective abstracts from various effects of vote share (but not the winner) proposed in literatures on electoral autocracies (Simpser Reference Simpser2013) or distributive politics (Catalinac, de Mesquita, and Smith Reference Catalinac, de Mesquita and Smith2020; Lindbeck and Weibull Reference Lindbeck and Weibull1987). However, the present framework provides the tools to rigorously develop alternate considerations. The calculation of aggregate electoral impact is not affected by the specific ethical objective. The decision rule, however, does depend on how this ethical objective is specified. Alternate objectives with respect to electoral outcomes can be formalized as different decision rules in cases where effects of changes in a winner’s margin of victory pose particularly concerning implications for the welfare of voters, candidates, or district residents.

The framework starts from the observation that we can never know precisely what an election outcome would be in the absence of an experimental manipulation. This limits our ability to design an experiment to minimize the probability that their interventions change the ex post distribution of office holders or ballot outcomes. I advocate the estimation of conservative bounds on the ex ante possible change in vote share. These bounds are calculated analytically. I then develop a decision rule that compares these bounds to the predicted closeness of an election in order to minimize the risks of altering electoral outcomes. By reporting these quantities in grant applications, pre-analysis plans, and ultimately research outputs, researchers can transparently justify the their design choices.

3.1 Research Design Decisions

I first consider the components of the research design controlled by the researcher, potentially in collaboration with a partner (as in Case #2). The researcher makes three design decisions. First, she chooses the set of districts, D, in which to experimentally manipulate an intervention. Indexing electoral districts by $d \in D$ , the number of registered voters in each district is denoted $n_d$ .

Second, researchers define the clustering of subjects within a district. I assume that voters in district d, indexed by $j \in \{1,..., n_d\}$ are partitioned into $C_d$ exhaustive and mutually exclusive clusters. I index clusters by $c \in C_d$ and denote the number of voters in each cluster by $n_c$ , such that $\sum _{c\in C_d} n_c = n_d$ . There is always a cluster, even when treatments are individually assigned. Individual randomization can be accommodated by assuming $n_c = 1 \forall c$ . Similarly, district-level clustering can be accommodated by assuming $n_c = n_d$ .

Finally, researchers decide the allocation of the intervention within a district. Consider two states of the world, $E \in \{e, \neg e\}$ , where e indicates an experiment and $\neg e$ indicates no experiment. These states represent the counterfactual pairs described in Table 1. A subject’s assignment to the intervention in state E is denoted $\pi (E)$ . In the experiment, intervention assignment is random. Absent an experiment, the intervention could be assigned by any allocation mechanism. This notation allows for characterization of four principal strata, described in Table 2. By asserting the possibility of four (non-empty) strata, I allow for cases in which a researcher’s partner would assign any proportion of the electorate (including all or none) to the intervention in the absence of the experiment. I use the notation $S_{11}^{cd}$ , $S_{10}^{cd}$ , $S_{01}^{cd}$ , and $S_{00}^{cd}$ to denote the set of voters belonging to each stratum in each cluster and district. The cases defined in Table 1 place assumptions on the relevant strata. Where the counterfactual is no intervention (Case #1), strata where $\pi (\neg e) = 1$ must be empty.

Table 2 Principal strata. Each individual (registered voter) belongs to exactly one stratum. The cases refer to those described in Table 1. The $|\cdot |$ notation refers to the cardinality of each set, or the number of voters in each stratum in cluster c in district d.

With this notation, members a district’s electorate are assigned or not assigned to the intervention because of the experiment belong to two strata: $S_{10}^{cd}$ —individuals exposed to the treatment because it is assigned experimentally—and $S_{01}^{cd}$ —individuals not exposed to the treatment because is assigned experimentally. The proportion of the electorate in a district that is exposed (resp. not exposed) to an intervention due to the experiment, heretofore the experimental saturation, $\mathcal {S}_d$ can thus be written:

(1) $$ \begin{align} \mathcal{S}_d &= \frac{\sum_{c \in d} |S_{10}^{cd} \cup S_{01}^{cd}|}{n_d}. \end{align} $$

In the context of electoral interventions that would not occur absent the experiment (Case #1), the interpretation of $\mathcal {S}_d$ is natural: it represents the proportion of potential (or registered) voters assigned to treatment. For interventions that would occur in the absence of an experiment, $\mathcal {S}_d$ represents the proportion of potential voters exposed (resp. not exposed) to the intervention due to experimental assignment of treatment.

3.2 Researcher Assumptions about Interference between Voters

To construct bounds on aggregate electoral impact, researchers must make some assumptions about the set of voters whose voting behavior could be affected by an intervention. First, consider the stable unit treatment value assumption (SUTVA), which is invoked to justify identification of most standard causal estimands in experimental research. SUTVA holds that a voter’s potential outcomes are independent of the assignment of any other voter outside her cluster, where the cluster represents the unit of assignment as defined above. Denoting the treatment assignment of registered voter j in cluster c as $z_{jc} \in \{0, 1\}$ , the SUTVA for electoral outcome $Y_{jc}(z_{jc})$ is written in Assumption 1.

I add a second within-cluster non-interference assumption to the baseline model. Note that, in contrast to SUTVA, this assumption is not necessary for identification of standard causal estimands in cluster-randomized experiments. This assumption holds that, in the case that treatment is assigned to clusters of more than one voter ( $n_c>1$ ), a voter’s potential outcomes are independent of the assignment of any other voter inside her cluster, where the cluster represents the unit of assignment to treatment.Footnote ³ In other words, Assumption 2 holds that an intervention could only influence the voting behavior of voters directly allocated to receive the intervention. Analysis of within-cluster “spillover” effects in experiments suggest that this assumption is not always plausible in electoral settings (i.e., Giné and Mansuri Reference Giné and Mansuri2018; Ichino and Schündeln Reference Ichino and Schündeln2012; Sinclair, McConnell, and Green Reference Sinclair, McConnell and Green2012), so I examine the implications of relaxing this assumption in Section 5.

3.3 Voters’ Response to the Treatment

In order to ascertain whether an experimental intervention could change aggregate election outcomes, consider voting outcomes. Given treatment assignment $z_{jc}$ , I assume that vote choice potential outcome $A_{jc}(z_{jc}) \in \{0,1\}$ is defined for all j and z, where 1 corresponds to a vote for the marginal (ex ante) winning candidate and 0 represents any other choice (another candidate, abstention, an invalid ballot, etc.).

Under Assumptions 1 and 2, I bound the plausible treatment effects on vote choice for the marginal “winner” among those whose assignment to treatment is changed by the use of an experiment, that is, any $j \in \{S_{10} \cup S_{01}\}$ . Given the binary vote choice outcome, one can bound the possible (unobservable) individual treatment effects, among subjects whose treatment status is changed through the use of an experiment as: $ITE_{jc} \in \{-a_{jc}(0), 1- a_{jc}(0)\}$ . If a voter would vote for the winner when untreated ( $a_{jc}(0) = 1$ ), she could be induced to vote for a different candidate ( $-a_{jc}(0) = -1$ ) or continue to support the winner ( $1-a_{jc}(0) = 0$ ) if treated. Conversely, if a voter would not vote for the winner if untreated ( $a_{jc} = 0$ ), her vote (for any non-winning candidate) could remain unchanged $-a_{jc}(0) = 0$ or she could be induced to vote for the winner ( $1-a_{jc}(0) = 1$ ) if treated. These possible ITEs serve as the basis for construction of extreme value bounds (EVBs) (Manski Reference Manski2003).

To construct EVBs—and thus to calculate aggregate electoral impact—the expectation of untreated potential outcome of vote choice $E[a_c(0)]$ plays an important role. When treatment is cluster-assigned and $n_c> 0$ , $E[a_c(0)]$ depends on which voters are assigned to the intervention. Random assignment of voters within a cluster ensures that $E[a_c(0)]$ is equivalent at varying levels of experimental saturation in treated clusters. This assumption can be relaxed when it is inappropriate, but the bound on aggregate electoral impact may increase.

EVBs can be very wide. Researchers may instead be tempted to use estimates from existing studies or less conservative bounding approaches. These approaches can be very misleading. Existing estimates are generally some form of average causal effect (i.e., an $ATE$ ). The ethical concern is not whether an experimental intervention changes outcomes on average. If $ATE$ estimates that are small in magnitude mask heterogeneity, bounds based upon existing estimates will be non-conservative. Moreover, the outcome measures on vote share for a specific party or the incumbent (party) do not uniformly correspond to the relevant ex ante marginal winning (or losing) candidates, which means that they do not measure the effect that directly corresponds to considerations of aggregate electoral impact. Relative to other bounding approaches, I adopt EVB to avoid incorrect assumptions about the plausible effects of an intervention. This means that these bounds do not assume that an intervention will produce the hypothesized directional effect.

4.1 Bounding Electoral Impact

Given these design choices, assumptions about interference, and the model of voter response to treatment, I proceed to construct an ex ante bound on the largest share of votes that could be changed by an experimental intervention. I term this quantity the maximum aggregate electoral impact in a district, the $MAEI_d$ . Under Assumptions 1 and 2, this quantity is defined as follows:

Definition 1. Maximum Aggregate Electoral Impact: The ex ante maximum aggregate electoral impact (MAEI) in district d is given by

(2) $$ \begin{align} MAEI_d &= \max\left\{ \frac{\sum_{c \in C_d} \left[E[a_c(0)] |S_{10}^{cd} \cup S_{01}^{cd}|\right]}{n_d}, \frac{\sum_{c \in C_d} \left[(1-E[a_c(0)]) |S_{10}^{cd} \cup S_{01}^{cd}|\right]}{n_d} \right\}. \end{align} $$

Consider the properties of $MAEI_d$ with respect to untreated levels of support for the winning candidate. Note that $E[a_c(0)] \in [0,1]$ for all $c\in C_d$ . This has two implications. First, because $E[a_c(0)]$ is unknown ex ante, a conservative bound can always be achieved by substituting $E[a_c(0)] = 1$ (equivalently 0). These conservative bounds are useful when the intervention is assigned to a non-random sample of registered voters within a cluster. This clarifies that researchers can generate a conservative measure of $MAEI_d$ simply by determining size of each stratum (in Table 2) in each cluster and district, without any additional electoral data or predicted outcomes. This means that researchers need only the number of registered voters in each district and cluster alongside their experimental design to calculate this quantity. Second, holding constant the experimental design, the $MAEI_d$ is minimized where $E[a_c(0)] = \frac {1}{2}$ for all clusters in a district, with non-empty $S_{10}^{cd}$ or $S_{01}^{cd}$ . Thus, going from the least conservative prediction of $E[a_c(0)] = \frac {1}{2}$ for all c to the most conservative assumption of $E[a_c(0)] = 1$ for all c, the magnitude of $MAEI_d$ doubles.

Inspection of Definition 1 yields several observations. An otherwise identical experiment clearly has less possibility of moving aggregate vote share or turnout in a large district relative to a small district. Researchers’ desire to work in low-information or low-turnout contexts has directed research focus to legislative or local elections. This result suggests that this decision carries greater risks of changing electoral outcomes, all else equal. Second, Definition 1 implies that a higher saturation of the experimentally manipulated treatment increases aggregate impact, holding constant $E[a_c(0)]$ . This suggests a possible trade-off between statistical power and the degree to which an experiment could alter aggregate electoral outcomes.

4.2 Assessing the Consequences of Electoral Interventions

The implications of $E[a_{c}(0)]$ for $MAEI_d$ prompt a discussion of the ability of electoral experiments to change electoral outcomes, that is, who wins. Analyses of electoral experiments typically focus on turnout or vote share, not the probability of victory (or seats won in a proportional representation system). The mapping of votes to an office or (discrete) seats implies the existence of at least one threshold, which, if crossed, yields a different set of office holders. For example, in a two candidate race without abstention, there exists a threshold at 50% that determines the winner. It is useful to denote the “ex ante margin of victory,” $\psi _d$ , as minimum change in vote share, as a proportion of registered voters, at which a different officeholder would be elected in district d. In a plurality election for a single seat, $\psi _d$ represents the margin of victory (as a share of registered voters). In a PR system, there are various interpretations of $\psi _d$ . Perhaps the most natural interpretation is the smallest change in any party’s vote share that would change the distribution of seats.

If $\psi _d> 2MAEI_d$ , then an experiment could not change the ultimate electoral outcome. In contrast, if $\psi _d \leq 2MAEI_d$ , the experiment could affect the ultimate electoral outcome. Appendix A3 of the Supplementary Material shows formally the derivation of this threshold for an n-candidate race. The intuition behind the result is straightforward: $n_d \psi _d$ gives the difference in the number of votes between the marginal winning and losing candidates. The minimum number of votes that could change the outcome is $\frac {n_d \psi _d}{2}$ , if all changed votes were transferred from the marginal winner to the marginal loser. Hence, the relevant threshold is $2MAEI_d$ , not simply $MAEI_d$ .

Unlike the other parameters of the design, $E[a_c(0)]$ and $\psi _d$ are not knowable in advance of an election, when researchers plan and implement an experiment. Imputing the maximum possible value of $E[a_c(0)] =1$ allows for construction of the most conservative (widest) bounds on the electoral impact of an experiment under present assumptions, maximizing $MAEI_d$ while fixing other aspects of the design. However, imputing the minimum value of $\psi _d = 0$ , the most “conservative” margin of victory, implies that $2MAEI_d> \psi _d$ and any experiment could change the electoral outcome. Yet, we know empirically that not all elections are close and, in some settings, election outcomes can be predicted with high accuracy. For this reason, bringing pretreatment data to predict $\psi _d$ allows researchers to more accurately quantify risk and make design decisions.

To this end, researchers can use available data to predict the parameters $\psi _d$ and, where relevant, $E[a_c(0)]$ . Given different election prediction technologies and available information, I remain agnostic as to a general prediction algorithm. Regardless of the method, we are interested in the predictive distribution of $\psi _d$ , $\widehat {f}(\psi _d) \sim f(\psi _d|\boldsymbol {\widehat {\theta }})$ , where $\boldsymbol {\widehat {\theta }}$ are estimates of the parameters of the predictive model.

4.3 Decision Rule: Which (If Any) Experimental Design Should Be Implemented?

Ultimately, our assessment of whether an experimental design is ex ante consistent with the ethical standard of not changing aggregate electoral outcomes requires a decision-making rule.Footnote ⁴ I propose the construction of a threshold based on the predictive distribution of $\psi _d$ . Specifically, I suggest that researchers calculate a threshold $\underline {\psi _d}$ , that satisfies $\widehat {F}^{-1}(0.05)=\underline {\psi _d}$ , where $\widehat {F}^{-1}(\cdot )$ indicates the quantile function of the predictive distribution of $\psi _d$ . This means that 5% of hypothetical realizations of the election are predicted to be closer than $\underline {\psi _d}$ . The decision rule then compares $MAEI_d$ to $\underline {\psi _d}$ , proceeding with the experimental design only if $2MAEI_d < \underline {\psi _d}$ . While the possibility of changing 5% of election outcomes may seem non-conservative, recall that $MAEI_d$ captures the maximum possible effect of an intervention. It is highly unlikely that interventions achieve this maximum effect, reducing substantially the probability that an electoral experiment changes who wins an election.

This decision rule rules out intervention in close elections entirely. It permits experiments with a relatively high experimental saturation of treatment only in predictable “landslide” races. Basing the decision rule on predictive distribution of $\psi _d$ , as opposed to a point prediction, penalizes uncertainty over the possible distribution of electoral outcomes. Globally, the amount of resources and effort expended on predicting different elections and the quality of such predictions vary substantially. For this reason, I do not prescribe a predictive model. In some places, researchers may be able to access off-the-shelf predictions as in one case of the simulation below. In others, researchers would need to generate their own predictions using the data at hand. One implication of this variation in available information is that we are better able to make precise predictions in some electoral contexts than others. Where elections are highly unpredictable (due to lack of information, high electoral volatility, or limited election integrity), researchers concerned about aggregate impact should be more circumspect about experimental intervention. Use of this framework should alert researchers to the inherent risk of intervention in such contexts.

5.1 Within-Cluster Interference

One limitation of the previous analysis is that an intervention might only change the votes of those that are directly exposed within a cluster (Assumption 2). In this instance, clusters consist of multiple voters ( $n_c> 1$ ), but not all voters in a treated cluster are treated or untreated due to the experiment. Yet, some “always assigned” (if present) or “never assigned” voters in assigned clusters may change their voting behavior in response to the treatment administered to other voters in their cluster. In electoral context, these spillovers may occur within households (Sinclair et al. Reference Sinclair, McConnell and Green2012), intra-village geographic clusters (Giné and Mansuri Reference Giné and Mansuri2018), or constituencies (Ichino and Schündeln Reference Ichino and Schündeln2012). In these cases, the maximum aggregate electoral impact with within-cluster interference, $MAEI_d^{w}$ can be rewritten as

(3) $$ \begin{align} MAEI_d^{w} &= \max\Big\{\frac{\sum_{c \in C_d}\left[ E[a_c(0)] n_c I[|S_{10}^{cd} \cup S_{01}^{cd}|> 0]\right]}{n_d}, \frac{\sum_{c \in C_d} \left[(1-E[a_c(0)]) n_c I [ |S_{10}^{cd} \cup S_{01}^{cd}| > 0]\right]}{n_d}\Big\}, \end{align} $$

where $I[\cdot ]$ denotes an indicator function. Note that the bound $MAEI_d^w$ maintains SUTVA (Assumption 1).

Two elements change from $MAEI_d$ to $MAEI_d^w$ . First, the number of voters whose voting behavior may be affected by the experimental intervention increases to include all voters in a cluster. This follows from the fact that $|S_{10}^{cd} \cup S_{01}^{cd}| \leq n_c$ . Second, the expectation of untreated turnout, $E[a_c(0)]$ is now evaluated over all registered voters in a cluster (not just subjects). In the context of randomized saturation designs, $E[a_c(0)]$ does not change because the cluster is randomly sampled. Random sampling within a cluster is sufficient to ensure that $MAEI_d^w \geq MAEI_d$ . In other words, within-cluster interference increases the size of the possible electoral impact of an intervention. This analysis implies that if the only form of interference is within-cluster, we can construct a conservative bound on the aggregate impact of an experiment under SUTVA alone.

5.2 Between-Cluster Interference

I now to proceed to relax SUTVA, Assumption 1. In order to account for between-cluster interference, a violation of SUTVA, I introduce a vector of parameters $\pi _c \in [0,1]$ , indexed by cluster (c), to measure researchers’ ex ante beliefs about the proportion of voters that could respond to treatment (or some manifestation thereof) in clusters where allocation of the intervention is not changed by the experiment. In experiments in which the intervention would not occur absent the experiment, this term refers to the set of registered voters in control clusters:

(4) $$ \begin{align} \begin{aligned}MAEI_d^{bw} = \max\Big\{&\frac{\sum_{c \in C_d} \left[E[a_c(0)] n_c I[ |S_{10}^{cd} \cup S_{01}^{cd}|> 0] + E[a_c(0)] n_c \pi_c I[ |S_{10}^{cd} \cup S_{01}^{cd}| = 0]\right]}{n_d}, \\ &\frac{\sum_{c \in C_d} \left[(1-E[a_c(0)]) n_c I[ |S_{10}^{cd} \cup S_{01}^{cd}| > 0] + (1-E[a_c(0)]) n_c \pi_c I[ |S_{10}^{cd} \cup S_{01}^{cd}| = 0]\right]}{n_d}\Big\}. \end{aligned} \end{align} $$

The new term in the numerator of both expressions in (4) reflects the possible changes in turnout in clusters where no subjects’ assignment to the intervention is changed due to the experiment. Intuitively, because $\pi _c \geq 0$ , it must be the case that the aggregate electoral impact of experiments that experience between- and within-cluster interference is greater than those with only within-cluster interference, $MAEI_{d}^{wb} \geq MAEI_d^w$ .

Now, consider the implications of conservatively setting $\pi _c = 1$ for all c, akin to an assumption that an experiment could affect the voting behavior of all registered voters in a district. In this case, (4) simplifies to

(5) $$ \begin{align} \max \left\{\frac{\sum_{c \in C_d} E[a_c(0)]n_c}{n_d}, \frac{\sum_{c \in C_d} (1-E[a_c(0)])n_c}{n_d}\right\}. \end{align} $$

However, it must always be the case that the ex ante margin of victory, $\psi _d \leq \frac {1}{n_d} \sum _{c \in C_d} E[a_c(0)] n_c$ , as this represents the case in which the winning candidate wins every vote. It therefore must be the case that if $\pi _c = 1 \forall c$ , $\psi _d \leq 2MAEI_d^{bw}$ . In other words, without circumscribing spillovers in some way, the decision rule is never satisfied in a contested election. Thus, a researcher should never run an electoral experiment if she anticipates between-cluster spillover effects that could reach all voters, even absent considerations of identification and inference.

6.1 Relation to Existing Electoral Experiments

I examine the application of this framework to existing experiments in two ways. In Appendix A5 of the Supplementary Material, I use replication datasets, administrative, and archival data to apply the framework to four published experiments that comprise mobilization, information, and persuasion interventions: Boas, Hidalgo, and Melo (Reference Boas, Hidalgo and Melo2019), Gerber and Green (Reference Gerber and Green2000), Bond et al. (Reference Bond2012), and López-Moctezuma et al. (Reference López-Moctezuma, Wantchekon, Rubenson, Fujiwara and Lero2021). And below, I focus on back-of-the-envelope calculation of the $MAEI_d$ on 14 studies of information and accountability that are classified by Enríquez et al. (Reference Enríquez, Larreguy, Marshall and Simpser2019). To compare these experiments, I use information reported in papers and appendices without consulting replication or ancillary data. These calculations provide a survey of whether the information necessary to consider aggregate impact is reported, and what barriers to these calculations present. I report the studies, their relationship to the proposed framework, and my calculations in Supplementary Table A8. I lack any ex ante information about how to predict these races, so I focus only on the calculation of $MAEI_d$ under Assumptions 1 and 2.

Thirteen of the 14 studies intervene in multiple races (districts). I calculate the average $MAEI_d$ across districts. The average $MAEI_d$ is an abstraction from the decision rule described in this paper. However, for the purposes of examining the literature, it serves as a measure of the variation in possible electoral impact. I am only able to estimate the $MAEI_d$ in 6 of 14 studies, varying $E[a_c(0)]$ from its minimum of 0.5 (for all c) to its maximum of 1 (for all c). I present these estimates in Figure 1. The graph suggests that the degree to which existing information experiments could have moved electoral outcomes varies widely. These back-of-the-envelope calculations, in isolation, cannot assess whether an intervention was consistent with the decision rule advocated here due to lack of information on the predicted margin of victory. However, any average $MAEI_d> 0.5$ cannot pass the decision rule (in at least one district) because $2MAEI_d> 1$ . Supplementary Table A8 suggests that existing cluster-assigned information treatments tend to have high saturation within districts.

Figure 1 Estimated average $MAEI_d$ for six electoral experiments on electoral accountability. The interval estimates in the cluster-randomized experiments indicate the range of average $MAEI_d$ ’s for any $E[a_c(0)] \in [0.5, 1]$ .

The barriers to estimation of the $MAEI_d$ in the remaining eight studies are informative for how we think of electoral impact. In general, these studies do not provide information on how the experimental units relate (quantitatively) to the electorate as a whole. This occurs either because: units (voters or clusters) were not randomly sampled from the district (four studies) or because there is insufficient information about constituency size, $n_d$ (four studies). The takeaway from this survey of is simply that considerations of aggregate electoral impact require analyses that are not (yet) standard practice. The variation in Figure 1 further suggests that research designs vary substantially on this dimension and justify the considerations I forward.

6.2 Implementing the Framework

Simulating the design of experiments under the decision rule allows for an application of the full framework. The simulation below uses electoral data from the U.S. state of Colorado. It relies upon real voter registration data, precinct-to-district mappings, and election predictions. Because U.S. elections are administered at the state level, the simulations are greatly simplified by focusing on a single state in one election: the 2018 midterms. All races in 2018 were at the state level or below. In the simulations, I assume that an experimental intervention would not occur absent the researcher. I show that the method can be used with off-the-shelf predictions or with researcher-generated predictions. To illustrate the use of off-the-shelf predictions, I use the 2018 U.S. House forecast by Morris (Reference Morris2018). I generate my admittedly naïve predictions for Colorado State House seats in 2018 using (limited) available data, namely partisan voter registration data and lagged voting outcomes. I fit a basic predictive model on electoral data from 2012 to 2016 (three previous election cycles) and then predict outcomes for 2018 (see Appendix A7.3 of the Supplementary Material for details).

Examining only the predictive intervals, Figure 2 depicts the 90% predictive intervals for Colorado’s 65 State House and 7 U.S. House seats in 2018. The 90% predictive intervals provide a useful visualization because when they bound 0 (the gray intervals), no experiment can pass the decision rule proposed in this paper. In sum, the predictive intervals for 33/65 State House races and 2/7 U.S. House races bound 0.

Figure 2 Predictive intervals for 65 State House seats and 7 U.S. House seats. Gray intervals represent grounds for declining to conduct an experiment in a district under the decision rule proposed here.

I consider two research designs, each invoking SUTVA and, by design, satisfying Assumption 2.Footnote ⁵ I first consider experiments that assign individual voters (not clusters) to treatment. I show calculations based on three types of sampling of individuals into the experimental sample that vary the calculation of $E[a_c(0)]$ and thus $MAEI_d$ . A best-case scenario sets $E[a_c(0)] = \frac {1}{2}$ and represents the case in which participants were pre-screened to evenly fall on both sides of the ideological spectrum. A worst-case scenario sets $E[a_c(0)]=0$ and could represent the case in which all experimental subjects would vote in the same way absent treatment. This approximates a sample composed of only strong partisans. The intermediate case represented by “random sampling” predicts $E[a_c(0)]$ from 2016 district vote totals.

Figure 3 depicts the theoretical maximum number of individuals that could be assigned to an intervention in State House and U.S. House elections, by district and race. The shading represents the three sampling assumptions described above. Several features are worth note. First, the experimental allocation of treatment can only pass the decision rule in sufficiently extreme (thus predictable) electorates. Ranking districts from the most Republican to most Democratic (in terms of predicted vote margin) on the x-axis, the maximum number of individuals assigned to treatment is 0 in competitive races. The more lopsided the race, the more subjects can be assigned to treatment under the decision rule. Second, the type of experimental sample conditions the permissible treatment group size. However, going from worst to best case can double the number of subjects, as implied by (2). Third, comparing the top and bottom plots in the left column, in larger districts, the maximum number of registered voters that could be assigned to treatment grows proportionately to district size (see Supplementary Table A9 for summary statistics). Finally, when describing the maximum number of treated subjects as a proportion of the electorate, only sparse treatments are permissible under the decision rule. Nevertheless, it implies that one could easily allocate an individually randomized treatment in a way such to power an experiment under the decision rule proposed by this paper. Supplementary Figure A11 reports the results of an analogous cluster-randomized treatment at the precinct level revealing that the number of “treatable” precincts is quite small, particularly in the case of State House races.

Figure 3 Maximum number of individuals (left) or proportion of registered voters (right) that can be assigned to treatment under decision rule.