3.1 Design

We define the design space $\mathcal {Z} = \mathbf {Z}(\Omega )$ , where Z is a bijective random vector $\mathbf {Z}:\Omega \to \mathbb {R}^N$ . The design is defined as the joint p.m.f. of $\mathbf {Z}$ , $f: \mathbb {R}^N \to [0,1]$ . We may define all objects with respect to the design vector $\mathbf {Z}$ , removing all notational dependence on $\omega $ . We assume that the design is determined by the researcher, grounding our discussion away from concerns about the uncertainty in the assignment procedure. More generally, a valid design exists in settings where the researcher is simply aware of the probabilities implied by an assignment process and the sample space, allowing for design-based inference in observational settings. This setup and notation are, so far, fairly standard in the design-based setting. However, we will use this setup to define novel estimands in Section 4. The conceptual differences between the design space and the design will be central to the interpretation of these estimands as discussed in Section 5.

For illustration, we define the design with respect to our stylized examples. In the rebel group survey, the design is the sampling procedure that the researcher uses to determine the probability of surveying each of the historical rebel groups, which consists of randomly sorting a list of the 10 groups and surveying the first three in order. One way to formally express this procedure is as follows:

$$\begin{align*}f(\mathbf{z}) = \begin{cases} \frac{(10-3)!}{10!} & : \mathbf{z} \in {\textrm {Sym} }\,((1,2,3,0,0,0,0,0,0,0)) \\ 0 & : \text{ otherwise, } \end{cases}\end{align*}$$

where ${\textrm {Sym} }\,(\cdot )$ represents the set/orbit of distinct permutations of a vector. The corresponding design space is $\mathcal {Z} \supseteq \{(0,0,0,0,0,0,0,0,0,0)\} \cup {\textrm {Sym} }\,((1,0,0,0,0,0,0,0,0,0)) \cup {\textrm {Sym} }\,((1,2,0,0,0,0,0,0,0,0)) \cup \dots \cup {\textrm {Sym} }\,( (1,2,3,4,5,6,7,8,9,10))$ .Footnote ⁵

In our school experiment, the design is the procedure we use to determine the probability that each student receives the treatment arm. Letting $\lceil 0 .2 N \rceil $ be the number of observations in treatment, the design is

$$\begin{align*}f(\mathbf{z}) = \begin{cases} \frac{(\lceil 0.2 N \rceil)! (N- \lceil0 .2 N \rceil)! }{N!} &\!\! : \sum_{k=1}^N z_k = \lceil0 .2 N \rceil \\ 0 &\!\! : \text{otherwise,} \end{cases}\end{align*}$$

where the corresponding design space is $\mathcal {Z} \supseteq \{0,1 \}^N$ .

3.2 Population

We follow Särndal (Reference Särndal1978) in defining the population as the collection of units about which we wish to conduct inference. This finite population is indexed by $i = 1,\dots ,N$ , where indexing is fixed. Our estimands are defined over this population on specified outcomes of interest. Unless otherwise specified, generalizations are to estimands defined over this same population. We represent the outcomes of interest associated with these units with a real-valued random vector, $\mathbf {Y}$ . The random vector $\mathbf {Y}$ characterizes the raw potential outcomes, which can be represented for each individual i as

(1) $$ \begin{align} y_i(\mathbf{z}) = Y_i(\mathbf{Z}^{-1}(\mathbf{z})), \forall i \in 1,\dots ,N, \mathbf{z} \in \mathcal{Z}. \end{align} $$

The raw potential outcome $y_i(\mathbf {z})$ for individual i is what we observe if we perform intervention $\mathbf {z}$ and measure unit i’s outcome. There are a total of $N \times | \mathcal {Z}|$ raw potential outcomes in the causal setting. There are substantively defined raw potential outcomes in the survey sampling setting.Footnote ⁶

In our rebel group survey, the population is the 10 historical rebel groups and the outcome is the measured number of reported members. In our school experiment, the population is the N students, and the outcome is an indicator of whether a student volunteered.

3.3 Exposure mapping

While the raw potential outcome characterizes the outcome at the most granular level, we are often interested in further characterizing a unit of observation by the “exposure” that our intervention has placed them in. Examples of exposures in the experimental literature include social influence (Centola Reference Centola2010), spatial exposure (Miguel and Kremer Reference Miguel and Kremer2004) or simple direct exposure—what is most commonly considered in experimental settings under SUTVA. The exposure mapping relates interventions to the exposures (“treatments”) experienced by the units of observation. The exposure mapping was originally defined, named and applied to the design-based setting in Aronow and Samii (Reference Aronow and Samii2017). When exposure mappings are properly specified, that is, for each unit, a given exposure is associated with a single, stable value of the outcome, and if this is satisfied for all units, then exposures align with what Manski (Reference Manski2013) denotes as “effective treatments.”

Formally, our exposure mapping for individual i is a function $g_i: \mathcal {Z} \to \mathbb {R}$ . Since $g_i$ is a function, each realization of the intervention associates each individual with only one exposure, although this can be easily relaxed to allow for settings where an observation is in multiple exposures, such as in multi-factorial designs. Given that we have knowledge of the support of the design vector, ${\textrm {Supp} }\,{(\mathbf {Z})}$ , we can define a function $g_i| _{{\textrm {Supp} }\, (\mathbf {Z})} : {\textrm {Supp} }\,(\mathbf {Z}) \to \mathbb {R} $ . We denote the N-length random exposure vector

(2) $$ \begin{align} \mathbf{D} = (g_1(\mathbf{Z}), g_2(\mathbf{Z}),\dots ,g_N(\mathbf{Z})). \end{align} $$

We condition on the support of the design in $g_i| _{{\textrm {Supp} }\, (\mathbf {Z})}$ because while there may be some interventions in the design space, $\mathcal {Z}$ , in which the exposure is not well-defined, for estimation, these are only consequential if they occur with positive probability.

In our rebel group survey, we might code a group as exposed if they had been surveyed and not exposed if they were not. This exposure mapping can be expressed mathematically as

$$\begin{align*}g_i(\mathbf{z}) = \begin{cases} 1 & : z_i>0 \\ 0 & : \text{ otherwise. } \end{cases}, \forall i \in \{1,\dots ,10\}, \mathbf{z} \in {\textrm {Sym} }\,((1,2,3,0,0,0,0,0,0,0)). \end{align*}$$

In our school experiment, we could use an “individualistic” exposure mapping, in which we code an individual as exposed if they were assigned to the program, and not exposed otherwise,

$$\begin{align*}g_i(\mathbf{z}) = \begin{cases} 1 & : z_i = 1\\ 0 & : \text{ otherwise } \end{cases}, \forall i \in \{1,\dots ,N\}, \mathbf{z} \in \{0,1\}^N. \end{align*}$$

Alternatively, suppose the researcher also has some pre-treatment measure of a network among peers, which may deviate from the true network in its representation of the underlying data-generating process. Using this pre-treatment measure of the social network, we could define exposures based on whether adjacent nodes were treated (Aronow and Samii Reference Aronow and Samii2017; Sävje Reference Sävje2024). One example is

(3) $$ \begin{align}g_i(\mathbf{z}) =\begin{cases}0 \ \text{(Control)} & : \text{if } z_i = 0 \text{ and } \theta_i \mathbf{z} = 0, \\[2pt]1 \ \text{(Isolated Direct)} & : \text{if } z_i = 1 \text{ and } \theta_i \mathbf{z} = 0, \\[2pt]2 \ \text{(Indirect)} & : \text{if } z_i = 0 \text{ and } \theta_i \mathbf{z} > 0, \\[2pt]3 \ \text{(Direct}\ \text{& Indirect)} & : \text{if } z_i = 1 \text{ and } \theta_i \mathbf{z} > 0,\end{cases}\quad\forall i \in \{1,\ldots,N\},\ \mathbf{z} \in \{0,1\}^N .\end{align} $$

where $\theta _i$ is the ith row of the network adjacency matrix.

Figure 1 illustrates this mapping: the left panel shows the intervention (treated in red, control in gray); the right panel shows resulting exposures under Equation (3): control (gray), isolated direct (red), indirect (pink) and direct plus indirect (maroon).

Figure 1

Network exposure. The left panel shows the intervention (treated in red, control in gray); the right panel shows resulting exposures under Equation (3): control (gray), isolated direct (red), indirect (pink) and direct plus indirect (maroon).

Figure 1 Long description

The left panel, labeled Treatment Assignment, displays a network of nodes connected by lines. Nodes are colored either red for treated or gray for control. Red nodes are distributed throughout the network, with no apparent clustering. The right panel, labeled Exposure Conditions, shows the same network structure but with four node colors: gray for control, red for isolated direct exposure, pink for indirect exposure, and maroon for direct plus indirect exposure. Maroon nodes are concentrated where red nodes were previously adjacent, indicating overlap of direct and indirect exposure. Pink nodes appear around clusters of red and maroon, representing indirect exposure through network connections. The overall structure remains constant, but node color distribution shifts to reflect exposure categories as defined by Equation 3.

In Section 5, we present a toy example where the exposure mapping does not match the true underlying decision rule and discuss implications. Sävje (Reference Sävje2024) further delineates the use of the exposure mapping to define causal estimands and for making assumptions about our potential outcomes.

4.1 Expected potential outcome

The first target we consider is the EPO. First, we denote the marginal probability that unit i enters exposure d under the design distribution of $\mathbf {Z}$ as

(4) $$ \begin{align} \pi_i(d) = \left(\sum_{\mathbf{z} \in {\textrm {Supp} }\, (\mathbf{Z}): g_i(\mathbf{z}) = d} {\textrm{Pr}}\,[\mathbf{Z} = \mathbf{z}] \right)= {\textrm{Pr}}\,[D_i = d]. \end{align} $$

We impose a condition that all observations have positive probability under the design of falling into the relevant exposure condition.

Note that unlike in conventional approaches to inverse probability weighting, in which positivity is a population-level assumption used for estimation, here it is an individual-level assumption also used in defining the estimands. More specifically, in conventional inverse probability weighting, an individual may receive a treatment deterministically if, when drawing from the population, a given covariate profile has a non-zero probability of being observed under any treatment condition (Cole and Hernán Reference Cole and Hernán2008; Hernán and Robins Reference Hernán and Robins2006). In our setting, each individual that we conduct inference over must have a non-zero probability of being in the relevant exposure condition. While this may seem like a stringent assumption, in practice, researchers can always trim the population to a subset where the assumption is valid to arrive at interpretable functions of EPOs—although this changes the target population and therefore the estimand.

Given that relevant individual positivity assumptions hold, the EPO for unit i under exposure d is defined as

(5) $$ \begin{align} \begin{aligned} \overline{y}_i(d) & = \frac{\sum\limits_{\mathbf{z} \in {\textrm {Supp} }\, (\mathbf{Z}): g_i(\mathbf{z}) = d} y_i(\mathbf{z}) {\textrm{Pr}}\,[\mathbf{Z} = \mathbf{z}]}{\sum\limits_{\mathbf{z} \in {\textrm {Supp} }\, (\mathbf{Z}): g_i(\mathbf{z}) = d} {\textrm{Pr}}\,[\mathbf{Z} = \mathbf{z}]}\\ & = {\textrm E}\,[y_i(\mathbf{Z}) | g_i(\mathbf{Z}) = d]\\ & = {\textrm E}\,[y_i(\mathbf{Z}) | D_i = d]. \end{aligned} \end{align} $$

This can be interpreted as the individual outcome that we would see, in expectation, when we intervene using design vector $\mathbf {Z}$ to expose unit i to exposure d.Footnote ⁷ To avoid notational overload, these definitions implicitly hold fixed the exposure mapping g and design $\mathbf {Z}$ . They could be augmented (e.g., $\overline {y}_i(d;(g,\mathbf {Z}))$ ) to facilitate comparisons across different designs and exposure mappings.

Returning to our stylized examples, in our rebel group survey with an individualistic exposure mapping, the quantity $\overline {y}_3(d = 1)$ represents the number of reported historical members we would find on average when we survey the third rebel group using our design. While we do not typically think of surveys as requiring SUTVA-like assumptions, survey outcomes are not generally well-defined in their absence. Consider a quantity such as “the potential outcome of rebel group three, when it is surveyed.” Our previously introduced exposure mapping for the survey setting implies that the number of members reported for this group should be unaffected by the act of surveying other groups (no interference) and whether the group is surveyed first, second or third (no hidden treatment variations). In our school experiment, the quantity $\overline {y}_3(d = 1)$ represents the rate at which an individual student would volunteer when they are directly exposed in isolation, in expectation, under our design. In practice, we are rarely, if ever, interested in the EPO of any particular unit, but rather, we will use EPOs as building blocks to more interesting estimands.

4.2 Average expected potential outcome (AEPO)

Under individual positivity (Condition 4.1) for exposure d and all individuals $i \in \{1, \dots , N\}$ , we average the EPOs over the units in the population to define the AEPO as

(6) $$ \begin{align} \overline{y}(d) = \frac{1}{N} \sum_{i=1}^N \overline{y}_i(d) = \frac{1}{N} \sum_{i = 1}^N {\textrm E}\, [y_i(\mathbf{Z}) |D_i = d]. \end{align} $$

This is akin to asking: averaging over all units, what is the outcome that we would see, in expectation, for each respective unit when we intervene using design $\mathbf {Z}$ to expose that unit to d? In the survey setting, the AEPO of the surveyed is of particular interest, because it amounts to asking: averaging over all units, what is the outcome that we would see, in expectation, for each respective unit when they were sampled under the design $\mathbf {Z}$ ?

4.3 Expected exposure difference (EED)

Similarly, causal “contrasts” can be defined. For an individual observation i, assuming that Condition 4.1 holds for exposures d and $d'$ for all individuals $i \in \{1, \dots , N\}$ , we can define the EED between d and $d'$ as

(7) $$ \begin{align} \tau_i (d, d') = \overline{y}_i(d) - \overline{y}_i (d'). \end{align} $$

This can be interpreted as the difference in individual outcomes that we would see, in expectation, when we intervene using the design vector $\mathbf {Z}$ to expose unit i to exposure d as compared to $d'$ .Footnote ⁸

4.4 Average expected exposure difference (AEED)

Just as we averaged the EPO to get the AEPO, we can average the EED to get the AEED. The AEED between d and $d'$ is

(8) $$ \begin{align} \tau(d, d') = \overline{y}(d) - \overline{y}(d') = \frac{1}{N} \sum_{i = 1}^N \tau_i(d, d'). \end{align} $$

We can also define a causal contrast in our survey setting where the AEED is between surveyed and unsurveyed groups. However, if we do not survey a group, we do not find any former members, so the causal contrast, while defined, is not empirically tractable.

5.1 Possible assumptions

First, we consider a new assumption that we call NURVA.

Condition 5.1 NURVA

$$\begin{align*}g_i(\mathbf{z}) = g_i(\mathbf{z}') \implies y_i(\mathbf{z}) = y_i(\mathbf{z}') = \overline{y}_i(g_i(\mathbf{z})), \forall \mathbf{z}, \mathbf{z}' \in {\textrm {Supp} }\, (\mathbf{Z}). \end{align*}$$

In contrast to the setting where we make no additional assumptions, under NURVA, each EPO is no longer composed of weighted averages of different possible raw potential outcomes, but instead consists of a unique and common raw potential outcome realized under any assignment $\mathbf {z} \in {\textrm {Supp} }\, (\mathbf {Z})$ that results in the individual being exposed to d. There is no unmodeled interference that could possibly be seen, even if we were able to observe the raw potential outcomes under every random assignment associated with the design. Under NURVA, the EED can be thought of as, “If I were to use my design to induce exposure d instead of exposure $d'$ on that unit, how much different would the result be?” We can drop the reliance on expectations over the design.

Second, we consider the more familiar assumption of Rubin’s SUTVA, which makes an assumption like NURVA, but does so across all feasible interventions.

Condition 5.2 SUTVA

$$\begin{align*}g_i(\mathbf{z}) = g_i(\mathbf{z}') \implies y_i(\mathbf{z}) = y_i(\mathbf{z}') = \overline{y}_i(g_i(\mathbf{z})), \forall \mathbf{z}, \mathbf{z}' \in \mathcal{Z}.\end{align*}$$

The distinction between SUTVA and NURVA hinges on how the design space of all feasible interventions is defined. In policy analysis, the inferential scope that a researcher is interested in may cover treatment or sampling interventions that go beyond those that can be assigned within a given experiment or sampling procedure, and so $\mathcal {Z}$ may include assignments outside the design. Under SUTVA, the EED can be interpreted as: “If I induce exposure d instead of exposure $d'$ on that unit, how much different would the result be?” We can now also drop any qualifiers related to the design. This interpretation now fully coincides with that of the (average) treatment effect as it is conventionally used. For a given design, NURVA and SUTVA are observationally equivalent: NURVA embeds all observable implications of SUTVA, as all $\mathbf {z} \in \mathcal {Z} \setminus {\textrm {Supp} }\,(\mathbf {Z})$ remain hidden under the experimental design. Hence, the gap is in scope.

5.2 Toy example: NURVA holds but SUTVA does not

Consider a household with two persons. We flip a fair coin: heads treats only Person one; tails treats only Person two. We observe whether each votes. The exposure mapping is individualistic, $g_i(\mathbf z) = z_i$ . The true (unknown) rule is that Person one votes $\iff $ Person two is treated, and Person two votes $\iff $ Person one is treated.

Table 1a shows revealable information. Under this design, the AEPO under treatment is $\overline {y}(1) = 0$ and under control is $\overline {y}(0) = 1$ , so $\tau (1,0) = -1$ . Given the design and unknown rule, the untreated person votes and the treated does not. Table 1b lists all raw potential outcomes in $\mathcal {Z} = \{ 0, 1 \}^2$ , including two interventions not revealable under the design.

Here, SUTVA fails but NURVA holds. Consider interventions $(1,0)$ and $(1,1)$ , where the latter is off-support in Table 1. Under the individualistic exposure mapping for Person one, these exposures are equivalent, $g_1(\mathbf {z} = (1,0)) = g_1(\mathbf {z} = (1,1)) = 1$ . However, the outcomes differ, $0 = y_1(\mathbf {z} = (1,0)) \neq y_1(\mathbf {z} = (1,1)) = 1 $ . Therefore, SUTVA does not hold. NURVA, unaffected by off-support instability, trivially holds.

Table 1

NURVA holds but SUTVA does not

Table 1 Long description

The top panel, labeled revealable information, has five columns: z, d, y one of z, y two of z, and probability of Z equals z. The first row lists z as (0,1), d as (0,1), y one as 1, y two as 0, and probability as 0.5. The second row lists z as (1,0), d as (1,0), y one as 0, y two as 1, and probability as 0.5. The third row is empty. The fourth row, spanning all columns, shows a rightward arrow followed by tau open parenthesis 1,0 close parenthesis equals negative one. The bottom panel, labeled feasible information, has the same five columns. The first row lists z as (0,0), d as (0,0), y one as 0, y two as 0, and probability as 0. The second row lists z as (0,1), d as (0,1), y one as 1, y two as 0, and probability as 0.5. The third row lists z as (1,0), d as (1,0), y one as 0, y two as 1, and probability as 0.5. The fourth row lists z as (1,1), d as (1,1), y one as 1, y two as 1, and probability as 0.

We stress that NURVA is an assumption that depends on the design, unlike SUTVA, which is concerned with the entirety of the design space. Using the same design space, we illustrate this with two examples of different designs implying different AEEDs in Table 2, which are both also different from the original AEED in Table 1a.

Table 2

Different designs yield different AEEDs

Table 2 Long description

The first table section, labeled feasible information, has five columns: z, d, y sub 1 of z, y sub 2 of z, and probability Pr of Z equals z. The column headers are left to right. Four data rows follow: for z and d equal to open parenthesis 0 comma 0 close parenthesis, y sub 1 and y sub 2 are 0, probability is 0.5; for open parenthesis 0 comma 1 close parenthesis, y sub 1 is 1, y sub 2 is 0, probability is 0; for open parenthesis 1 comma 0 close parenthesis, y sub 1 is 0, y sub 2 is 1, probability is 0; for open parenthesis 1 comma 1 close parenthesis, y sub 1 and y sub 2 are 1, probability is 0.5. The summary row at the bottom states tau of open parenthesis 1 comma 0 close parenthesis equals 1. The second table section, labeled alternate design 2, repeats the same column structure. Four data rows: for z and d equal to open parenthesis 0 comma 0 close parenthesis, y sub 1 and y sub 2 are 0, probability is 0.25; for open parenthesis 0 comma 1 close parenthesis, y sub 1 is 1, y sub 2 is 0, probability is 0.25; for open parenthesis 1 comma 0 close parenthesis, y sub 1 is 0, y sub 2 is 1, probability is 0.25; for open parenthesis 1 comma 1 close parenthesis, y sub 1 and y sub 2 are 1, probability is 0.25. The summary row at the bottom states tau of open parenthesis 1 comma 0 close parenthesis equals 0.

While our original, individualistic exposure mapping $d_i = g_i(\mathbf {z}) = z_i$ does not satisfy SUTVA, consider the exposure mapping, $d_i = g_i(\mathbf {z}) = z_{3-i}$ , which parallels the true decision rule (i.e., an individual is exposed if and only if their housemate is treated). Now the AEED is 1, regardless of what probabilities we assign to each feasible assignment (provided the relevant individual positivity conditions hold; illustrated in Table 3). Informally, SUTVA has been “restored” with respect to the exposure and we no longer have issues in characterizing the causal effect of exposing a unit on its outcome.

Table 3

Exposure mapping that satisfies SUTVA

Table 3 Long description

Panel (a), titled Alternate design 1, has columns labeled z, d, y sub 1 of z, y sub 2 of z, and probability that Z equals z. The first row lists (0,0), (0,0), 0, 0, 0.5. The second row lists (0,1), (1,0), 1, 0, 0. The third row lists (1,0), (0,1), 0, 1, 0. The fourth row lists (1,1), (1,1), 1, 1, 0.5. The final row spans all columns and shows tau of (1,0) equals 1. Panel (b), titled Alternate design 2, has the same column labels. The first row lists (0,0), (0,0), 0, 0, 0.25. The second row lists (0,1), (1,0), 1, 0, 0.25. The third row lists (1,0), (0,1), 0, 1, 0.25. The fourth row lists (1,1), (1,1), 1, 1, 0.25. The final row spans all columns and shows tau of (1,0) equals 1.

Whether each stability assumption holds depends on the outcome of interest, how exposures are defined, the design used to induce exposures, and the real, underlying structure of the outcome generating process. For example, consider a setting where there is partial interference on a given outcome. That is, there may be arbitrary interference within, but not across mutually exclusive clusters of units. In such a setting, NURVA will hold with respect to that outcome on all designs where all units in a cluster are assigned to a single exposure level, and the AEED will be equivalent to the global average treatment effect (the effect of treating all subjects relative to no subjects). However, NURVA will not necessarily hold for designs that result in within-cluster variation in treatment assignment.Footnote ⁹ We address considerations for cross-design generalization further in Section 6.

Because NURVA embeds all observable implications of SUTVA, the appropriate interpretation in a given setting should be determined by the researcher. The structure of $g(\cdot )$ is a modeling choice rather than a direct implication of the design and may be based on theory or other knowledge of the underlying data-generating processes. We also note that “coarsened” exposures—which do not represent unique potential outcomes—induce design-dependent version weights. If the researcher is concerned about hidden versions within a potentially coarse exposure, they may refine g to include the plausible versions, or they may report features of the induced version (e.g., the distribution of treated-neighbor counts among “indirectly exposed”). Additionally, when comparing across studies, researchers may wish to standardize to a common target version distribution—if such version-specific effects are estimable. To gain prior information about interference structure itself, researchers might use design and analytic recommendations in, for example, Bowers et al. (Reference Bowers, Desmarais, Frederickson, Ichino, Lee and Wang2018) and Baird et al. (Reference Baird, Aislinn Bohren, McIntosh and Özler2018).

Practically, the NURVA/SUTVA distinction changes not the estimator used within a study, but the claim attached to it. In the rebel-group survey, the analyst must specify the implemented assignment mechanism and its support (e.g., surveying 3 of 10 groups in random order), choose and justify $g(\cdot )$ (e.g., “surveyed”/“not” versus order- or enumerator-specific exposures) and state the set of feasible interventions relevant to the substantive question. Invoking NURVA means that $\tau (d,d')$ is interpreted as an effect of the researcher-defined exposure under the realized protocol, on ${\textrm {Supp} }\,(\mathbf {Z})$ . Invoking SUTVA adds the stronger claim that the same contrast is invariant to off-support allocations in the feasible set, a claim not implied by the design alone. Accordingly, applied reports should make $\mathbf {Z}$ , ${\textrm {Supp} }\,(\mathbf {Z})$ , $g(\cdot )$ , and inferential scope explicit, and, when feasible, probe fragility by refining $g(\cdot )$ within the design.

6.1 Generalizing across designs: General equilibrium

It is sometimes the case that we are interested in the effect of treating everyone versus treating no one, but what we have in practice is a design where some units are randomly assigned to each condition. Table 4 constructs two schedules of raw potential outcomes. In both cases, the AEED from the observed design, where ${\textrm {Supp} }\,(\mathbf {Z}) = \{(1,0), (0,1)\}$ , does not reflect the AEED in which treatment is delivered uniformly over the population, where ${\textrm {Supp} }\,(\mathbf {Z}') = \{(1,1), (0,0)\}$ .

Table 4

General equilibria

Table 4 Long description

Panel (a), titled Attenuation under uniform rollout, has five columns: z, d, y sub 1 of z, y sub 2 of z, and probability of Z equals z. The four rows list: (0,0), (0,0), 0.5, 0.5, 0; (0,1), (0,1), 0, 1, 0.5; (1,0), (1,0), 1, 0, 0.5; (1,1), (1,1), 0.5, 0.5, 0. At the bottom, two summary equations are shown: tau sub Z of (1,0) equals 1; tau sub Z prime of (1,0) equals 0. Panel (b), titled Amplification under uniform rollout, has the same column structure. The four rows list: (0,0), (0,0), 0, 0, 0; (0,1), (0,1), 0, 0, 0.5; (1,0), (1,0), 0, 0, 0.5; (1,1), (1,1), 1, 1, 0. At the bottom, two summary equations are shown: tau sub Z of (1,0) equals 0; tau sub Z prime of (1,0) equals 1.

These schedules capture mechanisms in which the effect of “being treated” changes as coverage moves from partial to universal—for example, job training that is advantageous only when few workers are trained or vaccinations whose effectiveness depends on population coverage reaching a critical threshold. The instability of AEEDs for this limited notion of generalization across designs with a fixed set of experimental units raises concerns for external validity—generalization to units outside our original experimental sample, as the term is commonly used in political science. The limitations of external validity under NURVA alone can be seen by expanding the table of raw potential outcomes with new units. Alternatively, we can explicitly embed a sampling–treatment design under this framework, which we do below.

6.2 Hidden treatment variations

Both SUTVA and NURVA require an assumption that potential outcomes are stable under the researcher-specified exposures but NURVA requires this stability only on the support of the design, while SUTVA requires it under all feasible designs. When SUTVA is violated, even if NURVA holds, design-specific AEEDs need not generalize to other designs.

To illustrate this, consider an experiment where the design space is $\mathcal {Z} \subseteq \{0,1,2\}^N$ , and the exposure mapping is

$$\begin{align*}g_i(\mathbf{z}) = \begin{cases} 1 & : z_i> 0 \\ 0 & : \text{ otherwise } \end{cases}, \forall i \in \{1, \dots, N\}, \mathbf{z} \in \{0,1,2\}^N. \end{align*}$$

For this example, assume that there is no interference in the sense that $y_i(\mathbf {z}) = y_i(\mathbf {z}')$ if $z_i = z^{\prime }_i$ . This allows us to focus on the implications of hidden treatment variations in isolation. If treatments 1 and 2 are associated with different potential outcomes, then the AEED could vary between designs that do and do not sample treatment 2, as in Table 5. (For simplicity, Table 5 displays only assignments in ${\textrm {Supp} }\,(\mathbf {Z})$ and ${\textrm {Supp} }\,(\mathbf {Z}')$ ; all other feasible assignments have probability zero under the respective designs.)

Table 5

Hidden treatment variations

Table 5 Long description

Panel (a) is titled ‘Design Z: assigns only in 0, 1’. The header row lists z, d, y sub 1 of z, y sub 2 of z, and probability that Z equals z. The first data row shows z equals open parenthesis 0, 1 close parenthesis, d equals open parenthesis 0, 1 close parenthesis, y sub 1 of z equals 0, y sub 2 of z equals 1, probability equals 0.5. The second row shows z equals open parenthesis 1, 0 close parenthesis, d equals open parenthesis 1, 0 close parenthesis, y sub 1 of z equals 1, y sub 2 of z equals 0, probability equals 0.5. The final row displays an arrow pointing to tau sub Z of open parenthesis 1, 0 close parenthesis equals 1. Panel (b) is titled ‘Design Z prime: assigns only in 0, 2’. The header row lists z, d, y sub 1 of z, y sub 2 of z, and probability that Z prime equals z. The first data row shows z equals open parenthesis 0, 2 close parenthesis, d equals open parenthesis 0, 1 close parenthesis, y sub 1 of z equals 0, y sub 2 of z equals 2, probability equals 0.5. The second row shows z equals open parenthesis 2, 0 close parenthesis, d equals open parenthesis 1, 0 close parenthesis, y sub 1 of z equals 2, y sub 2 of z equals 0, probability equals 0.5. The final row displays an arrow pointing to tau sub Z prime of open parenthesis 1, 0 close parenthesis equals 2.

In the rebel group survey, different question wordings or enumerator identities may elicit different reports, yet all such surveys are coded as “surveyed.” In the school experiment, untreated students with one or several treated friends are both coded as “indirectly treated.” When these versions are associated with different potential outcomes, the AEED under a given design depends on how that design weights these version-specific potential outcomes.

6.3 Generalizing across sampling–treatment designs

We now sketch how the framework extends to joint sampling–treatment designs, linking causal and sampling inference. Within a finite population of size N, consider a design $\mathbf {Z}$ that first draws a simple random sample of size n without replacement and then, among sampled units, assigns k to treatment and $n-k$ to control by complete randomization. Define an exposure mapping where $z_i = 2$ indicates that unit i is sampled and treated, $z_i = 1$ that it is sampled and untreated and $z_i = 0$ that it is not sampled. We can define the AEED, $\tau _{\mathbf {Z}}(2,1)$ which, under NURVA, admits a causal interpretation as the effect, under this joint sampling-assignment design, of exposing a unit to being “sampled and treated” rather than “sampled and untreated.”

Now consider a different design $\mathbf {Z}'$ on the same population, with sample size $n'$ and number treated $k'$ . Using the same exposure mapping, define the corresponding AEED $\tau _{Z'}(2,1)$ .Footnote ¹⁰

We define the exposure mapping as

$$\begin{align*}g_i(z) = \begin{cases} 2 & : z_i = 2\\ 1 & : z_i = 1\\ 0 & : z_i = 0 \end{cases}, \forall i \in \{1,\ldots ,N\}, \mathbf{z} \in \mathcal{Z}. \end{align*}$$

Even if NURVA holds for both $\mathbf {Z}$ and $\mathbf {Z}'$ , there is no guarantee that $\tau _{\mathbf {Z}}(2,1)$ will equal $\tau _{\mathbf {Z}'}(2,1)$ : changing either n to $n'$ or k to $k'$ changes the distribution of other units’ sampling or treatment statuses conditional on $D_i=d$ , and hence may change the EPOs $y_i(d)$ that enter the AEPOs. Learning what would happen under alternative values of $(n,k)$ or in alternative target populations requires additional stability assumptions on how $y_i(d)$ varies with the sampling and treatment scheme, as in the broader generalizability and transportability literature. For example, Tipton (Reference Tipton2013) imposes dual SUTVA requirements for both treatment assignment (SUTVA( $\mathcal {S}$ )) and sample selection (SUTVA( $\mathcal {P}$ )), and Lesko et al. (Reference Lesko, Buchanan, Westreich, Edwards, Hudgens and Cole2017) assume that the distribution of treatment versions and, under interference, the pattern of interference are the same in the study sample and the target population. Because assignments in $\mathcal {Z} \setminus {\textrm {Supp} }\,(\mathbf {Z})$ are never observed under a single fixed design, off-support invariance is not nonparametrically testable within that design. Evidence about SUTVA must therefore come from additional designs or auxiliary structure that place competing versions on support—for example, saturation designs that vary treated fractions, multi-version interventions, repeated implementations that vary protocol features, such as wording or enumerators, or observational settings with known assignment probabilities over a wider support. Such evidence can make off-support invariance more or less credible, but it is an additional empirical argument, not a consequence of the original design.