2.1 Formalizing Preferences in Multi-Attribute Elections

Consider a paired-profile, forced-choice conjoint experiment, where each respondent $i \in \{1,\ldots ,N\}$ completes K tasks in which the respondent casts hypothetical votes between two candidates varying across L attributes. Each of the L attributes takes on $D_l$ discrete levels, respectively, such that $l\in \{1, \ldots , L\}$ . One can view this design as a simulation of a two-candidate election in which citizens vote for one of two candidates varying across L observed attributes.

As an illustration, consider a toy example in which candidates are characterized by three binary attributes (i.e., $L=3, D_1=D_2=D_3=2$ ). We label these attributes A, B, and C, respectively, and denote their levels by $0$ and $1$ , such that $A\in \{0,1\}, B\in \{0,1\}$ , and $C\in \{0,1\}$ . These values then fully characterize the election’s candidates. We use $[abc]$ to denote a candidate (or conjoint profile) whose values on these attributes are such that $A=a$ , $B=b$ , and $C=c$ . There are $2^3 = 8$ possible unique candidates, that is, $[000]$ , $[001]$ , $[010]$ , $[011]$ , $[100]$ , $[101]$ , $[110]$ , and $[111]$ . More generally, there are $\prod _{l=1}^{L} D_l$ possible unique candidates.

Given a choice where alternatives are characterized by multiple attributes, a natural formalization of individual preferences is to consider a preference ordering over the full set of possible unique attribute combinations. Namely, we define each voter’s preferences to be binary relations over the set of possible unique candidates. To simplify exposition, we assume that each voter has a strict preference ordering over all $\prod _{l=1}^{L} D_l$ unique candidates. For example, consider the “Type 1” voter in Table 1. In the table, the eight possible candidates (defined in columns 2–4) are ordered from top to bottom according to the Type 1 voter’s preference ranking (column 5). This preference can also be represented using standard decision-theoretic notation, such that $[111]\succ [110]\succ [101]\succ [100]\succ [011]\succ [010]\succ [001]\succ [000]$ .

Table 1 Candidate preference rankings for three voter types.

Note: This table shows the preference ranking for three voter types over candidate profiles defined by binary attributes A, B, and C.

2.2 Defining the AMCE

Since elections are a means of preference aggregation, electoral researchers may ask how one can learn about collective decisions from individual preferences expressed through conjoint experiments. It is fruitful to begin with several desirable criteria for this objective. First, it would be valuable to have an aggregate measure that captures the multidimensionality of the typical electoral choice, in which voters choose between candidates differing across many dimensions simultaneously. Second, the measure should map onto a meaningful empirical phenomenon of interest, such that electoral researchers can make causal or predictive inferences about elections using it. Finally, the measure should be empirically tractable, in the sense that researchers can use observed data from conjoint experiments to estimate it with sufficient precision and ideally without strong modeling assumptions.

The AMCE is one quantity of interest in electoral studies that meets all three criteria. First, the AMCE aggregates preference orderings over all possible profiles in a systematic manner that accounts for the multidimensional nature of the electoral decision problem by incorporating the directionality and intensity of preferences. Second, the AMCE directly represents the causal effect of a particular attribute on a candidate’s expected vote share, which a literature review reveals as the most prominent quantity of interest for electoral scholars. Third, identification and unbiased estimation of the AMCE can proceed under a limited set of assumptions and via straightforward nonparametric methods (Hainmueller et al. Reference Hainmueller, Hopkins and Yamamoto2014). The remainder of this section highlights each feature.

To define the AMCE under the current setup, let $Y_i([abc],[a'b'c']) \in \{0,1\}$ denote voter i’s potential outcome given a paired forced-choice contest between profiles $[abc]$ and $[a'b'c']$ . The potential outcome equals $1$ if respondent i would choose the first candidate (i.e., $[abc]$ ) given the choice task, which would occur if and only if $[abc]\succ [a'b'c']$ for that respondent, and $Y_i([abc],[a'b'c']) = 0$ otherwise (i.e., if $[a'b'c']\succ [abc]$ ). Then, the AMCE for attribute A is defined as the expected difference between the potential outcomes for all paired contests where the first candidate’s attribute A equals $1$ and the potential outcomes for all contests where the first candidate’s attribute A equals 0, given a known, prespecified distribution of the other attributes. So, without loss of generality, the AMCE for attribute A is

(1) $$ \begin{align} \textit{AMCE}_{A} \ \equiv \ \mathbb{E}\big[Y_i([1BC],[A'B'C']) - Y_i([0BC],[A'B'C'])\big], \end{align} $$

where the expectation is defined over the joint distribution of candidate attributes from which all attributes other than A for the first candidate (i.e., $B,C,A',B'$ , and $C'$ ) are drawn, and the sampling distribution for the N respondents from the target population of voters. The AMCEs for attributes B and C are defined analogously, and the definition extends to conjoint designs with more attributes, attributes with more than two levels, non-forced-choice outcomes, and tasks with more than two profiles, with appropriate notational changes.

A few remarks are useful to highlight in particular what the AMCE does not represent, and thereby avoid confusion among researchers. First, note that the relevant contrast for the AMCE is between attribute $A = 1$ for the first profile and $A=0$ for the same profile, both against another profile randomly drawn from the prespecified distribution. Suppose attribute A is gender, such that $A=1$ means a female candidate and $A=0$ means a male candidate. Then, $\textit{AMCE}_{A}$ compares the probability of a female candidate profile chosen against another randomly generated profile (whether male or female) to the probability of a male profile chosen against a similarly generated profile. That is, the AMCE asks how much better or worse a randomly selected candidate would fare if gender switches from male to female. Specifically, the AMCE is not the probability of a female candidate being chosen against a randomly generated male candidate. This difference has been a point of confusion in some applied work.

Second, the AMCE aggregates individual preferences with respect to two dimensions: across attributes and voters. Specifically, the AMCE employs averaging of individual preferences both across the distributions of possible candidates and voters in the target population. This is in contrast to other means of preference aggregation. For instance, Abramson et al. (Reference Abramson, Koçak and Magazinnik2021) show that as a result of the way in which the AMCE aggregates preferences, the AMCE does not reflect a simple majority preference (i.e., a positive AMCE does not necessarily mean a majority of voters prefer the attribute level in question). This is similar to how a positive sign of a standard ATE also does not necessarily mean that a majority of units have positive individual-level treatment effects. However, while Abramson et al. (Reference Abramson, Koçak and Magazinnik2021) conclude from this that the AMCE is largely uninformative with respect to questions of interest to political scientists, in the following sections, we highlight how the AMCE’s properties actually make it an invaluable tool for studying voter choice and election outcomes.

2.3 The AMCE and Preference Intensity

The first of the AMCE’s desirable properties for studying voter choice and election outcomes is that it captures the multidimensionality of the conjoint choice task. It does so by incorporating both the direction and intensity of preferences about individual attributes through aggregating the probabilities of profiles winning pairwise comparisons.

2.3.1 The Simplest Case: The Independent Uniform Attribute Distribution

In the simplest setup with attributes distributed independently and uniformly, the AMCE’s aggregation of winning probabilities reduces to the simple averaging of the profile ranks. Continuing with the three-attribute example, let $r_i(a,b,c) \in \{1,\ldots ,8\}$ represent the rank of profile $[abc]$ for voter i. Then, consider a voter’s average rank for the profiles that contain a particular attribute level such that the average rank of $A=a$ for voter i is defined as $S_{i}^A(a) \equiv \frac {1}{4}\sum _{b\in \{0,1\}}\sum _{c\in \{0,1\}} r_i(a,b,c)$ . Comparing a voter’s average ranks with respect to different levels of an attribute (e.g., $S_i^A(1)$ vs. $S_i^A(0)$ ) captures the directionality and intensity of her preferences with respect to the attribute.

For example, consider the Type 1 voter in Table 1. Intuitively, this voter strongly favors $A=1$ to $A=0$ because profiles containing $A=1$ are more highly ranked than profiles containing $A=0$ irrespective of other attributes. For attribute B, the voter favors profiles with $B=1$ to those with ${B=0}$ , but only if the profiles are not better in terms of A. As for C, the voter generally likes profiles with $C=1$ better than those with $C=0$ , but C’s value only influences the final ranking when the profiles are tied on other attributes. Thus, we can summarize these preferences as an intense preference for $A=1$ over $A=0$ , a moderate preference for $B=1$ over $B=0$ , and a mild preference for $C=1$ over $C=0$ .

Considering the average profile ranks across different attribute levels captures these intuitions. For illustration, Table 2 provides the Type 1 voter’s average ranks. The average rank for a Type 1 voter i of $A=1$ , $S_i^{A}(1)$ , is equal to $2.5$ , whereas the average rank of $A=0$ is $6.5$ . This implies that the voter prefers $A=1$ to $A=0$ . Similarly, $S_i^B(1) = 3.5$ and $S_i^B(0)=5.5$ , implying $B=1$ is preferred to $B=0$ . Likewise, $S_i^C(1)=4$ and $S_i^C(0)=5$ , so that $C=1$ is preferred to $C=0$ . The relative values of the rank means provide a natural metric for the intensity of the voter’s preferences: for attributes A, B, and C, the rank means are $2.5$ versus $6.5$ (intense preference), $3.5$ versus $5.5$ (moderate preference), and $4$ versus $5$ (mild preference), respectively. Incorporating these differences in preference intensity is key for capturing the attributes’ importance for the resulting vote choices in contests between multidimensional profiles.

Table 2 Average ranks by attribute.

Notes: This table shows the average ranks for candidate profiles with and without a given attribute for three voter types. Type 1 voters have an intense preference for A, moderate preference for B, and mild preference for C. Type 2 voters have a mild preference for not A, moderate preference for B, and intense preference for C. Type 3 voters have a mild preference for not A, moderate preference for B, and intense preference for not C.

The AMCE can, in fact, be shown to be directly related to these average rankings. Using a difference between the average ranks as a measure of the extent to which a voter prefers a particular attribute level over the other (e.g., $S^A_i(1) - S^A_i(0)$ ), one can further quantify the aggregate preference for $A=1$ over $A=0$ across all voters by taking the average value of $S^A_i(1) - S^A_i(0)$ across $i\in \{1,\ldots ,N\}$ , which we denote by $\bar S^A(1) - \bar S^A(0)$ . Under a uniform joint distribution of all the attributes, the AMCE for $A=1$ relative to $A=0$ is proportional to $\bar S^A(1) - \bar S^A(0)$ , such that $\textit{AMCE}_{A} \propto \bar S^A(1) - \bar S^A(0)$ as defined in Equation (1). Seen this way, it is clear that the AMCE represents an aggregation of individual preferences that explicitly accounts for intensity: $S^A_i(1) - S^A_i(0)$ represents an individual voter’s relative preference intensity for $A = 1$ over $A = 0$ , and $\bar S^A(1) - \bar S^A(0)$ averages this across voters.

2.3.2 The General Case: Arbitrary Attribute Distributions

The above result largely reproduces a main result in Abramson et al. (Reference Abramson, Koçak and Magazinnik2021), who prove the proportionality between the AMCE and differences in Borda scores in the special case that attributes are uniformly and independently distributed. Here, we further expand upon this by contributing the proof of a more general result. Namely, we show that the AMCE of an attribute is proportional to what we call the expected Borda score (EBS) of that attribute under an arbitrary, potentially nonuniform and nonindependent randomization distribution of attributes, provided that profiles are independently drawn within pairs. While the Borda score of a profile equals the number of times the profile is chosen against all possible alternative profiles, the EBS represents the expected number of times a profile is chosen against Q independently randomly drawn profiles, thereby generalizing the Borda score to choice settings that are probabilistic rather than deterministic.

The generalization to an arbitrary randomization distribution is important for several reasons. First, theoretically, our probabilistic generalization builds a bridge from the original deterministic, choice-theoretic result of Abramson et al. (Reference Abramson, Koçak and Magazinnik2021) to the statistical analysis of randomized experiments and treatment effects, the latter of which has been the predominant framework for analyzing conjoint survey designs in the recent literature (e.g., De la Cuesta et al. Reference De la Cuesta, Egami and Imai2021; Hainmueller et al. Reference Hainmueller, Hopkins and Yamamoto2014; Leeper et al. Reference Leeper, Hobolt and Tilley2020). Second, recent methodological research has formally highlighted the importance of carefully choosing the randomization distribution of attributes and made proposals (and recommendations on the process) for deviating from uniform, independent distributions (De la Cuesta et al. Reference De la Cuesta, Egami and Imai2021; Ganter Reference Ganter2020). Third, partially prompted by these recommendations, a number of applied researchers adopt conjoint designs that utilize nonuniform, nonindependent attribute distributions for the purpose of realism and external validity (e.g., Huff and Kertzer Reference Huff and Kertzer2018; Leeper and Robison Reference Leeper and Robison2020).

To consider the general case of a paired conjoint with an arbitrary attribute distribution, we denote the jth profile in respondent i’s kth task by $X_{ijk} \equiv [X_{ijkl}, \tilde {X}_{ijk}] \in \mathcal {X}$ , $j \in \{1,2\}, k\in \{1,\ldots ,K\}, l\in \{1,\ldots ,L\}$ , where $X_{ijkl}$ represents the attribute of interest l and $\tilde {X}_{ijk}$ the collection of the remaining $L-1$ attributes. Then, the general potential outcome with respect to profile pair $[x_1, x_2]$ for respondent i can be defined as $Y_i(x_1, x_2) = Y_i(x, \tilde {x}_1, x_2)$ , where $x_1 = [x, \tilde {x}_1]$ , representing the respondent’s choice in a conjoint task comparing $x_1$ against $x_2$ . Under the standard consistency assumption, the observed outcome for respondent i’s kth task can be written as $Y_{ik} = Y_i(X_{i1k}, X_{i2k})$ .

Next, we introduce preference relations for the profiles. Let $Q \equiv |\mathcal {X}|$ , the number of unique profiles. For each respondent i, we index elements of $\mathcal {X}$ in the order of their preference, such that $\mathcal {X} = \{x_{(1i)}, \ldots , x_{(mi)}, \ldots , x_{(Qi)}\}$ , where $x_{(mi)} \succ x_{(m'i)}$ if and only if $m < m'$ . Then, assume the potential outcome to be a deterministic reflection of the respondent’s preference ordering,Footnote ² such that

(2) $$ \begin{align} Y_i(x_{(mi)}, x_{(m'i)}) \ = \ \begin{cases} 1, & \text{if} \ x_{(mi)} \succ x_{(m'i)} \ \Leftrightarrow \ m < m', \\ 0, & \text{if} \ x_{(mi)} \preceq x_{(m'i)} \ \Leftrightarrow \ m \geq m'. \end{cases} \end{align} $$

Under this setup, the AMCE for the population of N respondents with respect to the lth attribute can be written as

(3) $$ \begin{align} \textit{AMCE}_{l}(t_1, t_0; p) & \equiv \frac{1}{N} \sum_{i=1}^N \mathbb{E}_{p}\left[Y_i(t_1, \tilde{X}_{i1k}, X_{i2k}) - Y_i(t_0, \tilde{X}_{i1k}, X_{i2k}) \right] \nonumber \\[6pt] & = \frac{1}{N}\sum_{i=1}^{N}\sum_{\tilde{x}_1\in \tilde{\mathcal{X}}} \sum_{x_2\in\mathcal{X}} \left\{Y_i(t_1, \tilde{x}_1, x_2) - Y_i(t_0, \tilde{x}_1, x_2)\right\}p(\tilde{x}_1, x_2), \end{align} $$

where $\tilde {\mathcal {X}}$ is the support of $\tilde {X}_{ijk}$ and $p(\tilde {x}_1,x_2) = \Pr (\tilde {X}_{i1k}=\tilde {x}_1, X_{i2k}=x_2)$ , a known joint distribution of $\tilde {X}_{i1k}$ and $X_{i2k}$ .Footnote ³

As discussed initially by Hainmueller et al. (Reference Hainmueller, Hopkins and Yamamoto2014) and more extensively by De la Cuesta et al. (Reference De la Cuesta, Egami and Imai2021), the AMCE is defined with respect to the attribute distribution p. In theory, p can be set to any target distribution of interest, for example, the real-world distribution of the attributes.Footnote ⁴ In practice, it is common to set $p(\tilde {x}_{1},x_2)$ to the randomization distribution of the attributes actually used in the experiment, which we denote by $p^\ast (\tilde {x}_1,x_2)$ . This allows the AMCE to be identified by the observed difference in means with respect to $X_{ijkl}=t_1$ versus $X_{ijkl}=t_0$ . Although this is perhaps the most commonly used specification in applied research, it is not required for the results in the rest of this section to hold. In fact, the only assumption we need about the target distribution is the following independence assumption about profiles.

Assumption 1 Independently Generated Profiles

That is, we assume that profiles in each paired comparison are independently drawn from each other. (Note that attributes within each profile are allowed to be arbitrarily correlated.) Assumption 1 is satisfied in a vast majority of conjoint applications we are aware of, where researchers independently draw profiles for their actual experiment and use the actual randomization distribution $p^\ast $ for their (implicit) definition of the AMCE (i.e., $p=p^\ast $ ).Footnote ⁵

To generalize the proportionality between the AMCE and the Borda score for any attribute distribution satisfying Assumption 1, we introduce the following quantity.

Definition 1 Expected Borda Score

$g_i(x_{(mi)}; p) \equiv Q\Pr (X_{i2k} \prec x_{(mi)}),$ where $\Pr (\cdot )$ is defined with respect to the target attribute distribution p.

In words, the EBS of profile $x_{(mi)}$ represents the expected number of times the profile is chosen by respondent i against Q profiles randomly drawn from the target profile distribution p. It is straightforward to show that the original Borda score, $b_i(x_{(mi)}) \equiv \sum _{m'\neq m} \boldsymbol{1}\{x_{(mi)} \succ x_{(m'i)}\} = Q-m$ , is a special case of $g_i(x_{(mi)};p)$ when $\Pr (X_{i2k} = x_{(m)})=1/Q \ \forall \ m\in \mathcal {X}$ (i.e., the discrete uniform). This is because

$$ \begin{align*}g_i(x_{(mi)};p) = Q\sum_{m'\in\{1,\ldots,M\}, m'>m}\frac{1}{Q} = Q-m = b_i(x_{(mi)}).\end{align*} $$

Now, we are ready to state our main proposition for this section.

Proposition 1. Suppose that the target attribute distribution for the AMCE satisfies Assumption 1. Then, the difference between the EBS of attribute $t_1$ and of attribute $t_0$ is proportional to the AMCE of $t_1$ against $t_0$ .

A proof is provided in Section A of the Supplementary Material. Proposition 1 implies that the AMCE can generally be seen as an aggregation of individual preferences according to a probabilistic generalization of the Borda rule. That is, the AMCE is a summary of voters’ multidimensional preferences that incorporates both ordering and intensity.

2.4 The AMCE as the Effect on Vote Shares

The AMCE’s second desirable property is that it represents a quantity of broad interest to empirical elections scholars: the average causal effect of an attribute on vote shares, with the expectation taken with respect to a target election distribution as defined below. Specifically, in a forced-choice conjoint experiment, the AMCE equals the expected difference in the choice probability of a candidate with a treatment attribute level (e.g., gender $=$ female) and that of a candidate with the baseline level of the attribute (e.g., male) in an election with the same number of candidates (i.e., two in a typical paired conjoint). Importantly, this property holds regardless of the structure of individual voters’ preferences. AMCEs identify vote shares irrespective of whether the intensity of voters’ preferences about individual attributes is homogeneous or heterogeneous, or whether there are interactions between candidate attributes in shaping voter preferences. The property also holds independently of the electoral formulae used for aggregating votes into seats, making the AMCE a useful quantity for both majoritarian and proportional representation elections.

Taken literally, the AMCE is only well defined in the context of a conjoint experiment. That is, estimating $\textit{AMCE}_{A}$ corresponds to asking the following question: if we randomly draw a female candidate and her opponent from the set of possible candidates, how much more likely is the female candidate to win the paired conjoint task, compared to a male candidate randomly drawn in the same manner on average? This quantity is of interest to many applied researchers, since the conjoint choice tasks themselves can be a robust and reliable measure of attitudes and opinions (e.g., Bansak et al. Reference Bansak, Hainmueller, Hopkins and Yamamoto2021a; Jenke et al. Reference Jenke, Bansak, Hainmueller and Hangartner2021). Nonetheless, a crucial question for many elections scholars is whether the AMCE is also informative about elections and the aggregation of individual preferences into outcomes through such elections. Does the AMCE map onto any electoral quantity of interest?

Here, we show that the AMCE equals a quantity summarizing the causal effect of a candidate attribute on vote shares in an election matching the specifications of the conjoint. By vote share, we mean the percentage of votes cast for a candidate in an election. An attribute’s AMCE in a conjoint experiment resembling an election can be interpreted as the average causal effect of the attribute on the vote share of a randomly selected candidate with that attribute as opposed to the baseline level of the attribute. Thus, the AMCE is interpretable in terms that are directly relevant for the study of elections.

To make our point formally, we define a target election to be represented by a pair $\langle \mathcal {A}, \mathcal {V} \rangle $ , where $\mathcal {A}$ and $\mathcal {V}$ refer to the target attribute distribution and the target voter distribution, respectively. The attribute distribution $\mathcal {A}$ is a probability measure on the combinations of candidate attributes, whereas the voter distribution $\mathcal {V}$ is a probability measure on individual preferences over the attribute combinations in $\mathcal {A}$ ’s support. For example, consider Table 1, which represents the toy example of an election with candidates with three binary attributes and three voter types. The attribute distribution is a probability mass function over all possible attribute combinations or profiles (i.e., rows in the table’s left half). For instance, it could be a uniform categorical distribution over the eight possible profiles. The voter distribution, in turn, is a probability mass function over the three voter types (i.e., columns in the table’s right half), for example, ${\Pr (\text {Type 1}) = .3,} \Pr (\text {Type 2}) = .4$ , and $\Pr (\text {Type 3}) = .3$ . Note that “target” in these definitions indicates that these distributions usually correspond to populations of voters and candidates that are of interest to the researcher, such as those resembling candidates and voters in a real-world election.

Now, consider a conjoint experiment on a representative sample of respondents randomly drawn from the target voter distribution $\mathcal {V}$ . Furthermore, suppose profiles are randomly generated according to the target attribute distribution $\mathcal {A}$ . It follows that the AMCE of each attribute under the design can be interpreted as the attribute’s average effect on candidate vote shares in the target election $\langle \mathcal {A}, \mathcal {V}\rangle $ . The general result is stated in the following proposition.

The proposition follows trivially from the definitions of the AMCE and the target election, noting that the expected value of the conjoint potential outcome for a profile set (e.g., $\mathbb {E}[Y_i([abc],[a'b'c'])]$ ) equals the proportion of the votes cast for the first candidate in the corresponding target election. (A formal proof is omitted.) Of note, Proposition 2 holds more generally for designs with $J>2$ profiles per choice task. This means that the AMCE allows researchers to use a J-profile conjoint design to study vote shares in J-candidate single-vote elections.

Proposition 2 implies that election scholars can use appropriately designed conjoint survey experiments to predict candidates’ vote shares in elections and interpret the resulting AMCEs as the causal effects of candidate attributes on predicted vote shares. For example, an AMCE of $-0.02$ for a male candidate versus a female candidate indicates that male gender has an average causal effect of $-$ 2 percentage points on candidates’ vote shares in elections resembling the experiment’s design: on average, a randomly selected male candidate would earn 2 fewer percentage points of the total vote share than a female.

But how common are vote shares as a quantity of interest in empirical election research? We conducted a literature review including all articles on voting in four journals that commonly publish studies on voting behavior between 2015 and 2019.Footnote ⁷ Of the 82 articles reviewed, 87% include either aggregate vote shares or their individual-level analogs as one key outcome. The small minority of studies that do not feature outcomes related to vote shares instead have outcomes such as the probability of a candidate or party winning. Thus, not only does the AMCE recover a politically meaningful quantity, but also it recovers a quantity that has been the primary quantity of interest even in most non-conjoint studies in recent years.

3.1 Probability of Winning

In a paired forced-choice conjoint design simulating a two-candidate election, a natural quantity of interest is the probability of winning, or the probability that one candidate will win a majority of votes. To formalize this quantity, recall that respondent i chooses candidate $[abc]$ over candidate $[a'b'c']$ if and only if $Y_i([abc],[a'b'c']) = 1$ . Candidate $[abc]$ therefore wins a majority vote against candidate $[a'b'c']$ if and only if

(4) $$ \begin{align} \mathbb{E}_{\mathcal{V}}[Y_i([abc],[a'b'c'])] > \ 0.5, \end{align} $$

where the expectation $\mathbb {E}_{\mathcal {V}}$ is defined over the target voter distribution $\mathcal {V}$ . In words, candidate $[abc]$ wins a majority vote against candidate $[a'b'c']$ if more than half of respondents drawn from the target voter distribution would choose $[abc]$ over $[a'b'c']$ , or equivalently if $[abc]\succ [a'b'c']$ for more than half of respondents.

Equation (4) constitutes a building block for various possible quantities of interest we term probabilities of winning. Let $M([ABC],[A'B'C']) \equiv \boldsymbol{1}\{\mathbb {E}_{\mathcal {V}}[Y_i([ABC],[A'B'C'])]> 0.5\}$ , a binary random variable representing whether profile $[ABC]$ wins a majority of votes against $[A'B'C']$ . For example, suppose that the researcher is interested in how likely a candidate with attributes $A=a$ , $B=b$ , and $C=c$ is to win a majority against another candidate randomly drawn from the target population. This probability is

(5) $$ \begin{align} \mathbb{E}_{\mathcal{A}}\left[M([abc],[A'B'C'])\right], \end{align} $$

where the expectation $\mathbb {E}_{\mathcal {A}}$ is taken with respect to the target attribute distribution $\mathcal {A}$ , which the second candidate’s attributes $A'$ , $B'$ , and $C'$ are drawn from. Alternatively, the researcher might be interested in a particular attribute (e.g., $A=a$ ) and how likely a candidate with that attribute is to win under majority rule. This alternative quantity can be defined as

(6) $$ \begin{align} \mathbb{E}_{\mathcal{A}}\left[M([aBC],[A'B'C'])\right], \end{align} $$

where the expectation now averages over the first candidate’s attributes other than A and the second candidate’s attributes. Yet another quantity of interest is how often a candidate with attribute $A=a$ will win against a candidate with attribute $A=a'$ . This quantity is

(7) $$ \begin{align} \mathbb{E}_{\mathcal{A}}\left[M([aBC],[a'B'C'])\right], \end{align} $$

where the expectation is now defined with respect to the distribution of B, C, $B'$ , and $C'$ .

The choice between different conceptions of the probability of winning depends on researchers’ substantive question. Researchers may be interested in a real-world politician and ask about the likelihood a similar candidate would win a majority (Equation (5)). Alternatively, researchers might ask how likely a female candidate is to win a majority against a randomly drawn candidate (Equation (6)). Regardless of the estimand chosen, it is imperative to clarify one’s substantive question of interest and map it to a well-defined estimand in terms of potential outcomes.

Inference about the probability of winning proves more challenging than about AMCEs. This is due to the nonlinearity built into majority rule (or, more generally, into the electoral formula which translates votes into seats) and the resulting high-dimensional estimation problem. To see the challenge, consider estimating the probability of winning for a female candidate against a male candidate (Equation (7)) where $A=a$ ( $a'$ ) represents a female (male) candidate. Without additional assumptions about the potential outcomes’ functional form, we can obtain a sample analog of Equation (7) as follows: calculate the female candidate’s vote share for each of the W possible unique contests between females and males, determine whether the female candidate wins the majority in each, and calculate the average of the resulting indicators over the W contests.

Although this nonparametric plug-in estimator is consistent for Equation (7) as the numbers of respondents (N) and tasks (K) grow infinitely for a fixed number of attributes (L), a practical difficulty is that W is very large compared to the sample size ( $NK$ ) in typical conjoints, making the data too sparse for this inferential problem. For example, with eight binary attributes, there are $W=2^{(8-1)\times 2} - 1 = 16,383$ possible unique contests between female and male candidates. With 1,000 respondents each completing 20 tasks, we can only expect to have slightly more than one observation for each possible pairwise comparison. Thus, the fully nonparametric estimator is impractical in all but the simplest experiments.

More promising would be a model-based approach which explicitly models the majority indicator $M([ABC],[A'B'C'])$ as a function of the attributes, which can then be used to estimate any of the probability of winning quantities of interest defined above by averaging the estimated M over the distribution of the attributes corresponding to the target estimand. In Section C of the Supplementary Material, we sketch one such procedure and perform simulations that provide baseline evidence for the feasibility of using conjoint data to estimate quantities of interest related to the probability of winning. The results highlight how probabilities of winning, while substantially more difficult to estimate than AMCEs, are still promising quantities of interest to evaluate using conjoint data. In addition, some minor tailoring of the conjoint design specifically for this quantity of interest could be undertaken to further improve the estimation. For instance, if one were interested in Equation (7) with respect to one particular attribute, fixing $a$ for the first profile and $a^\prime $ for the second profile (while randomizing the other attributes) would improve the precision of the subsequent estimates. More research in this area is valuable, and it relates to the broader point that experimental design should be tailored for each causal quantity of interest.

3.2 Fraction of Voters Preferring an Attribute

Another set of possible quantities of interest pertains to the fraction of voters preferring attribute $A=a$ over $A=a'$ and whether that fraction constitutes a majority. To define this quantity meaningfully, we first must define preferences over individual attributes (as opposed to profiles as a whole), which had not previously been necessary. Drawing on Section 2’s definition of preferences, we say that a voter prefers attribute $A=a\ {}$ to $\ A=a'$  if and only if the average rank for  $a$  is less than the average rank for $a'$ .Footnote ¹¹ It is then straightforward that, assuming $\mathcal {A}$ to be uniform over the set of all possible attribute combinations, voter i prefers attribute $A=a$ over $A=a'$ if and only if $\mathbb {E}_{\mathcal {A}}[Y_i([aBC],[a'B'C'])]> 0.5$ , which follows from the fact that $S_i^A(a) < S_i^A(a')$ iff $\mathbb {E}_{\mathcal {A}}[Y_i([aBC],[a'B'C'])]> 0.5$ .Footnote ¹²

Based on this definition, we can define the fraction of voters preferring $A=a$ over $A=a'$ as

(8) $$ \begin{align} \mathbb{E}_{\mathcal{V}}\left[\boldsymbol{1}\{\mathbb{E}_{\mathcal{A}}[Y_i([aBC],[a'B'C'])]> 0.5 \} \right]. \end{align} $$

Note that this quantity does not equal the probability of winning defined in Equation (7) since the order of the two expectations is reversed. Instead, the quantity amounts to first classifying all voters into those (for example) preferring female versus male candidates and then calculating the proportion of female preferers.

The distinction between the two alternative quantities—the probability of winning and the fraction of voters preferring an attribute—is subtle but important. Equation (8) does not generally equal the probability of a female candidate winning a majority election against a male candidate, which may be of more interest to scholars focused on electoral outcomes. In contrast, the fraction of voters preferring attribute $A=a$ over $A=a'$ may be less informative about the importance of that attribute for actual voting behavior and outcomes in multi-attribute contexts. As our handedness example above illustrated, the vast majority of voters may prefer a right-handed candidate, but this attribute would almost never determine any voter’s actual choice, making its AMCE or effect on the probability of winning nearly zero.

In Section D of the Supplementary Material, we extend our discussion of this quantity to consider more restricted versions of Equation (8) focusing on the fraction of voters preferring attribute $A=a$ over $A=a'$ holding all other attributes equal, a potential quantity of interest noted by Abramson et al. (Reference Abramson, Koçak and Magazinnik2021). Among those, we propose Equation (8) as the definition if researchers remain interested in analyzing the fraction of voters who prefer a particular attribute regardless of its relative importance against other attributes. Nonetheless, estimating this quantity presents far greater challenges than estimating the probabilities of winning, since it requires explicitly analyzing voters’ preferences at the individual level.

That is, one first needs to estimate the equation’s inner expectation term, $\mathbb {E}_{\mathcal {A}}[Y_i([aBC], [a'B'C'])]$ , which equals the probability of choosing a profile containing $A=a$ versus another profile containing $A=a'$ for a specific voter i. Unfortunately, only a handful of observations per voter will be available to estimate that inner expectation for any particular comparison $a$ versus $a'$ in typical conjoint experiments. This is in contrast to the probability of winning quantities of interest, in which the inner expectation is taken with respect to all voters, and hence can be modeled and estimated on the basis of an entire dataset. Poor performance in estimating the fraction-preferring quantity is therefore likely, as the application of the indicator function to a noisy input will result in severe misclassification, and misclassification is always negatively correlated with the true value. Indeed, Section D of the Supplementary Material highlights this problem based on the same simulations used for assessing the probability of winning estimands. For researchers interested in a fraction-preferring quantity, alternative research designs are likely warranted.

3.3 Revisiting the AMCE

The above discussion of alternative quantities of interest returns us to the AMCE’s third desirable property: empirical tractability. As Hainmueller et al. (Reference Hainmueller, Hopkins and Yamamoto2014) detail, by virtue of attribute randomization, the AMCE can be nonparametrically identified via a simple difference in means, much like a standard experiment with a single treatment. Because the AMCE is a purely linear function of the potential outcomes, it does not require a functional form assumption as the probability of winning or the fraction of voters preferring an attribute does.

This discussion should not be novel to those familiar with causal inference and the ATE as a causal estimand. All of the quantities of interest discussed so far can be viewed as causal quantities, in that they involve counterfactual comparisons between possible combinations of attributes or treatment components. When making inferences about a causal quantity, one faces the problem of identifying counterfactual comparisons not directly observed in the data. As is well known, treatment randomization solves this problem for common causal estimands such as the ATE. Less well known, however, is that randomization solves the identification problem only for a certain class of causal estimands. Fortunately, this class of estimands includes causal effects such as the ATE. However, it excludes others, such as the median treatment effect, or the effect of the treatment on an individual unit at the median of the individual-level treatment effect distribution.

This is analogous to the relationship between the AMCE and alternative aggregations of treatment effects. Whereas the AMCE is nonparametrically identified by the observed difference in means due to random assignment, quantities involving nonlinear mappings such as the probability of winning require additional assumptions and/or complicated modeling techniques. This helps explain why recent empirical conjoint applications have gravitated toward AMCEs.

Scholars in various fields have focused on the ATE because it can be identified with minimal assumptions and provides a useful, interpretable summary of causal effects. In that regard, the fact that the AMCE combines both preference directionality and intensity is a feature, not a bug. If a small number of people always support a candidate with a specific attribute $a$ , they may overwhelm the majority of respondents who slightly prefer its inverse $a'$ . This is true of the ATE, too; if a small number of lives are saved by taking a medication, that may overwhelm the temporary, negative side effects that a larger number of people experience on any measure of long-term health. Combining directionality and intensity is fitting in many political applications: in many cases, a minority of people with intense preferences over a certain attribute can drive its electoral significance. Moreover, this is not merely a rhetorical point because, as illustrated above, the AMCE identifies the difference in expected vote shares.