1 Introduction
This paper is focused on a common design decision in conjoint experiments: whether to include or exclude odd attribute-level combinations. The benefit of conjoint designs is that their randomization of many attributes allows for the study of choices and preferences over multidimensional objects. However, because of that multidimensionality, combined with researchers’ interest in investigating distinct but related characteristics (e.g., education and profession), the resulting design may yield randomized profiles that are “odd,” i.e., that contain combinations of attribute levels that are atypical (incongruent) or generally unrealistic in the real world (nonsensical). For instance, consider a conjoint experiment that features profiles of political candidates and that includes the attributes of political party and position on abortion rights. The possible levels of the former attributes might include Democrat, Republican, and Independent, while the levels of the latter attribute might include strongly oppose, oppose, support, and strongly support. In such a case, it is possible for a randomly generated profile to take the Democrat level for the party attribute and the strongly oppose level for the abortion rights position attribute. This combination of attribute levels might then be viewed as odd by respondents. As a more extreme example, consider a design that includes both highest education and previous profession as attributes, which might result in nonsensical pairings like a doctor with only a high school education. Odd combinations such as these raise the question of how respondents cognitively coalesce information from one attribute that contradicts information from another attribute. This has become a common consideration in conjoint research, and applied researchers have routinely grappled and dealt with their concerns about odd profiles in various ways—such as restricting the attribute randomization to prevent certain pairings, omitting possibly interesting attributes or levels from their design altogether, or allowing for odd combinations and hoping for the best (e.g., Auerbach and Thachil Reference Auerbach and Thachil2018; Ballard-Rosa, Martin, and Scheve Reference Ballard-Rosa, Martin and Scheve2017; Bechtel, Genovese, and Scheve Reference Bechtel, Genovese and Scheve2019; Carnes and Lupu Reference Carnes and Lupu2016; Hainmueller and Hopkins Reference Hainmueller and Hopkins2015; Hartman and Morse Reference Hartman and Morse2020; Hemker and Rink Reference Hemker and Rink2017; Horiuchi, Smith, and Yamamoto Reference Horiuchi, Smith and Yamamoto2018; Huff and Kertzer Reference Huff and Kertzer2018; Oliveros and Schuster Reference Oliveros and Schuster2018).Footnote 1
This study provides the first systematic investigation, to our knowledge, of the effects of odd profiles on survey-taking attention and behavior. Specifically, how might the inclusion of odd attribute-level combinations affect the engagement, attention, information search behavior, and choices of respondents? As has been shown by decades of research in survey methods, survey tasks that require higher levels of cognitive effort can result in undesirable behavior on the part of respondents, such as satisficing and other effort-reducing heuristics that affect survey responses (and hence the researcher’s inferences) (Krosnick Reference Krosnick1999; Roberts et al. Reference Roberts, Gilbert, Allum and Eisner2019). Along these lines, the inclusion of odd attribute-level combinations in a conjoint design may induce unintended cognitive and behavioral effects in a number of ways, including increasing the amount of attention required to process the conjoint information, focusing respondents’ attention on certain attributes at the expense of others, or reducing respondents’ overall survey-taking seriousness and effort due to confusion or frustration. The behavior of respondents—which underlies the effects captured by the conjoint estimands—may differ based on whether researchers choose to include odd attribute combinations or not.
There are multiple considerations when researchers are deciding whether or not to allow odd attribute-level combinations within their conjoint designs. Previous research has highlighted how the inclusion or exclusion of particular attribute-level combinations can affect the estimation of key quantities of interest from theoretical and statistical perspectives (Abramson, Koçak, and Magazinnik Reference Abramson, Koçak and Magazinnik2022; Bansak et al. Reference Bansak, Hainmueller, Hopkins, Yamamoto, Druckman and Green2021; De la Cuesta, Egami, and Imai Reference De la Cuesta, Egami and Imai2022; Ganter Reference Ganter2023; Hainmueller, Hopkins, and Yamamoto Reference Hainmueller, Hopkins and Yamamoto2014; Horiuchi, Markovich, and Yamamoto Reference Horiuchi, Markovich and Yamamoto2022). For instance, one consideration is external validity (De la Cuesta et al. Reference De la Cuesta, Egami and Imai2022). However, understanding respondent attention and behavior in response to odd combinations has been far less studied. One previous study that analyzes these issues to any degree is Hainmueller et al. (Reference Hainmueller, Hopkins and Yamamoto2014). However, as these issues were not a central focus of that study, the corresponding analysis was relatively coarse as well as limited to considering impacts on respondent choice.
Here, we focus specifically and exclusively on a broader set of issues related to the impact of odd profiles on survey-taking behavior, and we employ a novel and pre-registered set of conjoint experiments that we designed expressly for that purpose. In addition, we further bolster the research design by using eye-tracking to obtain information about respondents’ information search and attention, following previous related work that employed eye-tracking to understand respondents’ reactions to other elements of conjoint design (Jenke et al. Reference Jenke, Bansak, Hainmueller and Hangartner2021). This approach allows us to analyze the effects of odd profiles on a wider variety of survey-taking behaviors.
Our results indicate that while odd combinations may have some impact on respondent attention and behavior, the impact is minimal, inconsistent, and is unlikely to meaningfully affect the first-order inferences researchers hope to draw from conjoint experiments. We emphasize that we built a variety of features into our research design to allow for an extremely comprehensive investigation, including (i) both within-condition and between-condition analyses, (ii) collection and use of both eye-tracking and choice data, (iii) multiple types of odd combinations, (iv) a high degree of variation in the presence of odd combinations, and (v) two different topical scenarios. Across all of our results, odd combinations appear not to have large or consistent effects on respondents’ information accrual or choice behavior. While we do not want to trivialize or rule out any impact of odd combinations, we do not find compelling evidence to substantiate concerns that odd combinations will lead to data quality deterioration or behavioral shifts, especially given the wide variety of ways in which we tested for effects.
Hence, our recommendation is that such concerns should not be prioritized when deciding on the inclusion/exclusion of odd attribute-level combinations. Instead, there is more at stake in terms of the substantive considerations (e.g., interest in unpacking the independent effects of attributes like party affiliation and issue positions) and statistical considerations (e.g., wanting the randomization distribution to better reflect an underlying target/population distribution of interest), and researchers should make their decisions to include/exclude odd combinations on the basis of these factors.
Additionally, our results contribute to the literature by revealing how respondents are thinking about their choices in conjoint experiments. Our findings suggest that respondents are not strongly considering attributes in concert with one another but rather thinking about the independent influence of the attributes on their preferences. This result offers an explanation as to why researchers may not find many inter-attribute interactive effects on choice in their conjoint experiments. It also raises the question of whether we expect people in the real world to have a similar decision-making process and, if not, why there is close correspondence between conjoint experimental and real world outcomes (Hainmueller, Hangartner, and Yamamoto Reference Hainmueller, Hangartner and Yamamoto2015; Louviere Reference Louviere1974, Reference Louviere1988; Louviere and Woodworth Reference Louviere and Woodworth1983). The answer to this question is important in clarifying the external validity of conjoint experiments.
2 Theory and Hypotheses
Conjoint experiment profiles are comprised of multiple attributes (i.e., variables), and each attribute is associated with a number of possible levels (i.e., values). For any given profile, the levels that each attribute takes are randomized. As a result, for any two attributes (or more generally any subset), there will be varying degrees of theoretical or real-world coherence between randomly chosen levels. At one end of the spectrum, there are combinations that may be viewed as “nonsensical” by respondents. For instance, a conjoint design featuring political candidates might include the candidates’ highest levels of education and their previous professions. A randomization resulting in a high school education and the profession of a medical doctor would be nonsensical. That same conjoint design might also include the candidates’ political parties and positions on immigration as two of the attributes. A randomization resulting in a Democrat and an extremely conservative position on immigration as the realized levels for those attributes would constitute an incongruent profile—slightly less odd than a nonsensical profile but nonetheless dissonant.
For the sake of brevity, we will refer to odd attribute-level combinations within a single profile as “odd combinations” from here on out. Attributes that are involved in odd combinations will be referred to as “odd attributes,” and attributes not involved in odd combinations will be referred to as “normal attributes.”Footnote 2 Experimental conditions that include odd combinations will be referred to as “odd conditions,” and conditions that do not include odd combinations will be referred to as “normal conditions.” Note that throughout this study the odd combinations that we consider are comprised only of pairs rather than larger subsets of attribute-levels. Finally, we operate under the premise of a paired-profile conjoint design (i.e., where each conjoint task is comprised of two profiles viewed side-by-side), which is the most common design in political science research. Next we present our hypotheses, which are also summarized in Table 1.
Summary of Hypotheses. All hypotheses are tested both (1) between odd conditions and the normal condition and (2) within-condition (when respondents are presented with more versus fewer odd combinations in a profile).
2.1 Information Search
Decades of research in survey methods have demonstrated that survey tasks requiring higher levels of cognitive effort can result in satisficing and other undesirable behavior on the part of respondents (Krosnick Reference Krosnick1999; Roberts et al. Reference Roberts, Gilbert, Allum and Eisner2019). As conjoint tables become more complex in terms of the number of profiles and attributes, participants in conjoint experiments take part in behavior consistent with a bounded rationality mechanism (Jenke et al. Reference Jenke, Bansak, Hainmueller and Hangartner2021). They focus on the attributes that matter most to them and ignore less relevant attributes to deal with the increased computational cost of larger tables. But a new type of complexity is added to the task when odd combinations are introduced, and their incorporation may induce unintended cognitive and behavioral effects in three ways.
First, respondents’ overall attention to profiles may change. The presence of odd combinations may lead to confusion, and respondents may seek to clarify their attitudes toward such profiles. Thus, the presence of odd combinations within a profile may increase the amount of attention respondents give to the profile in the form of attending to a greater number of attributes and/or increasing the overall number of fixations on the profile.
$H_A$
: Respondents will increase the overall number of fixations and/or look at a greater number of attributes when odd combinations are present (both in an odd condition vs. a normal condition and within-condition when presented with more odd combinations in a profile).
That said, odd combinations and respondents’ resulting confusion may alternatively lead respondents to take the choice tasks less seriously. This could reduce their overall effort on the task, and if so, the presence of odd combinations will lead respondents to pay less attention when evaluating a profile in an odd condition.
$H_B$
: Respondents will decrease the overall number of fixations and/or look at fewer attributes when odd combinations are present (both in an odd condition vs. a normal condition and within-condition when presented with more odd combinations in a profile).
Second, regardless of whether the total amount of attention differs as a function of odd combinations, the presence of odd combinations may lead respondents to focus on certain attributes at the expense of others or engage in selective attention. The neuroscience and psychology literatures have found that attention is drawn to stimuli that deviate from contextual expectations (Horstmann Reference Horstmann2005; Itti and Baldi Reference Itti and Baldi2009; Jerónimo, Volpert, and Bartholow Reference Jerónimo, Volpert and Bartholow2017), such as objects that are incongruent with a scene (Võ and Henderson Reference Võ and Henderson2009). This would lead respondents to direct more attention toward odd attributes and away from normal attributes. Following this logic, respondents should focus more on an attribute if it is involved in an odd combination than if it is not.
$H_C$
: Respondents will increase the number of fixations on an attribute if it is involved in an odd combination (compared to when that same attribute is not involved in an odd combination) both within-condition and between-condition.
On the other hand, odd combinations within a profile may lead respondents to ignore the odd attributes if they believe they cannot extract meaningful information from them. Other neuroscience research has shown that attention is biased toward information that is relevant to a task, such as reward-related stimuli (Anderson, Laurent, and Yantis Reference Anderson, Laurent and Yantis2011; Anderson and Yantis Reference Anderson and Yantis2013; Hickey, Chelazzi, and Theeuwes Reference Hickey, Chelazzi and Theeuwes2010). Less attention may be given to attribute pairings that are difficult to interpret and thus less relevant to the task of choosing between profiles.
$H_D$
: Respondents will decrease the number of fixations on an attribute if it is involved in an odd combination both within-condition and between-condition.
Third, the presence of odd combinations may impact respondents’ search transitions—whether respondents search “within-profile” (i.e., looking at all of one profile’s information before moving to the other profile) or “within-attribute” (i.e., looking at all profiles’ levels of an attribute before moving to the other attributes). The observed pattern of search allows for inference about a respondent’s information processing: within-profile transitions imply that the respondent builds a summary of a given profile before moving to the next profile, while within-attribute transitions imply that the respondent keeps a running tally of “wins” and “losses” for each profile on each attribute (Lau and Redlawsk Reference Lau and Redlawsk2006; Payne Reference Payne1982; Redlawsk Reference Redlawsk2004; Schulte-Mecklenbeck, Kühberger, and Ranyard Reference Schulte-Mecklenbeck, Kühberger and Ranyard2011; Tversky Reference Tversky1969). Odd combinations may shift respondents from thinking about attributes in a more singular fashion (which is encouraged by a within-attribute pattern) to considering attributes’ joint effects on their evaluations (which is encouraged by a within-profile pattern). Thus, respondents may search more within-profile when odd combinations are included in the profiles.
$H_E$
: Respondents will search more within-profile when viewing profiles with a greater number of odd combinations than when looking at profiles with a lesser number of odd combinations, within odd conditions. They will also search more within-profile when in odd conditions than when in a normal condition.
All of these attentional changes (
$H_A$
–
$H_E$
) may be proportional to or dependent upon the degree of oddness in the combinations. The observed changes may be more pronounced given nonsensical combinations than they are given incongruent combinations. For example, since nonsensical combinations are more challenging to interpret than incongruent combinations, it is possible that respondents will reduce attention to nonsensical combinations (compared to normal combinations) to a greater degree than they will reduce attention to incongruent combinations (compared to normal combinations). Because our design includes three odd conditions—one with only incongruent combinations, one with only nonsensical combinations, and one with both types of combinations—we can explore this possibility both within- and between-conditions.
$H_F$
: The observed changes will be more pronounced for the nonsensical attributes/condition compared to normal attributes/condition than they are for the incongruent attributes/condition compared to the normal attributes/condition.
2.2 Choice Behavior
The type of search transitions, the extent of search, and selective attention may impact the choices that people make in surveys (Krajbich et al. Reference Krajbich, Dingchao, Camerer and Rangel2012; Lau and Redlawsk Reference Lau and Redlawsk2006; Reutskaja et al. Reference Reutskaja, Nagel, Camerer and Rangel2011). Hence, odd combinations may impact profile choice through these channels. If respondents focus more on odd attributes and pay less attention to normal attributes (
$H_C$
), for instance, then the presence of odd combinations should attenuate the marginal effects or influence of normal attributes on profile choice while increasing the influence of odd attributes on profile choice. This may be the case both within-condition (the marginal effects of odd attributes may be greater in the presence of more odd combinations within a profile) as well as between-condition (the marginal effect of an attribute may be greater when it is an odd attribute in an odd condition than when that same attribute is in a normal condition).
$H_G$
: The presence of odd combinations will attenuate the marginal effects of normal attributes and increase the marginal effects of odd attributes on profile choice both within-condition (profiles with fewer odd combinations compared to profiles with more odd combinations) and between-condition (odd conditions compared to the normal condition).
On the other hand, if odd combinations within a profile lead respondents to focus less on those odd attributes and pay more attention to normal attributes (
$H_D$
), then the presence of odd combinations will strengthen the marginal effects/influence of normal attributes on profile choice while decreasing the influence of the odd attributes.
$H_H$
: The presence of odd combinations will attenuate the marginal effects of odd attributes and increase the marginal effects of normal attributes on profile choice both within-condition and between-condition.
If nonsensical combinations change respondents’ search behavior more substantially than incongruent combinations, then these hypotheses regarding choice behavior will be more pronounced for nonsensical combinations than they are for incongruent combinations.
$H_I$
: The observed changes in choice behavior will be more pronounced for the nonsensical attributes/condition compared to normal attributes/condition than they are for the incongruent attributes/condition compared to the normal attributes/condition.
3 Design
To test these hypotheses, we designed a pre-registered set of conjoint experiments tailored specifically for this purpose.Footnote 3
The experiments included two substantive scenarios. In one scenario, respondents were asked to choose between two political candidate profiles. In the United States’ two-party, polarized context, we might expect respondents to be particularly sensitive to odd attribute combinations involving the candidates’ partisanship and issue positions. In the other scenario, respondents were asked to choose between the profiles of two immigrants interested in residency in the U.S. Both of these are common scenarios in political science research employing conjoint experiments.
Information about the candidates and immigrants was presented in standard conjoint tables, with each table featuring two profiles (columns) and eight attributes (rows). Figure A.1 in the Supplementary Material shows an example. Respondents completed 60 decision tasks (i.e., viewed 60 tables and chose their preferred profile for each table) per scenario, for a total of 120 profiles evaluated per respondent per scenario. The order of the scenarios was randomized across respondents. This large number of tasks is necessary in order to increase the statistical power for our analyses. However, there is a possibility that respondents may have behaved differently than they would have in an experiment with fewer tasks. To test for this possibility, we include within- and between-condition analyses with only the first condition seen by respondents in the Supplementary Material. By testing our hypotheses on only the first condition seen by respondents, we control for concerns that respondents’ survey-taking behavior in our experiment may differ from that in typical political science conjoint experiments due to the greater number of tasks.
Each scenario had four treatment conditions: (1) a “normal” condition that did not contain any odd combinations, (2) an “incongruent” condition, which contained combinations that were unusual but reasonable (e.g., an immigrant who was a female construction worker), (3) a “nonsensical” condition, which contained generally implausible combinations (e.g., a medical doctor whose highest education is undergraduate college), and (4) a “combined” condition, which contained both the incongruent and nonsensical combinations. As mentioned earlier, the odd combinations we considered comprised only pairs (rather than larger subsets) of attribute levels. Including both incongruent and nonsensical odd conditions allows us to explore whether the degree of oddness makes a difference in respondents’ behavior. The combined condition enables robustness checks on any effects that are found for either the incongruent or nonsensical conditions. Theoretically, we would expect the replication of any significant results from the incongruent or nonsensical conditions in the combined condition. Given our large number of analyses, this allows us to guard against chance statistical significance.
The introduction of odd combinations to different conditions required modifying a select number of attribute levels and, in some cases, attributes themselves. Tables A.2 and A.3 in the Supplementary Material show the attributes and levels included in each condition for both scenarios. The combinations that are considered (and pre-registered as) “incongruent” and “nonsensical” are delineated in Tables A.4 and A.5 in the Supplementary Material. The probabilities of seeing at least one incongruent or nonsensical combination in a profile in each condition are between 0.35 and 0.79 (see Table A.1 in the Supplementary Material). Each condition involved 15 decision tasks. All respondents viewed all four conditions, with the order randomized independently across respondents. This design choice enhanced the comprehensiveness of our investigations by providing both within- and between-condition analyses, as further explained below.
We note that there is an inherent degree of subjectivity involved in deeming (or not deeming) a combination to be “incongruent” or “nonsensical,” and our analytical goals do not require these terms to be taken literally. Unavoidably, it is possible to quibble about some of our decisions on what counts as odd. More important for our analytical purposes is relative oddness, and in our view the combinations that we deem odd are clearly odder than those we do not. We also acknowledge that we cannot establish that every one of our pre-registered odd combinations was perceived as odd by all respondents (e.g., a respondent might have, for some reason, not considered a refugee from France to be strange). But for this reason, we included many odd combinations in each condition and made the probability of seeing an odd combination quite high. In particular, in the combined condition, respondents had a 0.72 probability (candidate choice) and a 0.79 probability (migrant choice) of seeing at least one odd combination in a trial. Hence, our design features a substantial increase in both the variety and probability of odd combinations over previous conjoint experiments, thereby creating ample opportunities to detect the effects of odd combinations.
We additionally did not merely use our own subjective judgment in deciding upon the odd combinations but rather identified odd combinations that other researchers have considered and worried about in previous scholarship (e.g., combinations of certain levels of education and profession). Our approach was to follow previous literature in identifying combinations that are salient for applied research rather than trying to establish in a deeper, theoretical manner what is “truly odd.” Our focus in this paper is on the high-level goal of providing applied researchers with practical guidance regarding how to approach decisions regarding odd profiles. And equally (if not more) importantly, we pre-registered all of our odd combination selections as well as our analyses to ensure that our evaluations are not susceptible to data dredging. Finally, researchers who still take issue with our coding of odd combinations could use our data to replicate many of our analyses (specifically our within-condition analyses) using their own coding of oddness.
The attribute levels shown in each profile were randomized independently and uniformly for all respondents. The attributes were presented in a randomized order across respondents and conditions but held fixed for the duration of the condition for each respondent. This consistency of the attribute order, in addition to an example task that was completed before beginning each condition’s first task, allowed respondents to become familiar with the layout of the table and avoid searching for the location of the desired information in each task.
3.1 Analytical Considerations
Because varying the degree of oddness across the four designs required changing the attribute levels and/or attributes themselves across conditions, it is not only oddness that is changing across the conditions but also the composition of attribute levels (and/or attributes) as well. From a purely technical perspective, this makes it impossible in the between-condition analyses to connect any differences we see across the conditions solely to variation in oddness. Therefore, we took steps to protect against our inferences of interest being driven by attribute-specific effects. Most importantly, we conducted a wide range of within-condition analyses, which are not affected by attribute/level substitutions across conditions.
In addition, for the between-condition analyses, all attribute levels that are substituted for each other across conditions are comparable in terms of length and reading comprehension levels. Thus, any changes in respondent attention and search behavior (i.e., where and how they look across the conjoint tables) between conditions should not be a function of requiring a higher level of effort to understand more difficult individual pieces of content.
Furthermore, the possibility that effects between conditions are partially driven by changes in attribute substance (and not just combination oddness) should bias our results in favor of finding differences across the conditions. Therefore, and as a preview of the results, the fact that we find few to no differences is convincing evidence of minimal to no effect of oddness on respondent attention and search. It is highly unlikely that the effects of changes in attribute levels and of oddness would be perfectly countervailing across the board in our between-condition analyses.
It is more difficult to dismiss concerns regarding the effect of entire attribute substitutions on differences in respondent choice across conditions (e.g., for the incongruent and nonsensical conditions in the immigrant choice scenario, an immigrant’s basis for choosing a settling location is replaced by an immigrant’s reason for leaving their country of origin). Changes of entire attributes are a more significant design modification than changes in certain levels of fixed attributes. Hence, our analyses of the effects on respondent choice will take this complication into account, as described later.
3.2 Eye-tracking Technology and Methods
As there is minimal lag between when a person’s gaze rests upon an object and related cognitive processing in the brain, it is well established that measurements of gaze locations are indicative of attention (Just and Carpenter Reference Just and Carpenter1976; Yarbus Reference Yarbus2013).Footnote 4 To measure respondents’ gaze patterns as they progressed through the conjoint experiment, the conjoint tables were shown to respondents on a Tobii T60XL eye-tracker screen (Tobii Technology, Stockholm, Sweden). This eye-tracker is a remote system that does not require any affixation of the head or chin and allows for moderate head movements. The system adjusts to various respondent physiological factors such as contact lenses and glasses.
The sampling rate of the Tobii T60XL is 60 Hz, meaning that at a rate of 60 times per second, we measured where on the screen each respondent was looking. These data were then mapped to eye movement patterns using fixation analysis. Whereas saccades describe rapid movements of the eyes between objects during which vision is suppressed, fixations describe periods in which the eyes are relatively still and focused on a coordinate. A fixation velocity algorithm with a velocity threshold of 30 ms was used to differentiate fixations from saccades (Holmqvist et al. Reference Holmqvist, Nyström, Andersson, Dewhurst, Jarodzka and Van de Weijer2011). These fixations were then used to measure attention using the fixation density, or number of fixations, associated with areas of interest (AOIs), namely the cells of a conjoint table. Fixation density is one of the most used metrics in eye-tracking research and is often used to represent semantic importance or utility (Jacob and Karn Reference Jacob and Karn2003; Orquin and Loose Reference Orquin and Loose2013; Poole, Ball, and Phillips Reference Poole, Ball and Phillips2005).
In sum, based on our eye-tracking data, for each conjoint task completed by a respondent (i.e., each conjoint table a respondent considered and then selected a preferred profile from), we can measure which attributes the respondent viewed for each profile, for how many fixations and for how long (i.e., number of milliseconds) they looked at each attribute, and the specific order in which they looked at attributes. Additional details on the eye-tracking technology, methods, and data processing are available in Section B of the Supplementary Material.
3.3 Sample
Our sample consisted of 147 respondents, who completed the conjoint experiment between June 15 and September 9, 2022. As each subject evaluated 60 profile pairs for each of the two scenarios, this gives us over 17,000 observations of tables (approximately 35,000 observations at the profile level) in our data. The respondents were from the Duke Fuqua Behavioral Research subject pool, which draws local community members as well as students and allows our sample to be more representative of the U.S. population than a typical college undergraduate sample.
Our sample included undergraduate and graduate students from the university (65% of the sample) as well as other members of the local community (35% of the sample). 79% of the sample identified as Democrats or Democrat-leaning, 14% identified as Republican or Republican-leaning, and 7% identified as not affiliating with or leaning toward either party. The mean and median age of the sample were 30 and 26 years old, respectively. 33% of the sample was male and 67% was female. The respondents were 59% white (exclusively) and 41% non-white (including respondents who identified with multiple racial/ethnic groups). Respondents received compensation of $20 for participating in the experiment.
4 Analysis and Results
In this section, we describe the analyses we performed to test each of our hypotheses along with the results.Footnote 5 We present both within-condition and between-condition analyses to test each of our hypotheses and, unless otherwise noted, each analysis that is presented was pre-registered. We follow the standard assumptions in the causal inference tradition of conjoint analysis laid out by Hainmueller et al. (Reference Hainmueller, Hopkins and Yamamoto2014), specifically the assumptions of stability and no carryover effects, no profile-order effects, and randomization of the profiles. Also following standard practice, in all of our analyses, we account for non-independence in our data due to cluster sampling (i.e., multiple observations being contributed by each respondent) by using cluster-robust variance estimation as the basis for our uncertainty measures (clustered by respondent). Finally, to account for possible violations of the assumption of no carryover effects, we also include supplementary analyses that directly test for cumulative effects of odd combinations as respondents progress through their conjoint tasks.
4.1 Information Search
We begin by analyzing the amount of attention respondents gave to the information provided in the conjoint tables, as measured by fixation density. We are interested in whether the presence of odd combinations affects the amount of attention given to the task overall (
$H_A$
and
$H_B$
) and to specific attributes (
$H_C$
and
$H_D$
).Footnote
6
Mean number of fixations and attributes viewed per profile, across conditions.
First, we consider the amount of attention on the profiles overall across the four conditions (
$H_A$
and
$H_B$
), measuring both the mean number of eye fixations respondents employed per profile and the mean number of attributes viewed per profile, where an attribute is coded as having been viewed if there is at least one fixation on it. Figure 1 shows the results, with panels (a) and (b) showing the mean fixation density per profile and mean number of attributes viewed per profile for the candidate scenario and panels (c) and (d) showing the same for the immigrant scenario. Any differences across the conditions are minimal, indicating that the total amount of attention a respondent is willing to expend is largely unaffected by the presence of odd profiles.Footnote
7
We also perform within-condition analyses of overall attention. As noted previously, our within-condition analyses are not impacted by changes in attribute substance across conditions. Using linear regressions within each treatment condition (other than the normal condition) and for both scenarios, we regressed the following dependent variables on the number of odd combinations that are present in the profile (along with respondent fixed effects): (a) the number of attributes that are viewed (i.e., at least one fixation) and (b) the number of fixations overall. These regressions directly test
$H_A$
and
$H_B$
. Furthermore, we include additional variants of these regressions that relate to
$H_C$
and
$H_D$
(discussed more below) using the following dependent variables: the number of attributes that could be involved in odd combinations within the condition in question (whether or not the odd combination is present in a specific profile) that are viewed; the number of attributes that could not be involved in odd combinations within the condition that are viewed; the number of fixations on attributes possibly involved in odd combinations (“odd fixations”); and the number of fixations on attributes not possibly involved in odd combinations (“normal fixations”). These additional regressions allow for intuition about whether any differences in the total number of attributes viewed or fixations are due to increases and/or decreases on odd and normal attributes. As shown in Table 2, of those 36 tests, the only statistically significant results at
$p < 0.05$
were small positive effects related to fixations in the combined condition of the candidate scenario and the incongruent condition of the immigrant scenario. If these effects were robust, we would have expected to see common results across the two scenarios and, for the immigrant scenario, to see an effect in the combined condition as well as the incongruent condition.
In addition, Table 2 also provides an evaluation of the substantive magnitude of the effects. Specifically, the table displays estimates for the standardized effects, which measure the number of standard deviations increased in the outcome due to one standard deviation increase in the input (i.e., in the number of odd combinations).Footnote
8
As can be seen, all of the standardized effects are minuscule—below
$0.04$
in absolute value. Table A.6 in the Supplementary Material shows these results when pooling across both the candidate and immigrant scenarios. There are a few more estimates that reach statistical significance when pooling the data, but the standardized effects all remain similarly minuscule. In sum, there is not compelling evidence of meaningful changes in overall attention as a result of the inclusion of odd profiles, in either our between-condition or within-condition analyses.
The regressions in Table 2 also bear upon our next consideration: the possibility of attention shift toward (or away) from specific attributes involved in odd combinations (
$H_C$
and
$H_D$
), which could occur regardless of whether or not the total amount of attention is changed. We consider additional within-condition as well as between-condition analyses, beginning with the within-condition analyses. For each odd condition, the attributes for which an odd combination was possible did not see this oddness materialize in every profile due to randomization. Hence, we can differentiate between profiles in which a potentially odd combination instantiated as odd and profiles in which a potentially odd combination did not instantiate as odd. We use linear regressions for each relevant attribute within each condition and regress the number of fixations on an attribute in a profile on an indicator for whether or not that attribute was in an odd combination in the profile (along with respondent fixed effects). Once again, note that our within-condition analyses are not impacted by changes in attribute substance across conditions.
Within-condition effects of number of odd combinations in a profile, candidate and immigrant scenarios. Standardized effects correspond to the effect of a one standard deviation increase in the odd combinations on the number of standard deviations increased in the outcome.
Effect of an odd combination instantiating for an odd attribute on the number of fixations on the attribute, within condition.
Figure 2 shows the results, with the candidate and immigrant scenarios shown in panels (a) and (b), respectively. The x-axis displays the effects, and the y-axis shows the different attributes. The effects are mostly small and statistically insignificant. Further, the effects that do achieve statistical significance are substantively small and inconsistent in their directionality. Additionally, several of the significant effects are inconsistent for the attribute across all treatment conditions (e.g., the prior profession attribute in the candidate scenario, for which the attribute being involved in an odd combination negatively impacted the number of fixations in the incongruent and nonsensical conditions but not in the combined condition; theoretically, this result is illogical). To further illustrate, Figure A.17 in the Supplementary Material shows the results in terms of standardized effects, measuring the increase in the number of standard deviations of the outcome (i.e., number of fixations on the attribute in question) resulting from that attribute being in an odd combination vs. not. The largest standardized effects (in absolute value) are
$0.16$
in the candidate scenario and
$0.04$
in the immigrant scenario, and the vast majority are much smaller. For reference, a traditional rule-of-thumb value to demarcate “small” standardized treatment effect sizes in the social sciences is
$0.2$
(Cohen Reference Cohen2013).
We conduct even more tests for the sake of robustness. We find similar results when performing this analysis only for the first condition seen by respondents, as well as when we use the duration each attribute was viewed for rather than number of fixations as the outcome.Footnote 9
In our between-condition analysis, we analyze the mean number of fixations on each attribute per profile. By estimating this separately for each attribute and each condition, we investigate whether an attribute that is involved in an odd combination is likely to have more or fewer fixations than when that same attribute is not involved in an odd combination in the normal condition. Figure 3 shows the results, with panel (a) corresponding to the candidate scenario and panel (b) the immigrant scenario.Footnote 10
Number of fixations on each attribute per profile, across conditions.
In each panel, the eight attributes are ordered on the x-axis, and the y-axis measures the mean number of fixations on that attribute per profile. For each attribute, the mean number of fixations per profile is shown for all four treatment conditions (in order, the normal, incongruent, nonsensical, and combined conditions). Hence, each bar corresponds to an attribute-condition pairing. The red outlining around certain bars denotes that those particular attributes were involved in odd combinations in those conditions.Footnote 11 For instance, for the age attribute in the candidate scenario, there was a possibility that age was involved in odd combinations in the nonsensical and combined conditions but not in the normal and incongruent conditions. Meanwhile, the gender attribute was never involved in any odd combinations in the candidate scenario, which is why none of the bars for gender are outlined in red.
Hence, to consider the effect of odd combinations on attention between-condition, for each attribute one should compare the (pink) normal condition bars to the bars outlined in red (i.e., the bars corresponding to the conditions in which that particular attribute may have been involved in an odd combination). We see mostly small and statistically insignificant differences. Furthermore, for the few differences where there is statistical significance, the directions of the differences are inconsistent, with involvement in odd combinations resulting in small attention increases for some attributes and small attention decreases for other attributes.Footnote 12
The lack of substantively notable and significant results is robust to alternative analyses. Similar analyses that consider alternative measures of attention on each attribute, namely the proportion of profiles in which an attribute was viewed at all (i.e., had at least one fixation) and the amount of time spent looking at an attribute, yield similar results as shown in Figures A.15 and A.16 in the Supplementary Material.
Finally, we analyze the effect of odd combinations on sequential search patterns (
$H_E$
). As explained above, odd combinations should theoretically lead to an increase in within-profile search. To investigate, we use the search metric proposed in Böckenholt and Hynan (Reference Böckenholt and Hynan1994), which compares the number of vertical to horizontal transitions in a respondents’ gaze progression and yields a measure of a respondent’s preference for within-profile vs. within-attribute transitions.Footnote
13
The results of our analysis are reported in Section D.1 of the Supplementary Material, and once again we find little evidence that odd profiles substantially affect respondents’ information search patterns.
In sum, any evidence that odd combinations affect information search and attention is limited and inconsistent, with only substantively small and idiosyncratic effects.
4.2 Choice Behavior
Next, we analyze whether the presence of odd combinations affects respondents’ choice behavior—that is, which profiles they choose (
$H_G$
–
$H_I$
). We have already shown that the presence of oddness does not lead to systematically different search behavior, such as respondent satisficing or taking the task less seriously. Here, we provide analyses of choice behavior rather than of attention in order to assess whether respondents are somehow maintaining the same levels of attention but nonetheless shifting the salience of the attributes based on their oddness.
As described earlier in the design section, the effects of odd combinations on respondent choice are more challenging to evaluate (relative to the effects on search and attention) in terms of associating any effects exclusively with the oddness of the attribute combinations. The reason is that the introduction of oddness into a pair of attributes requires changing at least one of the attributes and/or attribute levels, and respondent choice is directly tied to the actual substance of the included attributes and levels. Hence, any change in attribute and/or attribute levels could fundamentally affect the decisions that respondents make regardless of the oddness involved.
Hence, we must be careful in interpreting any changes in choice behavior. We examine differences in the average marginal component effects (AMCEs) (Hainmueller et al. Reference Hainmueller, Hopkins and Yamamoto2014) within and between treatment conditions, where the AMCEs are estimated using a standard linear regression procedure that regresses an indicator for whether or not a profile was chosen on dummy variables corresponding to each of the levels of each of the attributes (with reference levels omitted).Footnote 14 As with all of our analyses already presented, we employ clustered standard errors (clustered by respondent). In light of the challenges described above, we begin by presenting a set of within-condition analyses.Footnote 15
First, we assess the effects of the number of odd combinations present in a profile on each of the AMCEs, irrespective of which attributes the odd combinations pertain to. We focus on the combined condition for two reasons: first, it features the largest volume of odd combinations and thus allows for the strongest test of potential effects of oddness; and second, because there is more random variation in the number of odd combinations realized per task, there is less risk that the analysis may be affected by dependencies between the number of odd combinations and specific attribute levels.Footnote 16 Figures 4 and 5 show the results for the combined condition. The left panel shows the AMCEs when profiles contained less than the median number of odd combinations, the middle panels shows the AMCEs given greater than the median number of odd combinations, and the right panel shows the differences. As can be seen, there is no evidence of any systematic or meaningful differences.Footnote 17
Effects of attributes on respondent choice (candidate scenario), combined condition, by number of odd combinations in profile, and difference.
Effects of attributes on respondent choice (immigrant scenario), combined condition, by number of odd combinations in profile, and difference.
Additionally, in Section D.2 of the Supplementary Material, we present more fine-grained analyses of these effects. Specifically, we assess the effect of specific attributes being involved in an odd combination on the AMCEs for all other attributes. We find only a handful of the AMCE differences are statistically significant, and they are so in different directions, consistent with chance statistical significance and/or idiosyncratic changes. In other words, we do not find attributes to have more or less of an effect on respondent choice depending upon the presence of odd combinations.Footnote 18
Finally, we also re-estimate the models used to compute the AMCEs in another manner. Instead of subsetting across the number of odd combinations as in Figures 4 and 5, we include the number of odd combinations present in a profile as a variable in the linear regression models in addition to the attributes.Footnote
19
We do not find the estimated effect of the number of odd combinations on respondent choice to be statistically significant at
$p < 0.05$
for any of the models, nor are the signs of the estimated effects consistent across the models either.
Next, we turn to between-condition analyses. First, we focus on a context that should be less susceptible to attribute value-driven changes. Specifically, we consider differences in choice behavior across conditions in which the levels of an attribute may change but the attributes themselves do not change, since changes in certain levels of fixed attributes are a much less significant design modification than changes of an entire attribute. Concretely, this means focusing on the difference in choice behavior between the normal and nonsensical conditions and on the difference in choice behavior between the incongruent and combined conditions, separately.
We begin with the results of the analyses comparing across the normal and nonsensical conditions, shown in Figure 6 for the candidate scenario and Figure 7 for the immigrant scenario. In each figure, the left panel shows the estimated AMCEs for the normal condition, the middle panel shows the estimated AMCEs for the nonsensical condition, and the right panel shows the difference between the two. There are minimal differences in AMCEs across treatment conditions. For both scenarios, the only differences that are statistically significant are those for which the levels themselves are different between the conditions—that is, for attribute levels displayed in parentheses in the plots, such as the prior profession of law enforcement officer (normal condition) vs. lawyer (nonsensical condition) in the candidate scenario and the profession of janitor (normal) vs. medical doctor (nonsensical) in the immigrant scenario.Footnote 20 It makes sense that due to purely attribute-specific effects, different levels will have different AMCEs, and so these differences are not directly relevant to our investigation. In contrast, the fact that there are no detectable AMCE differences for any attribute-levels that do not change between the conditions (and all the point estimates of the differences are substantively very small) provides evidence that the introduction of oddness in the nonsensical condition (relative to the normal condition) did not meaningfully alter respondents’ overall choice behavior.
Effects of attributes on respondent choice (candidate scenario), results from the normal condition, nonsensical condition, and difference.
Effects of attributes on respondent choice (immigrant scenario), results from the normal condition, nonsensical condition, and difference.
Next, we compare the AMCEs in the normal condition against the incongruent (or combined) condition. In this case, there was substitution of entire attributes (e.g., substituting gun control views in for views on the retirement age in the candidate scenario), and the AMCEs across entirely different attributes are not comparable to each other. However, as seen in Figures A.21–A.24 in the Supplementary Material for the non-substituted attributes there are almost no differences. Indeed, the differences across the normal and incongruent/combined conditions are relatively small and/or consistent with the substitution of a higher salience attribute for a lower salience attribute, leading to larger effects for that substituted high-salience attribute and subsequent attenuated effects for other attributes.Footnote 21
In other words, we see little evidence that combination oddness per se altered the choice behavior of respondents when comparing AMCEs between the treatment conditions.Footnote 22 It does not appear that respondents took the task less seriously or were distracted by the oddness of the attribute combinations. The differences that we do observe are consistent with attribute-driven changes, rather than compelling evidence of oddness-driven effects. Additionally, because the number of fixations on an attribute indicates attribute importance (Jenke et al. Reference Jenke, Bansak, Hainmueller and Hangartner2021), and there were no notable differences in the number of fixations across treatment conditions, our findings provide strong evidence that these null results are not false negatives. Ultimately, the bulk of the substantive and statistical inferences we would draw are unchanged by the introduction of odd combinations.
4.3 Cumulative Effects
As described earlier, our analyses reported above proceed under the standard assumptions in the causal inference tradition of conjoint analysis laid out by Hainmueller et al. (Reference Hainmueller, Hopkins and Yamamoto2014), which includes the assumption of no carryover effects across tasks. It may be possible, however, that exposure to odd combinations has a cumulative effect over time as respondents are exposed to more (or fewer) odd combinations over the course of many tasks. Such cumulative effects could, for instance, lead to reduced attention on subsequent profiles, such that the more odd combinations a respondent has been cumulatively exposed to at any point, the less attentive they become.
To address this consideration, we undertake supplementary analyses that directly test for cumulative effects of odd combinations by leveraging the random variation in the number of odd combinations to which different respondents are exposed over time. Note that these analyses were not pre-registered. We conduct such analyses corresponding to all of our within-condition analyses reported above. Section D.3 of the Supplementary Material describes the analyses and reports the results. In these analyses, there is once again no meaningful evidence of odd combinations influencing information search or choice behavior.
5 Discussion and Conclusions
This study has explored the implications of including odd profiles in conjoint experiments. We designed and implemented a novel set of conjoint experiments that were tailored for this investigation and which we further complemented with eye-tracking analysis. In addition, to expand the amount of theoretical and empirical ground covered, our experiment expressly included different types of odd combinations (incongruent and nonsensical) as well as an unusually high density of odd combinations, and our investigation included the effects of odd combinations on several dimensions of respondent survey-taking behavior, including their information search, attention, and choices. For each of these investigations, we conducted both within-condition and between-condition analyses, and we considered two topical scenarios that are common in political science applications of conjoint analysis (candidate choice and immigrant choice). In sum, we engaged in an extensive analysis on the effects of odd profiles. Across all of these analyses, however, we did not find systematic or compelling evidence of effects of odd combinations. Our results evince very limited, haphazard, and substantively small respondent sensitivity to odd combinations.
To be clear, we do not conclude from our results that there are no effects whatsoever, nor do we argue that researchers should create conjoint designs that feature odd profiles heedlessly, without thinking about the implications. Indeed, there are a host of considerations in profile design that researchers must contend with when deciding whether or not to include odd profiles, including substantive theory, external validity, randomization, and analytical dependencies. There is thus a practical question of how much these different competing considerations can and should be prioritized. In this study, our evidence leads us to conclude that (a) the effects of odd profiles on survey-taking behavior are substantively limited and not systematically explainable, even given the unusually high degrees of oddness we built in, and hence, (b) concern about the effects of oddness on response behavior and quality should not be the guiding factor when deciding whether or not to allow for odd attribute-level combinations in one’s conjoint design. In other words, researchers should prioritize other considerations when making those decisions—for instance, substantive theoretical considerations (e.g., interest in unpacking the independent effects of attributes like party and issues) and statistical considerations (e.g., desire for the randomization distribution to reflect an underlying target/population distribution of interest).
In addition, our results have implications for understanding how survey respondents process information in conjoint and similar survey tasks. Our finding that the presence of odd combinations does not seem to significantly affect respondents’ survey-taking behavior suggests that respondents are not putting much cognitive weight on the interplay or relationship between attributes. That is, they are not thinking much about the attributes in concert with one another but rather putting the preponderance of their cognitive effort on the consideration of the attributes individually. This is inconsistent with rational choice models of decision making such as expected value models, which posit that choices are based on global evaluations of each item (e.g., profile) individually (Riedl, Brandstätter, and Roithmayr Reference Riedl, Brandstätter and Roithmayr2008; Sleboda and Sokolowska Reference Sleboda and Sokolowska2017). Previous work in survey experiments has similarly found that respondents tend not to use rational choice models of decision-making in the context of making candidate choices (Lau and Redlawsk Reference Lau and Redlawsk2006). There are a number of alternative models that favor the use of within-attribute transitions [e.g., the lexicographic rule (Fishburn Reference Fishburn1974) or the additive difference strategy (Tversky Reference Tversky1969)]. Future research on this topic would be valuable for bolstering our understanding of respondents’ cognitive engagement with such tasks.
This result also highlights questions about conjoint experiments’ external validity that future research should address. The political science literature has dealt with some challenges along these lines (e.g., De la Cuesta et al. Reference De la Cuesta, Egami and Imai2022; Hainmueller et al. Reference Hainmueller, Hangartner and Yamamoto2015). Additionally, the marketing literature, in which conjoint experiments have been a methodological fixture for decades, has found a high correlation between the choice proportions predicted by conjoint experiments and those observed in real markets (Louviere Reference Louviere1974, Reference Louviere1988; Louviere and Woodworth Reference Louviere and Woodworth1983). Nonetheless, the fact that respondents do not appear to be thinking about attributes in an integrated fashion gives us pause. Whether this is the case for decision-makers in the real world, we do not know. On the one hand, we do not expect people to be rational choice adherents at all times in their real world decision-making. Yet we do suspect that they would sometimes consider attributes in reference to one another. And whether our conclusions on this point would hold for surveys in general, not only conjoint experiments, we also do not know. An explanation of how this result fits with the close correspondence between conjoint experimental and real world choices is key for the validation of conjoint experiments as useful in predicting real world behavior and should be a focus of future research.
There are several limitations of our study. The first is that our sample is a convenience sample that is skewed toward more educated individuals relative to the U.S. population.Footnote 23 Furthermore, because taking the survey involved eye tracking (and the accompanying equipment), it is plausible that the respondents behaved differently than survey takers would in online settings. On the one hand, this non-representativeness (in terms of both respondent characteristics and instrument modality) limits the generalizability of our results. On the other hand, however, we expect that both of these features of non-representativeness would have led the respondents in our sample to make more cognitive effort and pay more attention than typical survey-taking samples. That is, the circumstances of our study should have made for an easier test of sensitivity and reactions to odd profiles. Arguably, then, the fact that we see such limited effects in our setting is strongly suggestive of limited effects more broadly.
Would our results hold for conjoint designs with fewer attributes? It could be the case that as the ratio of odd to normal combinations increases with a smaller number of attributes, the noticeability of the odd combinations increases. But our combined condition featured a large proportion of odd combinations. These proportions are similar to the proportion that might be expected in a scenario with a smaller number of attributes. If the noticeability of odd combinations were a function of their proportion, we would expect to see more pronounced effects for this condition—but we do not.
As with all studies with null results, it is possible that there are counterbalancing effects whereby certain respondents respond in one direction, others respond to the same extent but in the opposite direction, and the sum of these effects is approximately zero. We see no demographic variables that would be theoretically likely to induce this type of effect. Another possibility is that, as with all within-subject designs, each random ordering of the conditions results in some specific cognitive processes resulting (say) from experimenter demand effects. But recent scholarship has found that within-subjects designs are generally free of such effects (Clifford, Sheagley, and Piston Reference Clifford, Sheagley and Piston2021), and we have no specific reason to expect this is an issue in the case of our experiment.
Another limitation is that we consider only two substantive scenarios, candidate and immigrant choice, out of the panoply of scenarios that have been featured in conjoint experiments in political science (and social sciences more broadly). That being said, the scenarios we considered are common in political science research, and they are also reflective of the typical level of complexity of conjoint designs that have been employed in previous scholarship. Hence, while we cannot say for certain that our results would generalize to other scenarios, we do not find any evidence that points to different expectations in any other particular scenario. For instance, if we had found any evidence of differences between the incongruent and nonsensical combinations, this would have implications for other substantive scenarios where either incongruent or nonsensical combinations might play a larger role. However, we do not find such evidence. Additionally, we would expect the presence of odd combinations in the candidate scenario to be particularly salient to U.S. citizens given the two-party system and the current context of polarization. The fact that we did not find evidence of odd combinations mattering in this scenario strengthens our confidence that the results would extend to other countries with different political systems and contexts.
Acknowledgments
The authors thank Daniel Hopkins, participants of the 2023 Annual Meeting of the Society for Political Methodology (PolMeth XL), and four anonymous reviewers for helpful comments. The authors also thank Derrick Dwamena for assistance with data collection.
Data Availability Statement
Replication materials are available via Harvard Dataverse at https://doi.org/10.7910/DVN/6PXYGY (Bansak and Jenke Reference Bansak and Jenke2024).
Ethical Standards
The authors affirm that this article adheres to the APSA’s Principles and Guidance on Human Subject Research. The human subjects research in this article was reviewed and approved by the University of Houston (
$\#$
180138) and the University of Berkeley (
$\#$
804099) Institutional Review Boards.
Supplementary Material
For supplementary material accompanying this paper, please visit https://doi.org/10.1017/pan.2025.1.
Open access
