Political scientists have increasingly deployed conjoint survey experiments to understand multidimensional choices in various settings. In this paper, we show that the average marginal component effect (AMCE) constitutes an aggregation of individual-level preferences that is meaningful both theoretically and empirically. First, extending previous results to allow for arbitrary randomization distributions, we show how the AMCE represents a summary of voters’ multidimensional preferences that combines directionality and intensity according to a probabilistic generalization of the Borda rule. We demonstrate why incorporating both the directionality and intensity of multi-attribute preferences is essential for analyzing real-world elections, in which ceteris paribus comparisons almost never occur. Second, and in further empirical support of this point, we show how this aggregation translates directly into a primary quantity of interest to election scholars: the effect of a change in an attribute on a candidate’s or party’s expected vote share. These properties hold irrespective of the heterogeneity, strength, or interactivity of voters’ preferences and regardless of how votes are aggregated into seats. Finally, we propose, formalize, and evaluate the feasibility of using conjoint data to estimate alternative quantities of interest to electoral studies, including the effect of an attribute on the probability of winning.

A critical barrier to generating cumulative knowledge in political science and related disciplines is the inability of researchers to observe the results from the full set of research designs that scholars have conceptualized, implemented, and analyzed. For a variety of reasons, studies that produce null findings are especially likely to be unobserved, creating biases in publicly accessible research. While several approaches have been suggested to overcome this problem, none have yet proven adequate. We call for the establishment of a new discipline-wide norm in which scholars post short “null results reports” online that summarize their research designs, findings, and interpretations. To address the inevitable incentive problems that earlier proposals for reform were unable to overcome, we argue that decentralized research communities can spur the broader disciplinary norm change that would bring advantage to scientific advance. To facilitate our contribution, we offer a template for these reports that incorporates evaluation of the possible explanations for the null findings, including statistical power, measurement strategy, implementation issues, spillover/contamination, and flaws in theoretical priors. We illustrate the template’s utility with two experimental studies focused on the naturalization of immigrants in the United States and attitudes toward Syrian refugees in Jordan.

We introduce a constrained priority mechanism that combines outcome-based matching from machine learning with preference-based allocation schemes common in market design. Using real-world data, we illustrate how our mechanism could be applied to the assignment of refugee families to host country locations, and kindergarteners to schools. Our mechanism allows a planner to first specify a threshold $\bar g$ for the minimum acceptable average outcome score that should be achieved by the assignment. In the refugee matching context, this score corresponds to the probability of employment, whereas in the student assignment context, it corresponds to standardized test scores. The mechanism is a priority mechanism that considers both outcomes and preferences by assigning agents (refugee families and students) based on their preferences, but subject to meeting the planner’s specified threshold. The mechanism is both strategy-proof and constrained efficient in that it always generates a matching that is not Pareto dominated by any other matching that respects the planner’s threshold.

Conjoint experiments are popular, but there is a paucity of research on respondents’ underlying decision-making processes. We leverage eye-tracking methodology and a series of conjoint experiments, administered to university students and local community members, to examine how respondents process information in conjoint surveys. There are two main findings. First, attribute importance measures inferred from the stated choice data are correlated with attribute importance measures based on eye movement. This validation test supports the interpretation of common conjoint metrics, such as average marginal component effects (AMCEs), as measures of attribute importance. Second, when we experimentally increase the number of attributes and profiles in the conjoint table, respondents view a larger absolute number of cells but a smaller fraction of the total cells displayed. Moving from two to three profiles, respondents search more within-profile, rather than within-attribute, to build summary evaluations. However, respondents’ stated choices remain fairly stable regardless of the number of attributes and profiles in the conjoint table. Together, these patterns speak to the robustness of conjoint experiments and are consistent with a bounded rationality mechanism. Respondents adapt to complexity by selectively incorporating relevant new information to focus on important attributes, while ignoring less relevant information to reduce cognitive processing costs.

Recent years have seen a renaissance of conjoint survey designs within social science. To date, however, researchers have lacked guidance on how many attributes they can include within conjoint profiles before survey satisficing leads to unacceptable declines in response quality. This paper addresses that question using pre-registered, two-stage experiments examining choices among hypothetical candidates for US Senate or hotel rooms. In each experiment, we use the first stage to identify attributes which are perceived to be uncorrelated with the attribute of interest, so that their effects are not masked by those of the core attributes. In the second stage, we randomly assign respondents to conjoint designs with varying numbers of those filler attributes. We report the results of these experiments implemented via Amazon's Mechanical Turk and Survey Sampling International. They demonstrate that our core quantities of interest are generally stable, with relatively modest increases in survey satisficing when respondents face large numbers of attributes.

Multiplicative interaction models are widely used in social science to examine whether the relationship between an outcome and an independent variable changes with a moderating variable. Current empirical practice tends to overlook two important problems. First, these models assume a linear interaction effect that changes at a constant rate with the moderator. Second, estimates of the conditional effects of the independent variable can be misleading if there is a lack of common support of the moderator. Replicating 46 interaction effects from 22 recent publications in five top political science journals, we find that these core assumptions often fail in practice, suggesting that a large portion of findings across all political science subfields based on interaction models are fragile and model dependent. We propose a checklist of simple diagnostics to assess the validity of these assumptions and offer flexible estimation strategies that allow for nonlinear interaction effects and safeguard against excessive extrapolation. These statistical routines are available in both R and STATA.

In recent years, political and social scientists have made increasing use of conjoint survey designs to study decision-making. Here, we study a consequential question which researchers confront when implementing conjoint designs: How many choice tasks can respondents perform before survey satisficing degrades response quality? To answer the question, we run a set of experiments where respondents are asked to complete as many as 30 conjoint tasks. Experiments conducted through Amazon’s Mechanical Turk and Survey Sampling International demonstrate the surprising robustness of conjoint designs, as there are detectable but quite limited increases in survey satisficing as the number of tasks increases. Our evidence suggests that in similar study contexts researchers can assign dozens of tasks without substantial declines in response quality.

We study the impact of naturalization on the long-term social integration of immigrants into the host country society. Despite ongoing debates about citizenship policy, we lack reliable evidence that isolates the causal effect of naturalization from the nonrandom selection into naturalization. We exploit the quasi-random assignment of citizenship in Swiss municipalities that used referendums to decide on naturalization applications of immigrants. Comparing otherwise similar immigrants who narrowly won or lost their naturalization referendums, we find that receiving Swiss citizenship strongly improved long-term social integration. We also find that the integration returns to naturalization are larger for more marginalized immigrant groups and when naturalization occurs earlier, rather than later in the residency period. Overall, our findings support the policy paradigm arguing that naturalization is a catalyst for improving the social integration of immigrants rather than merely the crown on the completed integration process.

We propose the use of Kernel Regularized Least Squares (KRLS) for social science modeling and inference problems. KRLS borrows from machine learning methods designed to solve regression and classification problems without relying on linearity or additivity assumptions. The method constructs a flexible hypothesis space that uses kernels as radial basis functions and finds the best-fitting surface in this space by minimizing a complexity-penalized least squares problem. We argue that the method is well-suited for social science inquiry because it avoids strong parametric assumptions, yet allows interpretation in ways analogous to generalized linear models while also permitting more complex interpretation to examine nonlinearities, interactions, and heterogeneous effects. We also extend the method in several directions to make it more effective for social inquiry, by (1) deriving estimators for the pointwise marginal effects and their variances, (2) establishing unbiasedness, consistency, and asymptotic normality of the KRLS estimator under fairly general conditions, (3) proposing a simple automated rule for choosing the kernel bandwidth, and (4) providing companion software. We illustrate the use of the method through simulations and empirical examples.

In this case study of the impact of West German television on public support for the East German communist regime, we evaluate the conventional wisdom in the democratization literature that foreign mass media undermine authoritarian rule. We exploit formerly classified survey data and a natural experiment to identify the effect of foreign media exposure using instrumental variable estimators. Contrary to conventional wisdom, East Germans exposed to West German television were more satisfied with life in East Germany and more supportive of the East German regime. To explain this surprising finding, we show that East Germans used West German television primarily as a source of entertainment. Behavioral data on regional patterns in exit visa applications and archival evidence on the reaction of the East German regime to the availability of West German television corroborate this result.

Survey experiments are a core tool for causal inference. Yet, the design of classical survey experiments prevents them from identifying which components of a multidimensional treatment are influential. Here, we show how conjoint analysis, an experimental design yet to be widely applied in political science, enables researchers to estimate the causal effects of multiple treatment components and assess several causal hypotheses simultaneously. In conjoint analysis, respondents score a set of alternatives, where each has randomly varied attributes. Here, we undertake a formal identification analysis to integrate conjoint analysis with the potential outcomes framework for causal inference. We propose a new causal estimand and show that it can be nonparametrically identified and easily estimated from conjoint data using a fully randomized design. The analysis enables us to propose diagnostic checks for the identification assumptions. We then demonstrate the value of these techniques through empirical applications to voter decision making and attitudes toward immigrants.

This paper proposes entropy balancing, a data preprocessing method to achieve covariate balance in observational studies with binary treatments. Entropy balancing relies on a maximum entropy reweighting scheme that calibrates unit weights so that the reweighted treatment and control group satisfy a potentially large set of prespecified balance conditions that incorporate information about known sample moments. Entropy balancing thereby exactly adjusts inequalities in representation with respect to the first, second, and possibly higher moments of the covariate distributions. These balance improvements can reduce model dependence for the subsequent estimation of treatment effects. The method assures that balance improves on all covariate moments included in the reweighting. It also obviates the need for continual balance checking and iterative searching over propensity score models that may stochastically balance the covariate moments. We demonstrate the use of entropy balancing with Monte Carlo simulations and empirical applications.

A large literature argues that public opinion is vulnerable to various types of framing and cue effects. However, we lack evidence on whether existing findings, which are typically based on lab experiments involving low-salience issues, travel to salient and contentious political issues in real-world voting situations. We examine the relative importance of issue frames, partisan cues, and their interaction for opinion formation using a survey experiment conducted around a highly politicized referendum on immigration policy in Switzerland. We find that voters responded to frames and cues, regardless of their direction, by increasing support for the position that is in line with their pre-existing partisan attachment. This reinforcement effect was most visible among low knowledgeable voters that identified with the party that owned the issue. These results support some of the previous findings in the political communication literature, but at the same time also point toward possible limits to framing effects in the context of salient and contested policy issues.

We study discrimination against immigrants using microlevel data from Switzerland, where, until recently, some municipalities used referendums to decide on the citizenship applications of foreign residents. We show that naturalization decisions vary dramatically with immigrants’ attributes, which we collect from official applicant descriptions that voters received before each referendum. Country of origin determines naturalization success more than any other applicant characteristic, including language skills, integration status, and economic credentials. The average proportion of “no” votes is about 40% higher for applicants from (the former) Yugoslavia and Turkey compared to observably similar applicants from richer northern and western European countries. Statistical and taste-based discrimination contribute to varying naturalization success; the rewards for economic credentials are higher for applicants from disadvantaged origins, and origin-based discrimination is much stronger in more xenophobic municipalities. Moreover, discrimination against specific immigrant groups responds dynamically to changes in the groups’ relative size.

In their article in the February 2010 issue of APSR, Jens Hainmueller and Michael J. Hiscox (2010) asserted that they had “conducted a unique survey experiment that, for the first time, explicitly and separately examine[d] individuals’ attitudes toward highly skilled and low-skilled immigrants.” That claim was in error. A prior survey experiment, also published in the American Political Science Review, in February 2004, examined attitudes toward highly skilled and low-skilled immigrants in the Netherlands and assigned respondents randomly to alternative questions (Sniderman, Hagendoorn, and Prior 2004).

Past research has emphasized two critical economic concerns that appear to generate anti-immigrant sentiment among native citizens: concerns about labor market competition and concerns about the fiscal burden on public services. We provide direct tests of both models of attitude formation using an original survey experiment embedded in a nationwide U.S. survey. The labor market competition model predicts that natives will be most opposed to immigrants who have skill levels similar to their own. We find instead that both low-skilled and highly skilled natives strongly prefer highly skilled immigrants over low-skilled immigrants, and this preference is not decreasing in natives' skill levels. The fiscal burden model anticipates that rich natives oppose low-skilled immigration more than poor natives, and that this gap is larger in states with greater fiscal exposure (in terms of immigrant access to public services). We find instead that rich and poor natives are equally opposed to low-skilled immigration in general. In states with high fiscal exposure, poor (rich) natives are more (less) opposed to low-skilled immigration than they are elsewhere. This indicates that concerns among poor natives about constraints on welfare benefits as a result of immigration are more relevant than concerns among the rich about increased taxes. Overall the results suggest that economic self-interest, at least as currently theorized, does not explain voter attitudes toward immigration. The results are consistent with alternative arguments emphasizing noneconomic concerns associated with ethnocentrism or sociotropic considerations about how the local economy as a whole may be affected by immigration.

Many recent studies show that firms profit from connections to influential politicians, but less is known about how much politicians financially benefit from wielding political influence. We estimate the returns to serving in Parliament, using original data on the estates of recently deceased British politicians. Applying both matching and a regression discontinuity design to compare Members of Parliament (MPs) with parliamentary candidates who narrowly lost, we find that serving in office almost doubled the wealth of Conservative MPs, but had no discernible financial benefits for Labour MPs. Conservative MPs profited from office largely through lucrative outside employment they acquired as a result of their political positions; we show that gaining a seat in Parliament more than tripled the probability that a Conservative politician would later serve as a director of a publicly traded firm—enough to account for a sizable portion of the wealth differential. We suggest that Labour MPs did not profit from office largely because trade unions collectively exerted sufficient control over the party and its MPs to prevent members from selling their services to other clients.

Recent studies of individual attitudes toward immigration emphasize concerns about labor-market competition as a potent source of anti-immigrant sentiment, in particular among less-educated or less-skilled citizens who fear being forced to compete for jobs with low-skilled immigrants willing to work for much lower wages. We examine new data on attitudes toward immigration available from the 2003 European Social Survey. In contrast to predictions based on conventional arguments about labor-market competition, which anticipate that individuals will oppose immigration of workers with similar skills to their own but support immigration of workers with different skill levels, we find that people with higher levels of education and occupational skills are more likely to favor immigration regardless of the skill attributes of the immigrants in question. Across Europe, higher education and higher skills mean more support for all types of immigrants. These relationships are almost identical among individuals in the labor force (that is, those competing for jobs) and those not in the labor force. Contrary to the conventional wisdom, then, the connection between the education or skill levels of individuals and views about immigration appears to have very little, if anything, to do with fears about labor-market competition. This finding is consistent with extensive economic research showing that the income and employment effects of immigration in European economies are actually very small. We find that a large component of the link between education and attitudes toward immigrants is driven by differences among individuals in cultural values and beliefs. More educated respondents are significantly less racist and place greater value on cultural diversity than do their counterparts; they are also more likely to believe that immigration generates benefits for the host economy as a whole.The authors would like to thank Beth Simmons, Shigeo Herano, Mike Tomz, James Alt, Jeffry Frieden, Ron Rogowski, Ken Scheve, Torben Iversen, Andy Baker, and Peter Gourevitch for helpful comments on earlier drafts.