Skip to main content Accessibility help
×
Home

Measuring Subgroup Preferences in Conjoint Experiments

  • Thomas J. Leeper (a1), Sara B. Hobolt (a2) and James Tilley (a3)

Abstract

Conjoint analysis is a common tool for studying political preferences. The method disentangles patterns in respondents’ favorability toward complex, multidimensional objects, such as candidates or policies. Most conjoints rely upon a fully randomized design to generate average marginal component effects (AMCEs). They measure the degree to which a given value of a conjoint profile feature increases, or decreases, respondents’ support for the overall profile relative to a baseline, averaging across all respondents and other features. While the AMCE has a clear causal interpretation (about the effect of features), most published conjoint analyses also use AMCEs to describe levels of favorability. This often means comparing AMCEs among respondent subgroups. We show that using conditional AMCEs to describe the degree of subgroup agreement can be misleading as regression interactions are sensitive to the reference category used in the analysis. This leads to inferences about subgroup differences in preferences that have arbitrary sign, size, and significance. We demonstrate the problem using examples drawn from published articles and provide suggestions for improved reporting and interpretation using marginal means and an omnibus F-test. Given the accelerating use of these designs in political science, we offer advice for best practice in analysis and presentation of results.

Copyright

Corresponding author

Footnotes

Hide All

Authors’ note: We thank Benjamin Lauderdale, Jamie Druckman, Yusaku Horiuchi, the editor, and anonymous reviewers for feedback on this manuscript. Replication data and code for this article are available from the Political Analysis Dataverse: https://doi.org/10.7910/DVN/ARHZU4. This work was funded, in part, by the United Kingdom Economic and Social Research Council (Grant ES/R000573/1).

Contributing Editor: Jeff Gill

Footnotes

References

Hide All
Ballard-Rosa, C., Martin, L., and Scheve, K.. 2016. “The Structure of American Income Tax Policy Preferences.” The Journal of Politics 79(1):116.
Bansak, K., Hainmueller, J., and Hangartner, D.. 2016. “How Economic, Humanitarian, and Religious Concerns Shape European Attitudes Toward Asylum Seekers.” Science 354(6309):217222.
Bechtel, M. M., and Scheve, K. F.. 2013. “Mass Support for Global Climate Agreements Depends on Institutional Design.” Proceedings of the National Academy of Sciences 110(34):1376313768.
Bechtel, M. M., Genovese, F., and Scheve, K. F.. 2017. “Interests, Norms and Support for the Provision of Global Public Goods: The Case of Climate Co-operation.” British Journal of Political Science , forthcoming.
Bechtel, M. M., Hainmueller, J., and Margalit, Y.. 2017. “Policy Design and Domestic Support for International Bailouts.” European Journal of Political Research 56(4):864886.
Cairo, A. 2016. The Truthful Art . Indianapolis, IN: New Riders.
Campbell, R., Cowley, P., Vivyan, N., and Wagner, M.. 2019. “Legislator Dissent as a Valence Signal.” British Journal of Political Science 49(1):105128.
Carey, J. M., Carman, K. R., Clayton, K. P., Horiuchim, Y., Htun, M., and Ortiz, B.. 2018. “Who Wants to Hire a More Diverse Faculty? A Conjoint Analysis of Faculty and Student Preferences for Gender and Racial/Ethnic Diversity.” Politics, Groups, and Identities , forthcoming.
Carlson, E. 2015. “Ethnic Voting and Accountability in Africa: A Choice Experiment in Uganda.” World Politics 67(2):353385.
Carnes, N., and Lupu, N.. 2016. “Do Voters Dislike Working-Class Candidates? Voter Biases and the Descriptive Underrepresentation of the Working Class.” American Political Science Review 110(04):832844.
Clayton, K., Ferwerda, J., and Horiuchi, Y.. 2019. “Exposure to Immigration and Admission Preferences: Evidence from France.” Political Behavior , forthcoming.
Druckman, J. N., Green, D. P., Kuklinski, J. H., and Lupia, A.. 2006. “The Growth and Development of Experimental Research in Political Science.” American Political Science Review 100(4):627635.
Egami, N., and Imai, K.. 2018. “Causal Interaction in Factorial Experiments: Application to Conjoint Analysis.” Journal of the American Statistical Association 114(526):529540.
Eggers, A. C., Vivyan, N., and Wagner, M.. 2018. “Corruption, Accountability, and Gender: Do Female Politicians Face Higher Standards in Public Life? The Journal of Politics 80(1):321326.
Franchino, F., and Zucchini, F.. 2014. “Voting in a Multi-dimensional Space: A Conjoint Analysis Employing Valence and Ideology Attributes of Candidates.” Political Science Research and Methods 3(2):221241.
Gaines, B. J., Kuklinski, J. H., and Quirk, P. J.. 2007. “The Logic of the Survey Experiment Reexamined.” Political Analysis 15(1):120.
Gallego, A., and Marx, P.. 2017. “Multi-Dimensional Preferences for Labour Market Reforms: A Conjoint Experiment.” Journal of European Public Policy 24(7):10271047.
Green, D. P., and Kern, H. L.. 2012. “Modeling Heterogeneous Treatment Effects in Survey Experiments with Bayesian Additive Regression Trees.” Public Opinion Quarterly 76(3):491511.
Grimmer, J., Messing, S., and Westwood, S. J.. 2017. “Estimating Heterogeneous Treatment Effects and the Effects of Heterogeneous Treatments with Ensemble Methods.” Political Analysis 25(4):413434.
Hainmueller, J., and Hopkins, D. J.. 2015. “The Hidden American Immigration Consensus: A Conjoint Analysis of Attitudes toward Immigrants.” American Journal of Political Science 59(3):529548.
Hainmueller, J., Hopkins, D. J., and Yamamoto, T.. 2014. “Causal Inference in Conjoint Analysis: Understanding Multi-Dimensional Choices via Stated Preference Experiments.” Political Analysis 22:130.
Hankinson, M. 2018. “When Do Renters Behave Like Homeowners? High Rent, Price Anxiety, and NIMBYism.” American Political Science Review 112(3):473493.
Hansen, K. M., Olsen, A. L., and Bech, M.. 2014. “Cross-National Yardstick Comparisons: A Choice Experiment on a Forgotten Voter Heuristic.” Political Behavior 37(4):767789.
Kirkland, P. A., and Coppock, A.. 2017. “Candidate Choice Without Party Labels.” Political Behavior 40(3):571591.
Leeper, T. J.2018. Cregg: Simple Conjoint Analyses and Visualization. R package version 0.2.1.
Leeper, T. J., Hobolt, S. B., and Tilley, J.. 2019. “Replication Data for ‘Measuring Subgroup Preferences in Conjoint Experiments’.” https://doi.org/10.7910/DVN/ARHZU4, Harvard Dataverse, V1, UNF:6:AJX/mXwKNxNKsqJ7KMgTHw== [fileUNF].
Mummolo, J. 2016. “News from the Other Side: How Topic Relevance Limits the Prevalence of Partisan Selective Exposure.” The Journal of Politics 78(3):763773.
Mummolo, J., and Nall, C.. 2017. “Why Partisans Do Not Sort: The Constraints on Political Segregation.” The Journal of Politics 79(1):4559.
Mutz, D. C. 2011. Population-Based Survey Experiments . Princeton, NJ: Princeton University Press.
Oliveros, V., and Schuster, C.. 2018. “Merit, Tenure, and Bureaucratic Behavior: Evidence From a Conjoint Experiment in the Dominican Republic.” Comparative Political Studies 51(6):759792.
Ratkovic, M., and Tingley, D.. 2017. “Sparse Estimation and Uncertainty with Application to Subgroup Analysis.” Political Analysis 25(1):140.
Sen, M. 2017. “How Political Signals Affect Public Support for Judicial Nominations.” Political Research Quarterly 70(2):374393.
Shmueli, G. 2010. “To Explain or to Predict? Statistical Science 25(3):289310.
Sniderman, P. M. 2011. “The Logic and Design of the Survey Experiment: An Autobiography of a Methodological Innovation.” In Cambridge Handbook of Experimental Political Science , edited by Druckman, J. N., Green, D. P., Kuklinski, J. H., and Lupia, A., New York: Cambridge University Press.
Sobolewska, M., Galandini, S., and Lessard-Phillips, L.. 2017. “The Public View of Immigrant Integration: Multidimensional and Consensual: Evidence from Survey Experiments in the UK and the Netherlands.” Journal of Ethnic and Migration Studies 43(1):5879.
Teele, D. L., Kalla, J., and Rosenbluth, F.. 2018. “The Ties That Double Bind: Social Roles and Women’s Underrepresentation in Politics.” American Political Science Review 112(3):525541.
Vivyan, N., and Wagner, M.. 2016. “House or Home? Constituent Preferences over Legislator Effort Allocation.” European Journal of Political Research 55(1):8199.
Wright, M., Levy, M., and Citrin, J.. 2016. “Public Attitudes Toward Immigration Policy Across the Legal/Illegal Divide: The Role of Categorical and Attribute-Based Decision-Making.” Political Behavior 38(1):229253.
Xie, Y. 2015. Dynamic Documents with R and Knitr . 2nd ed. Boca Raton, Florida: Chapman and Hall/CRC. ISBN 978-1498716963.
MathJax
MathJax is a JavaScript display engine for mathematics. For more information see http://www.mathjax.org.

Keywords

Type Description Title
UNKNOWN
Supplementary materials

Leeper et al. supplementary material
Leeper et al. supplementary material 1

 Unknown (227 KB)
227 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed