Hostname: page-component-6766d58669-kl59c Total loading time: 0 Render date: 2026-05-24T07:05:15.437Z Has data issue: false hasContentIssue false
  • English
  • Français

Measuring Subgroup Preferences in Conjoint Experiments

Published online by Cambridge University Press:  07 August 2019

Thomas J. Leeper Open the ORCID record for Thomas J. Leeper [Opens in a new window] ,
Sara B. Hobolt Open the ORCID record for Sara B. Hobolt [Opens in a new window]  and
James Tilley Open the ORCID record for James Tilley [Opens in a new window]
Thomas J. Leeper*
Affiliation:
Department of Methodology, London School of Economics and Political Science, London WC2A 2AE, UK. Email: thosjleeper@gmail.com
Sara B. Hobolt
Affiliation:
Department of Government, London School of Economics and Political Science, London WC2A 2AE, UK. Email: s.b.hobolt@lse.ac.uk
James Tilley
Affiliation:
Department of Politics and International Relations, University of Oxford, Oxford OX1 3UQ, UK. Email: james.tilley@politics.ox.ac.uk
*
*Email: thosjleeper@gmail.com
Get access Rights & Permissions [Opens in a new window]

Abstract

Conjoint analysis is a common tool for studying political preferences. The method disentangles patterns in respondents’ favorability toward complex, multidimensional objects, such as candidates or policies. Most conjoints rely upon a fully randomized design to generate average marginal component effects (AMCEs). They measure the degree to which a given value of a conjoint profile feature increases, or decreases, respondents’ support for the overall profile relative to a baseline, averaging across all respondents and other features. While the AMCE has a clear causal interpretation (about the effect of features), most published conjoint analyses also use AMCEs to describe levels of favorability. This often means comparing AMCEs among respondent subgroups. We show that using conditional AMCEs to describe the degree of subgroup agreement can be misleading as regression interactions are sensitive to the reference category used in the analysis. This leads to inferences about subgroup differences in preferences that have arbitrary sign, size, and significance. We demonstrate the problem using examples drawn from published articles and provide suggestions for improved reporting and interpretation using marginal means and an omnibus F-test. Given the accelerating use of these designs in political science, we offer advice for best practice in analysis and presentation of results.

Keywords

conjoint analysissurvey experimentsfactorial experimentssurvey design

Information

Type
Articles
Information
Political Analysis , Volume 28 , Issue 2 , April 2020 , pp. 207 - 221
Copyright
Copyright © The Author(s) 2019. Published by Cambridge University Press on behalf of the Society for Political Methodology.

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

Authors’ note: We thank Benjamin Lauderdale, Jamie Druckman, Yusaku Horiuchi, the editor, and anonymous reviewers for feedback on this manuscript. Replication data and code for this article are available from the Political Analysis Dataverse: https://doi.org/10.7910/DVN/ARHZU4. This work was funded, in part, by the United Kingdom Economic and Social Research Council (Grant ES/R000573/1).

Contributing Editor: Jeff Gill

References

Ballard-Rosa, C., Martin, L., and Scheve, K.. 2016. “The Structure of American Income Tax Policy Preferences.” The Journal of Politics 79(1):1–16.Google Scholar
Bansak, K., Hainmueller, J., and Hangartner, D.. 2016. “How Economic, Humanitarian, and Religious Concerns Shape European Attitudes Toward Asylum Seekers.” Science 354(6309):217–222.Google Scholar
Bechtel, M. M., and Scheve, K. F.. 2013. “Mass Support for Global Climate Agreements Depends on Institutional Design.” Proceedings of the National Academy of Sciences 110(34):13763–13768.Google Scholar
Bechtel, M. M., Genovese, F., and Scheve, K. F.. 2017. “Interests, Norms and Support for the Provision of Global Public Goods: The Case of Climate Co-operation.” British Journal of Political Science, forthcoming.Google Scholar
Bechtel, M. M., Hainmueller, J., and Margalit, Y.. 2017. “Policy Design and Domestic Support for International Bailouts.” European Journal of Political Research 56(4):864–886.Google Scholar
Cairo, A. 2016. The Truthful Art. Indianapolis, IN: New Riders.Google Scholar
Campbell, R., Cowley, P., Vivyan, N., and Wagner, M.. 2019. “Legislator Dissent as a Valence Signal.” British Journal of Political Science 49(1):105–128.Google Scholar
Carey, J. M., Carman, K. R., Clayton, K. P., Horiuchim, Y., Htun, M., and Ortiz, B.. 2018. “Who Wants to Hire a More Diverse Faculty? A Conjoint Analysis of Faculty and Student Preferences for Gender and Racial/Ethnic Diversity.” Politics, Groups, and Identities, forthcoming.Google Scholar
Carlson, E. 2015. “Ethnic Voting and Accountability in Africa: A Choice Experiment in Uganda.” World Politics 67(2):353–385.Google Scholar
Carnes, N., and Lupu, N.. 2016. “Do Voters Dislike Working-Class Candidates? Voter Biases and the Descriptive Underrepresentation of the Working Class.” American Political Science Review 110(04):832–844.Google Scholar
Clayton, K., Ferwerda, J., and Horiuchi, Y.. 2019. “Exposure to Immigration and Admission Preferences: Evidence from France.” Political Behavior, forthcoming.Google Scholar
Druckman, J. N., Green, D. P., Kuklinski, J. H., and Lupia, A.. 2006. “The Growth and Development of Experimental Research in Political Science.” American Political Science Review 100(4):627–635.Google Scholar
Egami, N., and Imai, K.. 2018. “Causal Interaction in Factorial Experiments: Application to Conjoint Analysis.” Journal of the American Statistical Association 114(526):529–540.Google Scholar
Eggers, A. C., Vivyan, N., and Wagner, M.. 2018. “Corruption, Accountability, and Gender: Do Female Politicians Face Higher Standards in Public Life?” The Journal of Politics 80(1):321–326.Google Scholar
Franchino, F., and Zucchini, F.. 2014. “Voting in a Multi-dimensional Space: A Conjoint Analysis Employing Valence and Ideology Attributes of Candidates.” Political Science Research and Methods 3(2):221–241.Google Scholar
Gaines, B. J., Kuklinski, J. H., and Quirk, P. J.. 2007. “The Logic of the Survey Experiment Reexamined.” Political Analysis 15(1):1–20.Google Scholar
Gallego, A., and Marx, P.. 2017. “Multi-Dimensional Preferences for Labour Market Reforms: A Conjoint Experiment.” Journal of European Public Policy 24(7):1027–1047.Google Scholar
Green, D. P., and Kern, H. L.. 2012. “Modeling Heterogeneous Treatment Effects in Survey Experiments with Bayesian Additive Regression Trees.” Public Opinion Quarterly 76(3):491–511.Google Scholar
Grimmer, J., Messing, S., and Westwood, S. J.. 2017. “Estimating Heterogeneous Treatment Effects and the Effects of Heterogeneous Treatments with Ensemble Methods.” Political Analysis 25(4):413–434.Google Scholar
Hainmueller, J., and Hopkins, D. J.. 2015. “The Hidden American Immigration Consensus: A Conjoint Analysis of Attitudes toward Immigrants.” American Journal of Political Science 59(3):529–548.Google Scholar
Hainmueller, J., Hopkins, D. J., and Yamamoto, T.. 2014. “Causal Inference in Conjoint Analysis: Understanding Multi-Dimensional Choices via Stated Preference Experiments.” Political Analysis 22:1–30.Google Scholar
Hankinson, M. 2018. “When Do Renters Behave Like Homeowners? High Rent, Price Anxiety, and NIMBYism.” American Political Science Review 112(3):473–493.Google Scholar
Hansen, K. M., Olsen, A. L., and Bech, M.. 2014. “Cross-National Yardstick Comparisons: A Choice Experiment on a Forgotten Voter Heuristic.” Political Behavior 37(4):767–789.Google Scholar
Kirkland, P. A., and Coppock, A.. 2017. “Candidate Choice Without Party Labels.” Political Behavior 40(3):571–591.Google Scholar
Leeper, T. J.2018. Cregg: Simple Conjoint Analyses and Visualization. R package version 0.2.1.Google Scholar
Leeper, T. J., Hobolt, S. B., and Tilley, J.. 2019. “Replication Data for ‘Measuring Subgroup Preferences in Conjoint Experiments’.” https://doi.org/10.7910/DVN/ARHZU4, Harvard Dataverse, V1, UNF:6:AJX/mXwKNxNKsqJ7KMgTHw== [fileUNF].Google Scholar
Mummolo, J. 2016. “News from the Other Side: How Topic Relevance Limits the Prevalence of Partisan Selective Exposure.” The Journal of Politics 78(3):763–773.Google Scholar
Mummolo, J., and Nall, C.. 2017. “Why Partisans Do Not Sort: The Constraints on Political Segregation.” The Journal of Politics 79(1):45–59.Google Scholar
Mutz, D. C. 2011. Population-Based Survey Experiments. Princeton, NJ: Princeton University Press.Google Scholar
Oliveros, V., and Schuster, C.. 2018. “Merit, Tenure, and Bureaucratic Behavior: Evidence From a Conjoint Experiment in the Dominican Republic.” Comparative Political Studies 51(6):759–792.Google Scholar
Ratkovic, M., and Tingley, D.. 2017. “Sparse Estimation and Uncertainty with Application to Subgroup Analysis.” Political Analysis 25(1):1–40.Google Scholar
Sen, M. 2017. “How Political Signals Affect Public Support for Judicial Nominations.” Political Research Quarterly 70(2):374–393.Google Scholar
Shmueli, G. 2010. “To Explain or to Predict?” Statistical Science 25(3):289–310.Google Scholar
Sniderman, P. M. 2011. “The Logic and Design of the Survey Experiment: An Autobiography of a Methodological Innovation.” In Cambridge Handbook of Experimental Political Science, edited by Druckman, J. N., Green, D. P., Kuklinski, J. H., and Lupia, A., New York: Cambridge University Press.Google Scholar
Sobolewska, M., Galandini, S., and Lessard-Phillips, L.. 2017. “The Public View of Immigrant Integration: Multidimensional and Consensual: Evidence from Survey Experiments in the UK and the Netherlands.” Journal of Ethnic and Migration Studies 43(1):58–79.Google Scholar
Teele, D. L., Kalla, J., and Rosenbluth, F.. 2018. “The Ties That Double Bind: Social Roles and Women’s Underrepresentation in Politics.” American Political Science Review 112(3):525–541.Google Scholar
Vivyan, N., and Wagner, M.. 2016. “House or Home? Constituent Preferences over Legislator Effort Allocation.” European Journal of Political Research 55(1):81–99.Google Scholar
Wright, M., Levy, M., and Citrin, J.. 2016. “Public Attitudes Toward Immigration Policy Across the Legal/Illegal Divide: The Role of Categorical and Attribute-Based Decision-Making.” Political Behavior 38(1):229–253.Google Scholar
Xie, Y. 2015. Dynamic Documents with R and Knitr. 2nd ed.Boca Raton, Florida: Chapman and Hall/CRC. ISBN 978-1498716963.Google Scholar
Supplementary material: File

Leeper et al. supplementary material

Leeper et al. supplementary material 1

Download Leeper et al. supplementary material(File)
File 227.1 KB