Skip to main content Accessibility help
×
Home
Hostname: page-component-559fc8cf4f-sbc4w Total loading time: 0.25 Render date: 2021-03-06T06:36:18.738Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "metricsAbstractViews": false, "figures": false, "newCiteModal": false, "newCitedByModal": true }

Measuring Subgroup Preferences in Conjoint Experiments

Published online by Cambridge University Press:  07 August 2019

Thomas J. Leeper
Affiliation:
Department of Methodology, London School of Economics and Political Science, London WC2A 2AE, UK. Email: thosjleeper@gmail.com
Sara B. Hobolt
Affiliation:
Department of Government, London School of Economics and Political Science, London WC2A 2AE, UK. Email: s.b.hobolt@lse.ac.uk
James Tilley
Affiliation:
Department of Politics and International Relations, University of Oxford, Oxford OX1 3UQ, UK. Email: james.tilley@politics.ox.ac.uk

Abstract

Conjoint analysis is a common tool for studying political preferences. The method disentangles patterns in respondents’ favorability toward complex, multidimensional objects, such as candidates or policies. Most conjoints rely upon a fully randomized design to generate average marginal component effects (AMCEs). They measure the degree to which a given value of a conjoint profile feature increases, or decreases, respondents’ support for the overall profile relative to a baseline, averaging across all respondents and other features. While the AMCE has a clear causal interpretation (about the effect of features), most published conjoint analyses also use AMCEs to describe levels of favorability. This often means comparing AMCEs among respondent subgroups. We show that using conditional AMCEs to describe the degree of subgroup agreement can be misleading as regression interactions are sensitive to the reference category used in the analysis. This leads to inferences about subgroup differences in preferences that have arbitrary sign, size, and significance. We demonstrate the problem using examples drawn from published articles and provide suggestions for improved reporting and interpretation using marginal means and an omnibus F-test. Given the accelerating use of these designs in political science, we offer advice for best practice in analysis and presentation of results.

Type
Articles
Copyright
Copyright © The Author(s) 2019. Published by Cambridge University Press on behalf of the Society for Political Methodology.

Access options

Get access to the full version of this content by using one of the access options below.

Footnotes

Authors’ note: We thank Benjamin Lauderdale, Jamie Druckman, Yusaku Horiuchi, the editor, and anonymous reviewers for feedback on this manuscript. Replication data and code for this article are available from the Political Analysis Dataverse: https://doi.org/10.7910/DVN/ARHZU4. This work was funded, in part, by the United Kingdom Economic and Social Research Council (Grant ES/R000573/1).

Contributing Editor: Jeff Gill

References

Ballard-Rosa, C., Martin, L., and Scheve, K.. 2016. “The Structure of American Income Tax Policy Preferences.” The Journal of Politics 79(1):116.CrossRefGoogle Scholar
Bansak, K., Hainmueller, J., and Hangartner, D.. 2016. “How Economic, Humanitarian, and Religious Concerns Shape European Attitudes Toward Asylum Seekers.” Science 354(6309):217222.CrossRefGoogle ScholarPubMed
Bechtel, M. M., and Scheve, K. F.. 2013. “Mass Support for Global Climate Agreements Depends on Institutional Design.” Proceedings of the National Academy of Sciences 110(34):1376313768.CrossRefGoogle ScholarPubMed
Bechtel, M. M., Genovese, F., and Scheve, K. F.. 2017. “Interests, Norms and Support for the Provision of Global Public Goods: The Case of Climate Co-operation.” British Journal of Political Science, forthcoming.Google Scholar
Bechtel, M. M., Hainmueller, J., and Margalit, Y.. 2017. “Policy Design and Domestic Support for International Bailouts.” European Journal of Political Research 56(4):864886.CrossRefGoogle Scholar
Cairo, A. 2016. The Truthful Art. Indianapolis, IN: New Riders.Google Scholar
Campbell, R., Cowley, P., Vivyan, N., and Wagner, M.. 2019. “Legislator Dissent as a Valence Signal.” British Journal of Political Science 49(1):105128.CrossRefGoogle Scholar
Carey, J. M., Carman, K. R., Clayton, K. P., Horiuchim, Y., Htun, M., and Ortiz, B.. 2018. “Who Wants to Hire a More Diverse Faculty? A Conjoint Analysis of Faculty and Student Preferences for Gender and Racial/Ethnic Diversity.” Politics, Groups, and Identities, forthcoming.CrossRefGoogle Scholar
Carlson, E. 2015. “Ethnic Voting and Accountability in Africa: A Choice Experiment in Uganda.” World Politics 67(2):353385.CrossRefGoogle Scholar
Carnes, N., and Lupu, N.. 2016. “Do Voters Dislike Working-Class Candidates? Voter Biases and the Descriptive Underrepresentation of the Working Class.” American Political Science Review 110(04):832844.CrossRefGoogle Scholar
Clayton, K., Ferwerda, J., and Horiuchi, Y.. 2019. “Exposure to Immigration and Admission Preferences: Evidence from France.” Political Behavior, forthcoming.CrossRefGoogle Scholar
Druckman, J. N., Green, D. P., Kuklinski, J. H., and Lupia, A.. 2006. “The Growth and Development of Experimental Research in Political Science.” American Political Science Review 100(4):627635.CrossRefGoogle Scholar
Egami, N., and Imai, K.. 2018. “Causal Interaction in Factorial Experiments: Application to Conjoint Analysis.” Journal of the American Statistical Association 114(526):529540.CrossRefGoogle Scholar
Eggers, A. C., Vivyan, N., and Wagner, M.. 2018. “Corruption, Accountability, and Gender: Do Female Politicians Face Higher Standards in Public Life?The Journal of Politics 80(1):321326.CrossRefGoogle Scholar
Franchino, F., and Zucchini, F.. 2014. “Voting in a Multi-dimensional Space: A Conjoint Analysis Employing Valence and Ideology Attributes of Candidates.” Political Science Research and Methods 3(2):221241.CrossRefGoogle Scholar
Gaines, B. J., Kuklinski, J. H., and Quirk, P. J.. 2007. “The Logic of the Survey Experiment Reexamined.” Political Analysis 15(1):120.CrossRefGoogle Scholar
Gallego, A., and Marx, P.. 2017. “Multi-Dimensional Preferences for Labour Market Reforms: A Conjoint Experiment.” Journal of European Public Policy 24(7):10271047.CrossRefGoogle Scholar
Green, D. P., and Kern, H. L.. 2012. “Modeling Heterogeneous Treatment Effects in Survey Experiments with Bayesian Additive Regression Trees.” Public Opinion Quarterly 76(3):491511.CrossRefGoogle Scholar
Grimmer, J., Messing, S., and Westwood, S. J.. 2017. “Estimating Heterogeneous Treatment Effects and the Effects of Heterogeneous Treatments with Ensemble Methods.” Political Analysis 25(4):413434.CrossRefGoogle Scholar
Hainmueller, J., and Hopkins, D. J.. 2015. “The Hidden American Immigration Consensus: A Conjoint Analysis of Attitudes toward Immigrants.” American Journal of Political Science 59(3):529548.CrossRefGoogle Scholar
Hainmueller, J., Hopkins, D. J., and Yamamoto, T.. 2014. “Causal Inference in Conjoint Analysis: Understanding Multi-Dimensional Choices via Stated Preference Experiments.” Political Analysis 22:130.CrossRefGoogle Scholar
Hankinson, M. 2018. “When Do Renters Behave Like Homeowners? High Rent, Price Anxiety, and NIMBYism.” American Political Science Review 112(3):473493.CrossRefGoogle Scholar
Hansen, K. M., Olsen, A. L., and Bech, M.. 2014. “Cross-National Yardstick Comparisons: A Choice Experiment on a Forgotten Voter Heuristic.” Political Behavior 37(4):767789.CrossRefGoogle Scholar
Kirkland, P. A., and Coppock, A.. 2017. “Candidate Choice Without Party Labels.” Political Behavior 40(3):571591.CrossRefGoogle Scholar
Leeper, T. J.2018. Cregg: Simple Conjoint Analyses and Visualization. R package version 0.2.1.Google Scholar
Leeper, T. J., Hobolt, S. B., and Tilley, J.. 2019. “Replication Data for ‘Measuring Subgroup Preferences in Conjoint Experiments’.” https://doi.org/10.7910/DVN/ARHZU4, Harvard Dataverse, V1, UNF:6:AJX/mXwKNxNKsqJ7KMgTHw== [fileUNF].CrossRefGoogle Scholar
Mummolo, J. 2016. “News from the Other Side: How Topic Relevance Limits the Prevalence of Partisan Selective Exposure.” The Journal of Politics 78(3):763773.CrossRefGoogle Scholar
Mummolo, J., and Nall, C.. 2017. “Why Partisans Do Not Sort: The Constraints on Political Segregation.” The Journal of Politics 79(1):4559.CrossRefGoogle Scholar
Mutz, D. C. 2011. Population-Based Survey Experiments. Princeton, NJ: Princeton University Press.Google Scholar
Oliveros, V., and Schuster, C.. 2018. “Merit, Tenure, and Bureaucratic Behavior: Evidence From a Conjoint Experiment in the Dominican Republic.” Comparative Political Studies 51(6):759792.CrossRefGoogle Scholar
Ratkovic, M., and Tingley, D.. 2017. “Sparse Estimation and Uncertainty with Application to Subgroup Analysis.” Political Analysis 25(1):140.CrossRefGoogle Scholar
Sen, M. 2017. “How Political Signals Affect Public Support for Judicial Nominations.” Political Research Quarterly 70(2):374393.CrossRefGoogle Scholar
Shmueli, G. 2010. “To Explain or to Predict?Statistical Science 25(3):289310.CrossRefGoogle Scholar
Sniderman, P. M. 2011. “The Logic and Design of the Survey Experiment: An Autobiography of a Methodological Innovation.” In Cambridge Handbook of Experimental Political Science, edited by Druckman, J. N., Green, D. P., Kuklinski, J. H., and Lupia, A., New York: Cambridge University Press.Google Scholar
Sobolewska, M., Galandini, S., and Lessard-Phillips, L.. 2017. “The Public View of Immigrant Integration: Multidimensional and Consensual: Evidence from Survey Experiments in the UK and the Netherlands.” Journal of Ethnic and Migration Studies 43(1):5879.CrossRefGoogle Scholar
Teele, D. L., Kalla, J., and Rosenbluth, F.. 2018. “The Ties That Double Bind: Social Roles and Women’s Underrepresentation in Politics.” American Political Science Review 112(3):525541.CrossRefGoogle Scholar
Vivyan, N., and Wagner, M.. 2016. “House or Home? Constituent Preferences over Legislator Effort Allocation.” European Journal of Political Research 55(1):8199.CrossRefGoogle Scholar
Wright, M., Levy, M., and Citrin, J.. 2016. “Public Attitudes Toward Immigration Policy Across the Legal/Illegal Divide: The Role of Categorical and Attribute-Based Decision-Making.” Political Behavior 38(1):229253.CrossRefGoogle Scholar
Xie, Y. 2015. Dynamic Documents with R and Knitr. 2nd ed.Boca Raton, Florida: Chapman and Hall/CRC. ISBN 978-1498716963.Google Scholar

Leeper et al. supplementary material

Leeper et al. supplementary material 1

File 227 KB

Altmetric attention score

Full text views

Full text views reflects PDF downloads, PDFs sent to Google Drive, Dropbox and Kindle and HTML full text views.

Total number of HTML views: 355
Total number of PDF views: 1549 *
View data table for this chart

* Views captured on Cambridge Core between 07th August 2019 - 6th March 2021. This data will be updated every 24 hours.

Send article to Kindle

To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Measuring Subgroup Preferences in Conjoint Experiments
Available formats
×

Send article to Dropbox

To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

Measuring Subgroup Preferences in Conjoint Experiments
Available formats
×

Send article to Google Drive

To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

Measuring Subgroup Preferences in Conjoint Experiments
Available formats
×
×

Reply to: Submit a response


Your details


Conflicting interests

Do you have any conflicting interests? *