Hostname: page-component-5b777bbd6c-7mr9c Total loading time: 0 Render date: 2025-06-18T05:10:11.765Z Has data issue: false hasContentIssue false
  • English
  • Français

Improving the External Validity of Conjoint Analysis: The Essential Role of Profile Distribution

Published online by Cambridge University Press:  14 January 2021

Brandon de la Cuesta Open the ORCID record for Brandon de la Cuesta [Opens in a new window] ,
Naoki Egami Open the ORCID record for Naoki Egami [Opens in a new window]  and
Kosuke Imai Open the ORCID record for Kosuke Imai [Opens in a new window]
Brandon de la Cuesta
Affiliation:
Postdoctoral Research Fellow, King Center on Global Development, Stanford University, Palo Alto, CA94305, USA. Email: brandon.delacuesta@stanford.edu, URL: https://brandondelacuesta.com
Naoki Egami*
Affiliation:
Assistant Professor, Department of Political Science, Columbia University, New York, NY10027, USA. Email: naoki.egami@columbia.edu, URL: https://naokiegami.com
Kosuke Imai
Affiliation:
Professor, Department of Government and Department of Statistics, Harvard University, 1737 Cambridge Street, Institute for Quantitative Social Science, Cambridge, MA02138, USA. Email: imai@harvard.edu, URL: https://imai.fas.harvard.edu
*
Corresponding author Naoki Egami
Get access Rights & Permissions [Opens in a new window]

Abstract

Conjoint analysis has become popular among social scientists for measuring multidimensional preferences. When analyzing such experiments, researchers often focus on the average marginal component effect (AMCE), which represents the causal effect of a single profile attribute while averaging over the remaining attributes. What has been overlooked, however, is the fact that the AMCE critically relies upon the distribution of the other attributes used for the averaging. Although most experiments employ the uniform distribution, which equally weights each profile, both the actual distribution of profiles in the real world and the distribution of theoretical interest are often far from uniform. This mismatch can severely compromise the external validity of conjoint analysis. We empirically demonstrate that estimates of the AMCE can be substantially different when averaging over the target profile distribution instead of uniform. We propose new experimental designs and estimation methods that incorporate substantive knowledge about the profile distribution. We illustrate our methodology through two empirical applications, one using a real-world distribution and the other based on a counterfactual distribution motivated by a theoretical consideration. The proposed methodology is implemented through an open-source software package.

Keywords

causal inferenceconjoint analysisfactorial experimentsexternal validity
Type
Article
Information
Political Analysis , Volume 30 , Issue 1 , January 2022 , pp. 19 - 45
Copyright
© The Author(s) 2021. Published by Cambridge University Press on behalf of the Society for Political Methodology

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

Edited by Jeff Gill

Authors’ note: The proposed methodology is implemented via an open-source software R package factorEx, available through the Comprehensive R Archive Network (https://cran.r-project.org/package=factorEx).

References

Arrow, K. J. 1998. “What Has Economics to Say about Racial Discrimination?” The Journal of Economic Perspectives 12(2):91–100.CrossRefGoogle Scholar
Ballard-Rosa, C., Martin, L., and Scheve, K.. 2017. “The Structure of American Income Tax Policy Preferences.” The Journal of Politics 79(1):1–16.CrossRefGoogle Scholar
Bansak, K., Hainmueller, J., and Hangartner, D.. 2016. “How Economic, Humanitarian, and Religious Concerns Shape European Attitudes Toward Asylum Seekers.” Science 354(6309):217–222.CrossRefGoogle ScholarPubMed
Barnes, L., Blumenau, J., and Lauderdale, B.. 2019. “Measuring Attitudes towards Public Spending using a Multivariate Tax Summary Experiment.” Technical report, University College London.Google Scholar
Bartels, L. M. 2000. “Partisanship and Voting Behavior, 1952–1996.” American Journal of Political Science 44(1):35–50.CrossRefGoogle Scholar
Blair, G., Cooper, J., Coppock, A., and Humphreys, M.. 2019. “Declaring and Diagnosing Research Designs.” American Political Science Review 113(3):838–859.CrossRefGoogle ScholarPubMed
Bolsen, T., Druckman, J. N., and Cook, F. L.. 2014. “The Influence of Partisan Motivated Reasoning on Public Opinion.” Political Behavior 36(2):235–262.CrossRefGoogle Scholar
Bonica, A. 2015. “Database on Ideology, Money in Politics, and Elections (DIME).” https://doi.org/10.7910/DVN/O5PX0B, Harvard Dataverse, V3.CrossRefGoogle Scholar
Bullock, J. G. 2011. “Elite Influence on Public Opinion in an Informed Electorate.” The American Political Science Review 105(3):496–515.CrossRefGoogle Scholar
Campbell, A., Converse, P., Miller, W., and Stokes, D.. 1960. The American Voter. Hoboken, NJ: Chicago University Press.Google Scholar
Chernozhukov, V. et al. 2018. “Double Machine Learning for Treatment and Structural Parameters.” Econometrics Journal 21:C1 – C68.CrossRefGoogle Scholar
Coppock, A., Leeper, T. J., and Mullinix, K. J.. 2018. “Generalizability of Heterogeneous Treatment Effect Estimates Across Samples.” Proceedings of the National Academy of Sciences 115(49):12441–12446.CrossRefGoogle ScholarPubMed
Cox, D. R. 1958. Planning of Experiments. Hoboken, NJ: Wiley.Google Scholar
de la Cuesta, B., Egami, N., and Imai, K.. 2020a. “Replication Data for: Improving the External Validity of Conjoint Analysis: The Essential Role of Profile Distribution.” Code Ocean, V1. doi: https://doi.org/10.24433/CO.9475665.v1 . CrossRefGoogle Scholar
de la Cuesta, B., Egami, N., and Imai, K.. 2020b. “Replication Data for: Improving the External Validity of Conjoint Analysis: The Essential Role of Profile Distribution.” https://doi.org/10.7910/DVN/HVY5GR, Harvard Dataverse, V1.CrossRefGoogle Scholar
Druckman, J. N. 2014. “Pathologies of Studying Public Opinion, Political Communication, and Democratic Responsiveness.” Political Communication 31(3):467–492.CrossRefGoogle Scholar
Egami, N., and Imai, K.. 2019. “Causal Interaction in Factorial Experiments: Application to Conjoint Analysis.” Journal of the American Statistical Association 114(526):529–540.CrossRefGoogle Scholar
Gertheiss, J., and Tutz, G. . 2010. “Sparse Modeling of Categorial Explanatory Variables.” The Annals of Applied Statistics 4(4):2150–2180.CrossRefGoogle Scholar
Green, P. E., Krieger, A. M., and Wind, Y.. 2001. “Thirty Years of Conjoint Analysis: Reflections and Prospects.” Interfaces 31(3_supplement):56–73.CrossRefGoogle Scholar
Greene, W. H. 2011. Econometric Analysis. London: Pearson.Google Scholar
Hainmueller, J., Hangartner, D., and Yamamoto, T.. 2015. “Validating Vignette and Conjoint Survey Experiments against Real-World Behavior.” Proceedings of the National Academy of Sciences 112(8):2395–2400.CrossRefGoogle ScholarPubMed
Hainmueller, J., and Hopkins, D. J.. 2015. “The Hidden American Immigration Consensus: A Conjoint Analysis of Attitudes Toward Immigrants.” American Journal of Political Science 59(3):529–548.CrossRefGoogle Scholar
Hainmueller, J., Hopkins, D. J., and Yamamoto, T.. 2014. “Causal Inference in Conjoint Analysis: Understanding Multidimensional Choices via Stated Preference Experiments.” Political Analysis 22(1):1–30.CrossRefGoogle Scholar
Hájek, J. 1971. “Comment on ‘An Essay on the Logical Foundations of Survey Sampling, Part One’.” In The Foundations of Survey Sampling, edited by V. P., Godambe and D. A., Sprott, 236. New York: Holt, Rinehart, and Winston. Google Scholar
Huff, C., and Kertzer, J. D.. 2018. “How the Public Defines Terrorism.” American Journal of Political Science 62(1):55–71.CrossRefGoogle Scholar
Kish, L. 1965. Survey Sampling. New York: John Wiley & Sons.Google Scholar
Lau, R. R., and Redlawsk, D. P.. 2001. “Advantages and Disadvantages of Cognitive Heuristics in Political Decision Making.” American Journal of Political Science 45(4):951–971.CrossRefGoogle Scholar
Leeper, T. J., and Robison, J.. 2018. “More Important, but for What Exactly? The Insignificant Role of Subjective Issue Importance in Vote Decisions.” Political Behavior 42:239–259.CrossRefGoogle Scholar
Marshall, P., and Bradlow, E. T.. 2002. “A Unified Approach to Conjoint Analysis Models.” Journal of the American Statistical Association 97(459):674–682.CrossRefGoogle Scholar
McDermott, M. 1997. “Voting Cues in Low-Information Elections: Candidate Gender as a Social Information Variable in Contemporary United States Elections.” American Journal of Political Science 41(1):270–283.CrossRefGoogle Scholar
McFadden, D. 1974. Conditional Logit Analysis of Qualitative Choice Behavior. In Frontiers in Econometrics, edited by Zarembka, P.. New York: Academic Press.Google Scholar
Miratrix, L. W., Sekhon, J. S., Theodoridis, A. G., and Campos, L. F.. 2018. “Worth Weighting? How to Think About and Use Weights in Survey Experiments.” Political Analysis 26(3):275–291.CrossRefGoogle Scholar
Mullinix, K. J., Leeper, T. J., Druckman, J. N., and Freese, J.. 2015. “The Generalizability of Survey Experiments.” Journal of Experimental Political Science 2(2):109–138.CrossRefGoogle Scholar
Mutz, D. C. 2011. Population-Based Survey Experiments. Princeton, NJ: Princeton University Press.Google Scholar
Neyman, J. 1923. “On the Application of Probability Theory to Agricultural Experiments. Essay on Principles (with discussion). Section 9 (translated).” Statistical Science 5(4):465–472.Google Scholar
Ono, Y., and Burden, B. C.. 2019. “The Contingent Effects of Candidate Sex on Voter Choice.” Political Behavior 41:583–607.CrossRefGoogle Scholar
Peterson, E. 2017. “The Role of the Information Environment in Partisan Voting.” The Journal of Politics 79(4):1191–1204.CrossRefGoogle Scholar
Rubin, D. B. 1974. “Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies.” Journal of Educational Psychology 66(5):688.CrossRefGoogle Scholar
Rubin, D. B. 1990. “Comment: Neyman (1923) and Causal Inference in Experiments and Observational Studies.” Statistical Science 5(4):472–480.CrossRefGoogle Scholar
Teele, D. L., Kalla, J., and Rosenbluth, F.. 2018. “The Ties That Double Bind: Social Roles and Women’s Underrepresentation in Politics.” American Political Science Review 112(3):525–541.CrossRefGoogle Scholar
Tibshirani, R. J., and Taylor, J.. 2011. “The Solution Path of the Generalized Lasso.” The Annals of Statistics 39(3):1335 – 1371.CrossRefGoogle Scholar
Supplementary material: Link

de la Cuesta et al. Dataset

Dataset

https://doi.org/10.7910/DVN/HVY5GR
Link
Supplementary material: PDF

de la Cuesta et al. supplementary material

Appendix

Download de la Cuesta et al. supplementary material(PDF)
PDF 502 KB