Skip to main content

Analyzing Causal Mechanisms in Survey Experiments

  • Avidit Acharya (a1), Matthew Blackwell (a2) and Maya Sen (a3)

Researchers investigating causal mechanisms in survey experiments often rely on nonrandomized quantities to isolate the indirect effect of treatment through these variables. Such an approach, however, requires a “selection-on-observables” assumption, which undermines the advantages of a randomized experiment. In this paper, we show what can be learned about casual mechanisms through experimental design alone. We propose a factorial design that provides or withholds information on mediating variables and allows for the identification of the overall average treatment effect and the controlled direct effect of treatment fixing a potential mediator. While this design cannot identify indirect effects on its own, it avoids making the selection-on-observable assumption of the standard mediation approach while providing evidence for a broader understanding of causal mechanisms that encompasses both indirect effects and interactions. We illustrate these approaches via two examples: one on evaluations of US Supreme Court nominees and the other on perceptions of the democratic peace.

Corresponding author
Hide All

Contributing Editor: R. Michael Alvarez

Many thanks to John Ahlquist, Josh Kertzer, Ryan T. Moore, Paul Testa, and Teppei Yamamoto for helpful feedback. Special thanks to Jessica Weeks and Mike Tomz for sharing their survey instrument with us. Replication data and code can be found in Acharya, Blackwell, and Sen (2018).

Hide All
Acharya, Avidit, Blackwell, Matthew, and Sen, Maya. 2016. Explaining causal findings without bias: detecting and assessing direct effects. American Political Science Review 110(3):512529.
Acharya, Avidit, Blackwell, Matthew, and Sen, Maya. 2018. Replication data for: analyzing causal mechanisms in survey experiments. doi:10.7910/DVN/KHE44F, Harvard Dataverse, V1, UNF:6:VaeMnsesFmBVwg98eK/heA==.
Alesina, Alberto, Giuliano, Paola, and Nunn, Nathan. 2013. On the origins of gender roles: women and the plough. Quarterly Journal of Economics 128(2):469530.
Bansak, Kirk, Hainmueller, Jens, Hopkins, Daniel J., and Yamamoto, Teppei. 2017. Beyond the breaking point? Survey satisficing in conjoint experiments. Stanford University Graduate School of Business Research Paper No. 17-33; MIT Political Science Department Research Paper No. 2017-16.
Cochran, William G., and Cox, Gertrude M.. 1957. Experimental designs . New York: John Wiley & Sons.
Dafoe, Allan, Zhang, Baobao, and Caughey, Devin. 2017. Information equivalence in survey experiments. Political Analysis , forthcoming.
Gerber, Alan S., and Green, Donald P.. 2012. Field experiments: design, analysis, and interpretation . New York: W.W. Norton.
Hainmueller, Jens, Hopkins, Daniel J., and Yamamoto, Teppei. 2013. Causal inference in conjoint analysis: understanding multidimensional choices via stated preference experiments. Political Analysis 22(1):130.
Holland, Paul W. 1986. Statistics and causal inference. Journal of the American Statistical Association 81(396):945960.
Imai, Kosuke, Tingley, Dustin, and Yamamoto, Teppei. 2013. Experimental designs for identifying causal mechanisms. Journal of the Royal Statistical Society: Series A 176(1):551.
Imai, Kosuke, Keele, Luke, Tingley, Dustin, and Yamamoto, Teppei. 2011. Unpacking the black box of causality: learning about causal mechanisms from experimental and observational studies. American Political Science Review 105(4):765789.
Imai, Kosuke, Keele, Luke, and Yamamoto, Teppei. 2010. Identification, inference and sensitivity analysis for causal mediation effects. Statistical Science 25(1):5171.
Kirkland, Patricia, and Coppock, Alexander. 2017. Candidate choice without party labels: new insights from conjoint survey experiments. Political Behavior . Epub ahead of print, doi:10.1007/s11109-017-9414-8.
McDermott, Monika L. 1998. Race and gender cues in low-information elections. Political Research Quarterly 51(4):895918.
Neyman, Jerzy. 1923. On the application of probability theory to agricultural experiments. Essay on principles. Section 9. Statistical Science 5:465480, translated in 1990 with discussion.
Petersen, Maya L., Sinisi, Sandra E., and van der Laan, Mark J.. 2006. Estimation of direct causal effects. Epidemiology 17(3):276284.
Robins, James M. 2003. Semantics of causal DAG models and the identification of direct and indirect effects. In Highly structured stochastic systems , ed. Green, P. J., Hjort, N. L., and Richardson, S.. Oxford: Oxford University Press, pp. 7081.
Robins, James M., and Greenland, Sander. 1992. Identifiability and exchangeability for direct and indirect effects. Epidemiology 3(2):143155.
Rosenbaum, Paul R. 1984. The consquences of adjustment for a concomitant variable that has been affected by the treatment. Journal of the Royal Statistical Society. Series A (General) 147(5):656666.
Rubin, Donald B. 1974. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology 6(5):688701.
Sen, Maya. 2017. How political signals affect public support for judicial nominations: evidence from a conjoint experiment. Political Research Quarterly 70(2):374393.
Tomz, Michael R., and Weeks, Jessica L. P.. 2013. Public opinion and the democratic peace. American Political Science Review 107(04):849865.
VanderWeele, Tyler J. 2014. A unification of mediation and interaction: a 4-way decomposition. Epidemiology 25(5):749761.
VanderWeele, Tyler J. 2015. Explanation in causal inference: methods for mediation and interaction . New York: Oxford University Press.
VanderWeele, Tyler J., and Tchetgen Tchetgen, Eric J.. 2014. Attributing effects to interactions. Epidemiology 25(5):711722.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Type Description Title
Supplementary materials

Acharya et al. supplementary material
Acharya et al. supplementary material 1

 Unknown (90 KB)
90 KB


Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed