Skip to main content Accessibility help
×
×
Home

Unexpected Event during Survey Design: Promise and Pitfalls for Causal Inference

  • Jordi Muñoz (a1), Albert Falcó-Gimeno (a1) and Enrique Hernández (a2)

Abstract

An increasing number of studies exploit the occurrence of unexpected events during the fieldwork of public opinion surveys to estimate causal effects. In this paper, we discuss the use of this identification strategy based on unforeseen and salient events that split the sample of respondents into treatment and control groups: the Unexpected Event during Survey Design. In particular, we focus on the assumptions under which unexpected events can be exploited to estimate causal effects and we discuss potential threats to identification, paying especial attention to the observable and testable implications of these assumptions. We propose a series of best practices in the form of various estimation strategies and robustness checks that can be used to lend credibility to the causal estimates. Drawing on data from the European Social Survey, we illustrate the discussion of this method with an original study of the impact of the Charlie Hebdo terrorist attacks (Paris, 01/07/2015) on French citizens’ satisfaction with their national government.

Copyright

Corresponding author

Footnotes

Hide All

Authors’ note: This research has received financial support from the Spanish Ministry of Science, Innovation, and Universities through the research grant CSO2017-89085-P. We are grateful to Lucas Leeman, Silja Häusermann, Macarena Ares, Guillem Rico, Salomo Hirvonen, the editor, and anonymous reviewers of Political Analysis and participants at the University of Zurich IPZ publication seminar, the Autonomous University of Barcelona DEC Seminar, and the 2018 EPSA conference for helpful comments and suggestions. We are especially grateful to Erik Gahner Larsen for his detailed and insightful comments and for generously sharing with us his collection of UESD references. Enrique Hernández also thanks the University of Zurich Political Science Department for hospitality during the spring of 2018. Replication materials are available at the Political Analysis Dataverse: doi.org/10.7910/DVN/RDIIVL (Muñoz, Falcó-Gimeno, and Hernández 2019).

Contributing Editor: Jeff Gill

Footnotes

References

Hide All
Ares, M., and Hernández, E.. 2017. “The Corrosive Effect of Corruption on Trust in Politicians: Evidence From a Natural Experiment.” Research and Politics 4(2):18.
Balcells, L., and Torrats-Espinosa, G.. 2018. “Using a Natural Experiment to Estimate the Electoral Consequences of Terrorist Attacks.” Proceedings of the National Academy of Sciences 115(42):1062410629.
Barabas, J., and Jerit, J.. 2010. “Are Survey Experiments Externally Valid? American Political Science Review 104(2):226242.
Berinsky, A. J. 2017. “Measuring Public Opinion with Surveys.” Annual Review of Political Science 20:309329.
Boomgaarden, H. G., and de Vreese, C. H.. 2007. “Dramatic Real-world Events and Public Opinion Dynamics: Media Coverage and its Impact on Public Reactions to an Assassination.” International Journal of Public Opinion Research 19(3):354366.
Boydstun, A. E., Feezell, J. T., and Glazier, R. A.. 2018. “In the Wake of a Terrorist Attack, do Americans’ Attitudes Toward Muslims Decline? Research and Politics 5(4):17.
Branton, R., Martinez-Ebers, V., Carey, T. E., and Matsubayashi, T.. 2015. “Social Protest and Policy Attitudes: The Case of the 2006 Immigrant Rallies.” American Journal of Political Science 59(2):390402.
Brehm, J. 1993. The Phantom Respondents: Opinion Surveys and Political Representation . University of Michigan Press.
Coupe, T. 2017. “The Impact of Terrorism on Expectations, Trust and Happiness – the Case of the November 13 Attacks in Paris, France.” Applied Economics Letters 24(15):10841087.
Csikszentmihalyi, M., and Hunter, J.. 2003. “Happiness in Everyday Life: The Uses of Experience Sampling.” Journal of Happiness Studies 4(2):185199.
De Vries, C. E. 2018. Euroscepticism and the Future of European Integration . Oxford University Press.
Dinesen, P. T., and Jæger, M. M.. 2013. “The Effect of Terror on Institutional Trust: New Evidence From the 3/11 Madrid Terrorist Attack.” Political Psychology 34(6):917926.
Eggers, A. C., Freier, R., Grembi, V., and Nannicini, T.. 2018. “Regression Discontinuity Designs Based on Population Thresholds: Pitfalls and Solutions.” American Journal of Political Science 62(1):210229.
ESS Round 7: European Social Survey Round 7 Data. Data file edition 2.1. NSD - Norwegian Centre for Research Data, Norway – Data Archive and distributor of ESS data for ESS ERIC.N.d.
Finseraas, H., and Listhaug, O.. 2013. “It Can Happen Here: the Impact of the Mumbai Terror Attacks on Public Opinion in Western Europe.” Public Choice 156(1–2):213228.
Flores, R. D. 2018. “Can Elites Shape Public Attitudes Toward Immigrants? Evidence from the 2016 US Presidential Election.” Social Forces 96(4):16491690.
Gaines, B. J., Kuklinski, J. H., and Quirk, P. J.. 2006. “The Logic of the Survey Experiment Reexamined.” Political Analysis 15(1):120.
Geys, B., and Qari, S.. 2017. “Will You Still Trust me Tomorrow? The Causal Effect of Terrorism on Social Trust.” Public Choice 96(3–4):289305.
Hainmueller, J. 2012. “Entropy Balancing for Causal Effects: A Multivariate Reweighting Method to Produce Balanced Samples in Observational Studies.” Political Analysis 20(1):2546.
Hetherington, M. J., and Nelson, M.. 2003. “Anatomy of a Rally Effect: George W. Bush and the War on Terrorism.” PS: Political Science and Politics 36(1):3742.
Hofstetter, C. R. 1969. “Political Disengagement and the Death of Martin Luther King.” The Public Opinion Quarterly 33(2):174179.
Imbens, G. W., and Lemieux, T.. 2008. “Regression Discontinuity Designs: A Guide to Practice.” Journal of Econometrics 142(2):615635.
Jakobsson, N., and Blom, S.. 2014. “Did the 2011 Terror Attacks in Norway Change Citizens’ Attitudes Toward Immigrants? International Journal of Public Opinion Research 26(4):475486.
Jensen, C., and Naumann, E.. 2016. “Increasing Pressures and Support for Public Healthcare in Europe.” Health Policy 120(6):698705.
Johnston, R., and Brady, H. E.. 2002. “The Rolling Cross-Section Design.” Electoral Studies 21(2):283295.
Keele, L., and Titiunik, R.. 2016. “Natural Experiments Based on Geography.” Political Science Research and Methods 4(1):6595.
Kim, J. W., and Kim, E.. 2019. “Identifying the Effect of Political Rumor Diffusion Using Variations in Survey Timing.” Quarterly Journal of Political Science , forthcoming.
Kocher, M. A., and Monteiro, N. P.. 2016. “Lines of Demarcation: Causation, Design-Based Inference, and Historical Research.” Perspectives on Politics 14(4):952975.
Larsen, E. G. 2018. “Welfare Retrenchments and Government Support: Evidence from a Natural Experiment.” European Sociological Review 34(1):4051.
Legewie, J. 2013. “Terrorist Events and Attitudes toward Immigrants: A Natural Experiment.” American Journal of Sociology 118(5):11991245.
Lever, H. 1969. “The Johannesburg Station Explosion and Ethnic Attitudes.” The Public Opinion Quarterly 33(2):180189.
MacKinlay, A. C. 1997. “Event Studies in Economics and Finance.” Journal of Economic Literature 35(1):1339.
Metcalfe, R., Powdthavee, N., and Dolan, P.. 2011. “Destruction and Distress: Using a Quasi-Experiment to Show the Effects of the September 11 Attacks on Mental Well-Being in the United Kingdom.” The Economic Journal 121(550):F81F103.
Minkus, L., Deutschmann, E., and Delhey, J.. 2018. “A Trump Effect on the EU’s Popularity? The US Presidential Election as a Natural Experiment.” Perspectives on Politics 118, forthcoming.
Montgomery, J. M., Nyhan, B., and Torres, M.. 2018. “How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It.” American Journal of Political Science 62(3):760775.
Motta, M. 2018. “The Polarizing Effect of the March for Science on Attitudes toward Scientists.” PS: Political Science and Politics 51(4):782788.
Muñoz, J., Falcó-Gimeno, A., and Hernández, E.. 2019. “Replication Data for: Unexpected Event during Survey Design: Promise and Pitfalls for Causal Inference.” doi.org/10.7910/DVN/RDIIVL, Harvard Dataverse, V1.
Perrin, A. J., and Smolek, S. J.. 2009. “Who Trusts? Race, Gender, and the September 11 Rally Effect Among Young Adults.” Social Science Research 38(1):134145.
Pierce, L., Rogers, T., and Snyder, J. A.. 2016. “Losing Hurts: The Happiness Impact of Partisan Electoral Loss.” Journal of Experimental Political Science 3(1):4459.
Pollock, W., Barabas, J., Jerit, J., Schoonvelde, M., Banducci, S., and Stevens, D.. 2015. “Studying Media Events in the European Social Surveys Across Research Designs, Countries, Time, Issues, and Outcomes.” European Political Science 14(4):394421.
Silber Mohamed, H. 2013. “Can Protests make Latinos “American” Identity, Immigration Politics, and the 2006 Marches.” American Politics Research 41(2):298327.
Silva, B. C. 2018. “The Non-Impact of the 2015 Paris Terrorist Attacks on Political Attitudes.” Personality and Social Psychology Bulletin 44(6):838850.
Solaz, H., De Vries, C. E., and de Geus, R. A.. 2018. “In-Group Loyalty and the Punishment of Corruption.” Comparative Political Studies OnlineFirst:131.
St. Clair, T., Cook, T. D., and Hallberg, K.. 2014. “Examining the Internal Validity and Statistical Precision of the Comparative Interrupted Time Series Design by Comparison With a Randomized Experiment.” American Journal of Evaluation 35(3):311327.
Stokes, S. C. 2014. “A Defense of Observational Research.” In Field Experiments and Their Critics: Essays on the Uses and Abuses of experimentation in the Social Sciences , 3357. Yale University Press: Dawn Teele, New Haven.
Stoop, I. A. L. 2004. “Surveying Nonrespondents.” Field Methods 16(1):2354.
Tiokhin, L., and Hruschka, D.. 2017. “No Evidence That an Ebola Outbreak Influenced Voting Preferences in the 2014 Elections After Controlling for Time-Series Autocorrelation: A Commentary on Beall, Hofer, and Schaller (2016).” Psychological Science 28(9):13581360.
Zepeda-Millán, C., and Wallace, S. J.. 2013. “Racialization in Times of Contention: How Social Movements Influence Latino Racial Identity.” Politics, Groups, and Identities 1(4):510527.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
MathJax

Keywords

Type Description Title
UNKNOWN
Supplementary materials

Muñoz et al. Dataset
Dataset

 Unknown
UNKNOWN
Supplementary materials

Muñoz et al. supplementary material
Muñoz et al. supplementary material 1

 Unknown (570 KB)
570 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed