Skip to main content
×
×
Home

The Dangers of Extreme Counterfactuals

  • Gary King (a1) and Langche Zeng (a2)
Abstract

We address the problem that occurs when inferences about counterfactuals—predictions, “what-if” questions, and causal effects—are attempted far from the available data. The danger of these extreme counterfactuals is that substantive conclusions drawn from statistical models that fit the data well turn out to be based largely on speculation hidden in convenient modeling assumptions that few would be willing to defend. Yet existing statistical strategies provide few reliable means of identifying extreme counterfactuals. We offer a proof that inferences farther from the data allow more model dependence and then develop easy-to-apply methods to evaluate how model dependent our answers would be to specified counterfactuals. These methods require neither sensitivity testing over specified classes of models nor evaluating any specific modeling assumptions. If an analysis fails the simple tests we offer, then we know that substantive results are sensitive to at least some modeling choices that are not based on empirical evidence. Free software that accompanies this article implements all the methods developed.

    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      The Dangers of Extreme Counterfactuals
      Available formats
      ×
      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      The Dangers of Extreme Counterfactuals
      Available formats
      ×
      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      The Dangers of Extreme Counterfactuals
      Available formats
      ×
Copyright
References
Hide All
Bishop, Christopher M. 1995. Neural Networks for Pattern Recognition. Oxford: Oxford University Press.
Cuadras, C. M., and Fortiana, J. 1995. “A Continuous Metric Scaling Solution for a Random Variable.” Journal of Multivariate Analysis 52: 114.
Cuadras, C. M., Fortiana, J., and Oliva, F. 1997. “The Proximity of an Individual to a Population with Applications to Discriminant Analysis.” Journal of Classification 14: 117136.
de Berg, Mark, van Krevald, Marc, Overmars, Mark, and Schwarzkopf, Otfried. 1998. Computational Geometry: Algorithms and Applications, 2nd rev. ed. New York: Springer.
Esty, Daniel C., Goldstone, Jack, Gurr, Ted Robert, Harff, Barbara, Surko, Pamela T., Unger, Alan N., and Chen, Robert S. 1998. The State Failure Task Force Report: Phase II Findings. McLean, VA: Science Applications International Corporation.
Frangakis, Constantine E., and Rubin, Donald. 2002. “Principal Stratification in Causal Inference.” Biometrics 58: 2129.
Gelman, Andrew, and King, Gary. 1994. “Party Competition and Media Messages in U.S. Presidential Election Campaigns.” In The Parties Respond: Changes in the American Party System, ed. Maisel, Sandy L. Boulder, CO: Westview, pp. 255295. (Available from http://gking.harvard.edu/files/abs/partycomp-abs.shtml.)
Gower, J. C. 1966. “Some Distance Properties of Latent Root and Vector Methods Used in Multivariate Analysis.” Biometrika 53 (3/4): 325388.
Gower, J. C. 1971. “A General Coefficient of Similarity and Some of Its Properties.” Biometrics 27: 857872.
Greenland, Sander, Pearl, Judea, and Robins, James M. 1999. “Causal Diagrams for Epidemiologic Research.” Epidemiology 10(1): 3748.
Hastie, Trevor, Tibshirani, R., and Friedman, J. 2001. The Elements of Statistical Learning. New York: Springer Verlag.
Heckman, James, Ichimura, H., and Todd, P. 1998. “Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Program.” Review of Economic Studies 64: 605654.
Heckman, James J., Ichimura, Hidehiko, Smith, Jeffrey, and Todd, Petra. 1998. “Characterizing Selection Bias Using Experimental Data.” Econometrika 66(5): 10171098.
Ho, Daniel, Imai, Kosuke, King, Gary, and Stuart, Elizabeth. 2005. “Matching as Nonparametric Preprocessing for Parametric Causal Inference.” http://gking.harvard.edu/files/matchp.pdf.
Hoeting, Jennifer A., Madigan, David, Raftery, Adrian E., and Volinsky, Chris T. 1999. “Bayesian Model Averaging: A Tutorial.” Statistical Science 14(4): 382417.
Holland, Paul W. 1986. “Statistics and Causal Inference.” Journal of the American Statistical Association 81: 945960.
Imai, Kosuke, and van Dyk, David A. 2004. “Causal Inference with General Treatment Regimes: Generalizing the Propensity Score.” Journal of the American Statistical Association 99(467): 854866.
Imai, Kosuke, and King, Gary. 2004. “Did Illegal Overseas Absentee Ballots Decide the 2000 U.S. Presidential Election?Perspectives on Politics 2(3): 537549.
Kallay, Michael. 1986. “Convex Hull Made Easy.” Information Processing Letters 22 (March): 161.
King, Gary. 1991. “‘Truth’ Is Stranger than Prediction, More Questionable than Causal Inference.” American Journal of Political Science 35(4): 10471053. http://gking.harvard.edu/files/abs/truth-abs.shtml.
King, Gary, Keohane, Robert O., and Verba, Sidney. 1994. Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton, NJ: Princeton University Press.
King, Gary, Tomz, Michael, and Wittenberg, Jason. 2000. “Making the Most of Statistical Analyses: Improving Interpretation and Presentation.” American Journal of Political Science 44(2): 341355. http://gking.harvard.edu/files/abs/making-abs.shtml.
King, Gary, and Zeng, Langche. 2002. “Improving Forecasts of State Failure.” World Politics 53(4): 623658. http://gking.harvard.edu/files/abs/civil-abs.shtml.
Klee, Victor. 1980. “On the Complexity of d-Dimensional Voronoi Diagrams.” Archive der Mathematik 34: 7580.
Kuo, Yen-Hong. 2001. “Extrapolation of Association between Two Variables in Four General Medical Journals.” Presented at the Fourth International Congress on Peer Review in Biomedical Publication, Barcelona, Spain.
Lechner, Michael. 1999. Identification and Estimation of Causal Effects of Multiple Treatments under the Conditional Independence Assumptions.” IZA Discussion Papers no. 91, University St. Gallen.
Madych, W. R., and Nelson, S. A. 1992. Bounds on Multivariate Polynomials and Exponential Error Estimates for Multiquadric Interpolation.” Journal of Approximation Theory 70: 94114.
Manski, Charles F. 1995. Identification Problems in the Social Sciences. Cambridge, MA: Harvard University Press.
Meng, Xiao-Li, and Romero, Marin. 2003. “Discussion: Efficiency and Self-Efficiency.” International Statistical Review 71(3): 607618.
O'Rourke, Joseph. 1998. Computational Geometry in C. New York: Cambridge University Press.
Pearl, Judea. 2000. Causality: Models, Reasoning, and Inference. Cambridge, UK: Cambridge University Press.
Robins, James M. 1999a. Marginal Structural Models versus Structural Nested Models as Tools for Causal Inference.” In Statistical Models in Epidemiology: The Environment and Clinical Trials, vol. 16, eds. Halloran, M. E. and Berry, D. New York: Springer-Verlag, pp. 95134.
Robins, James M. 1999b. Association, Causation, and Marginal Structural Models.” Synthese 121: 151179.
Rosenbaum, Paul. 1984. “The Consequences of Adjusting for a Concomitant Variable That Has Been Affected by the Treatment.” Journal of the Royal Statistical Society, A 147(5): 656666.
Rosenbaum, Paul R., and Rubin, Donald B. 1983. The Central Role of the Propensity Score in Observational Studies for Causal Effects.” Biometrika 70: 4155.
Rosenbaum, Paul R., and Rubin, Donald B. 1984. Reducing Bias in Observational Studies Using Subclassification on the Propensity Score.” Journal of the American Statistical Association 79: 515524.
Rubin, Donald B. 1974. Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies.” Journal of Educational Psychology 6: 688701.
Schaback, R. 1996. Approximation by Radia Basis Functions with Finitely Many Centers.” Constructive Approximation 12: 331340.
Sisson, Scott A. 2005. Transdimensional Markov Chains: A Decade of Progress and Future Perspectives.” Journal of the American Statistical Association 100(471): 10771089.
Stoll, Heather, King, Gary, and Zeng, Langche. 2005. WhatIf: Software for Evaluating Counterfactuals.” http://gking.harvard.edu/whatif/.
Valentine, Frederick Albert. 1964. Convex Sets. New York: McGraw-Hill.
Winship, Christopher, and Morgan, Stephen L. 1999. The Estimation of Causal Effects from Observational Data.” American Review of Sociology 25: 659707.
Wu, Z., and Schaback, R. 1993. Local Error Estimates for Radial Basis Function Interpolation of Scattered Data.” Journal of Numerical Analysis 13: 1327.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
MathJax

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed