Skip to main content Accessibility help
×
Home
Hostname: page-component-8bbf57454-6xp8w Total loading time: 0.185 Render date: 2022-01-23T00:52:09.854Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "metricsAbstractViews": false, "figures": true, "newCiteModal": false, "newCitedByModal": true, "newEcommerce": true, "newUsageEvents": true }

The Dangers of Extreme Counterfactuals

Published online by Cambridge University Press:  04 January 2017

Gary King
Affiliation:
Department of Government and Institute for Quantitative Social Science, Harvard University, 1737 Cambridge Street, Cambridge MA 02138. e-mail: king@harvard.edu
Langche Zeng
Affiliation:
Department of Political Science, University of California, San Diego, 9500 Gilman Drive, La Jolla, CA 92093-0521. e-mail: zeng@ucsd.edu
Rights & Permissions[Opens in a new window]

Abstract

HTML view is not available for this content. However, as you have access to this content, a full PDF is available via the ‘Save PDF’ action button.

We address the problem that occurs when inferences about counterfactuals—predictions, “what-if” questions, and causal effects—are attempted far from the available data. The danger of these extreme counterfactuals is that substantive conclusions drawn from statistical models that fit the data well turn out to be based largely on speculation hidden in convenient modeling assumptions that few would be willing to defend. Yet existing statistical strategies provide few reliable means of identifying extreme counterfactuals. We offer a proof that inferences farther from the data allow more model dependence and then develop easy-to-apply methods to evaluate how model dependent our answers would be to specified counterfactuals. These methods require neither sensitivity testing over specified classes of models nor evaluating any specific modeling assumptions. If an analysis fails the simple tests we offer, then we know that substantive results are sensitive to at least some modeling choices that are not based on empirical evidence. Free software that accompanies this article implements all the methods developed.

Type
Research Article
Copyright
Copyright © The Author 2005. Published by Oxford University Press on behalf of the Society for Political Methodology 

References

Bishop, Christopher M. 1995. Neural Networks for Pattern Recognition. Oxford: Oxford University Press.Google Scholar
Cuadras, C. M., and Fortiana, J. 1995. “A Continuous Metric Scaling Solution for a Random Variable.” Journal of Multivariate Analysis 52: 114.CrossRefGoogle Scholar
Cuadras, C. M., Fortiana, J., and Oliva, F. 1997. “The Proximity of an Individual to a Population with Applications to Discriminant Analysis.” Journal of Classification 14: 117136.CrossRefGoogle Scholar
de Berg, Mark, van Krevald, Marc, Overmars, Mark, and Schwarzkopf, Otfried. 1998. Computational Geometry: Algorithms and Applications, 2nd rev. ed. New York: Springer.Google Scholar
Esty, Daniel C., Goldstone, Jack, Gurr, Ted Robert, Harff, Barbara, Surko, Pamela T., Unger, Alan N., and Chen, Robert S. 1998. The State Failure Task Force Report: Phase II Findings. McLean, VA: Science Applications International Corporation.Google Scholar
Frangakis, Constantine E., and Rubin, Donald. 2002. “Principal Stratification in Causal Inference.” Biometrics 58: 2129.CrossRefGoogle ScholarPubMed
Gelman, Andrew, and King, Gary. 1994. “Party Competition and Media Messages in U.S. Presidential Election Campaigns.” In The Parties Respond: Changes in the American Party System, ed. Maisel, Sandy L. Boulder, CO: Westview, pp. 255295. (Available from http://gking.harvard.edu/files/abs/partycomp-abs.shtml.)Google Scholar
Gower, J. C. 1966. “Some Distance Properties of Latent Root and Vector Methods Used in Multivariate Analysis.” Biometrika 53 (3/4): 325388.CrossRefGoogle Scholar
Gower, J. C. 1971. “A General Coefficient of Similarity and Some of Its Properties.” Biometrics 27: 857872.CrossRefGoogle Scholar
Greenland, Sander, Pearl, Judea, and Robins, James M. 1999. “Causal Diagrams for Epidemiologic Research.” Epidemiology 10(1): 3748.CrossRefGoogle ScholarPubMed
Hastie, Trevor, Tibshirani, R., and Friedman, J. 2001. The Elements of Statistical Learning. New York: Springer Verlag.CrossRefGoogle Scholar
Heckman, James, Ichimura, H., and Todd, P. 1998. “Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Program.” Review of Economic Studies 64: 605654.CrossRefGoogle Scholar
Heckman, James J., Ichimura, Hidehiko, Smith, Jeffrey, and Todd, Petra. 1998. “Characterizing Selection Bias Using Experimental Data.” Econometrika 66(5): 10171098.CrossRefGoogle Scholar
Ho, Daniel, Imai, Kosuke, King, Gary, and Stuart, Elizabeth. 2005. “Matching as Nonparametric Preprocessing for Parametric Causal Inference.” http://gking.harvard.edu/files/matchp.pdf.Google Scholar
Hoeting, Jennifer A., Madigan, David, Raftery, Adrian E., and Volinsky, Chris T. 1999. “Bayesian Model Averaging: A Tutorial.” Statistical Science 14(4): 382417.Google Scholar
Holland, Paul W. 1986. “Statistics and Causal Inference.” Journal of the American Statistical Association 81: 945960.CrossRefGoogle Scholar
Imai, Kosuke, and van Dyk, David A. 2004. “Causal Inference with General Treatment Regimes: Generalizing the Propensity Score.” Journal of the American Statistical Association 99(467): 854866.CrossRefGoogle Scholar
Imai, Kosuke, and King, Gary. 2004. “Did Illegal Overseas Absentee Ballots Decide the 2000 U.S. Presidential Election?Perspectives on Politics 2(3): 537549.CrossRefGoogle Scholar
Kallay, Michael. 1986. “Convex Hull Made Easy.” Information Processing Letters 22 (March): 161.CrossRefGoogle Scholar
King, Gary. 1991. “‘Truth’ Is Stranger than Prediction, More Questionable than Causal Inference.” American Journal of Political Science 35(4): 10471053. http://gking.harvard.edu/files/abs/truth-abs.shtml.CrossRefGoogle Scholar
King, Gary, Keohane, Robert O., and Verba, Sidney. 1994. Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton, NJ: Princeton University Press.Google Scholar
King, Gary, Tomz, Michael, and Wittenberg, Jason. 2000. “Making the Most of Statistical Analyses: Improving Interpretation and Presentation.” American Journal of Political Science 44(2): 341355. http://gking.harvard.edu/files/abs/making-abs.shtml.CrossRefGoogle Scholar
King, Gary, and Zeng, Langche. 2002. “Improving Forecasts of State Failure.” World Politics 53(4): 623658. http://gking.harvard.edu/files/abs/civil-abs.shtml.CrossRefGoogle Scholar
Klee, Victor. 1980. “On the Complexity of d-Dimensional Voronoi Diagrams.” Archive der Mathematik 34: 7580.CrossRefGoogle Scholar
Kuo, Yen-Hong. 2001. “Extrapolation of Association between Two Variables in Four General Medical Journals.” Presented at the Fourth International Congress on Peer Review in Biomedical Publication, Barcelona, Spain.Google Scholar
Lechner, Michael. 1999. Identification and Estimation of Causal Effects of Multiple Treatments under the Conditional Independence Assumptions.” IZA Discussion Papers no. 91, University St. Gallen.Google Scholar
Madych, W. R., and Nelson, S. A. 1992. Bounds on Multivariate Polynomials and Exponential Error Estimates for Multiquadric Interpolation.” Journal of Approximation Theory 70: 94114.CrossRefGoogle Scholar
Manski, Charles F. 1995. Identification Problems in the Social Sciences. Cambridge, MA: Harvard University Press.Google Scholar
Meng, Xiao-Li, and Romero, Marin. 2003. “Discussion: Efficiency and Self-Efficiency.” International Statistical Review 71(3): 607618.CrossRefGoogle Scholar
O'Rourke, Joseph. 1998. Computational Geometry in C. New York: Cambridge University Press.CrossRefGoogle Scholar
Pearl, Judea. 2000. Causality: Models, Reasoning, and Inference. Cambridge, UK: Cambridge University Press.Google Scholar
Robins, James M. 1999a. Marginal Structural Models versus Structural Nested Models as Tools for Causal Inference.” In Statistical Models in Epidemiology: The Environment and Clinical Trials, vol. 16, eds. Halloran, M. E. and Berry, D. New York: Springer-Verlag, pp. 95134.Google Scholar
Robins, James M. 1999b. Association, Causation, and Marginal Structural Models.” Synthese 121: 151179.CrossRefGoogle Scholar
Rosenbaum, Paul. 1984. “The Consequences of Adjusting for a Concomitant Variable That Has Been Affected by the Treatment.” Journal of the Royal Statistical Society, A 147(5): 656666.CrossRefGoogle Scholar
Rosenbaum, Paul R., and Rubin, Donald B. 1983. The Central Role of the Propensity Score in Observational Studies for Causal Effects.” Biometrika 70: 4155.CrossRefGoogle Scholar
Rosenbaum, Paul R., and Rubin, Donald B. 1984. Reducing Bias in Observational Studies Using Subclassification on the Propensity Score.” Journal of the American Statistical Association 79: 515524.CrossRefGoogle Scholar
Rubin, Donald B. 1974. Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies.” Journal of Educational Psychology 6: 688701.CrossRefGoogle Scholar
Schaback, R. 1996. Approximation by Radia Basis Functions with Finitely Many Centers.” Constructive Approximation 12: 331340.CrossRefGoogle Scholar
Sisson, Scott A. 2005. Transdimensional Markov Chains: A Decade of Progress and Future Perspectives.” Journal of the American Statistical Association 100(471): 10771089.CrossRefGoogle Scholar
Stoll, Heather, King, Gary, and Zeng, Langche. 2005. WhatIf: Software for Evaluating Counterfactuals.” http://gking.harvard.edu/whatif/.Google Scholar
Valentine, Frederick Albert. 1964. Convex Sets. New York: McGraw-Hill.Google Scholar
Winship, Christopher, and Morgan, Stephen L. 1999. The Estimation of Causal Effects from Observational Data.” American Review of Sociology 25: 659707.CrossRefGoogle Scholar
Wu, Z., and Schaback, R. 1993. Local Error Estimates for Radial Basis Function Interpolation of Scattered Data.” Journal of Numerical Analysis 13: 1327.CrossRefGoogle Scholar
You have Access
255
Cited by

Send article to Kindle

To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

The Dangers of Extreme Counterfactuals
Available formats
×

Send article to Dropbox

To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

The Dangers of Extreme Counterfactuals
Available formats
×

Send article to Google Drive

To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

The Dangers of Extreme Counterfactuals
Available formats
×
×

Reply to: Submit a response

Please enter your response.

Your details

Please enter a valid email address.

Conflicting interests

Do you have any conflicting interests? *