Skip to main content
×
×
Home

The sensitivity of sensitivity analysis

  • Thomas Plümper (a1) and Richard Traunmüller (a2)
Abstract

This article evaluates the reliability of sensitivity tests. Using Monte Carlo methods we show that, first, the definition of robustness exerts a large influence on the robustness of variables. Second and more importantly, our results also demonstrate that inferences based on sensitivity tests are most likely to be valid if determinants and confounders are almost uncorrelated and if the variables included in the true model exert a strong influence on outcomes. Third, no definition of robustness reliably avoids both false positives and false negatives. We find that for a wide variety of data-generating processes, rarely used definitions of robustness perform better than the frequently used model averaging rule suggested by Sala-i-Martin. Fourth, our results also suggest that Leamer’s extreme bounds analysis and Bayesian model averaging are extremely unlikely to generate false positives. Thus, if based on these inferential criteria a variable is robust, it is almost certain to belong into the empirical model. Fifth and finally, we also show that researchers should avoid drawing inferences based on lack of robustness.

Copyright
Corresponding author
*Corresponding author. Email: thomas.pluemper@wu.ac.at
References
Hide All
Bartels, LM (1997) Specification Uncertainty and Model Averaging. American Journal of Political Science 41, 641674.
Cederman, LE, Wimmer, A Min, B (2010) Why do Ethnic Groups Rebel. New Data and Analysis. World Politics 62(1), 87119.
Gleditsch, KS (2007) Transnational Dimensions of Civil War. Journal of Peace Research 44(3), 293309.
Granger, CWJ Uhlig, HF (1990) Reasonable Extreme Bounds Analysis. Journal of Econometrics 44, 159170.
Hafner-Burton, EM (2005) Right or Robust. The Sensitive Nature of Repression to Globalization. Journal of Peace Research 42(6), 679698.
Hegre, H Sambanis, N (2006) Sensitivity Analysis of Empirical Results on Civil War Onset. Journal of Conflict Resolution 50(4), 508535.
Hoeting, JA, Madigan, D, Raftery, AE Volinsky, CT (1999) Bayesian Model Averaging: A Tutorial. Statistical Science 14, 382401.
Leamer, EE (1978) Specification Searches: Ad Hoc Inference with Nonexperimental Data. New York: Wiley.
Leamer, EE (1983) Let’s Take the Con Out of Econometrics. American Economic Review 73(1), 3143.
Leamer, EE (1985) Sensitivity Analysis Would Help. American Economic Review 57(3), 308313.
Leamer, EE (2010) Tantalus on the Road to Asymptopia. The Journal of Economic Perspectives 24(2), 3146.
Leamer, E Leonard, H (1983) Reporting the Fragility of Regression Estimates. The Review of Economics and Statistics 65(2), 306317.
Levine, R Renelt, D (1992) A Sensitivity Analysis of Cross-Country Growth Regressions. American Economic Review 82(4), 942963.
Neumayer, E Plümper, T (2017) Robustness Tests for Quantitative Research. Cambridge: Cambridge University Press.
Raftery, AE (1995) Bayesian Model Selection in Social Research. Sociological Methodology 25, 111163.
Sala-i-Martin, X (1997) I Just Ran Two Million Regressions. American Economic Review 87(2), 178183.
Sturm, JE, Berger, H de Haan, J (2005) Which Variables Explain Decisions on IMF Credit. An Extreme Bounds Analysis. Economics & Politics 17(2), 177213.
Sturm, JE de Haan, J (2005) Determinants of Long-Term Growth: New Results Applying Estimation and Extreme Bounds Analysis. Empirical Economics 30(3), 597617.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Science Research and Methods
  • ISSN: 2049-8470
  • EISSN: 2049-8489
  • URL: /core/journals/political-science-research-and-methods
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
Type Description Title
UNKNOWN
Supplementary materials

Plümper and Traunmüller Dataset
Dataset

 Unknown

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 1
Total number of PDF views: 68 *
Loading metrics...

Abstract views

Total abstract views: 259 *
Loading metrics...

* Views captured on Cambridge Core between 29th August 2018 - 19th September 2018. This data will be updated every 24 hours.