Skip to main content Accessibility help
×
Home
Hostname: page-component-558cb97cc8-24rpz Total loading time: 0.417 Render date: 2022-10-07T20:23:47.264Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "useRatesEcommerce": false, "displayNetworkTab": true, "displayNetworkMapGraph": false, "useSa": true } hasContentIssue true

The sensitivity of sensitivity analysis

Published online by Cambridge University Press:  29 August 2018

Thomas Plümper*
Affiliation:
Department of Socioeconomics, Vienna University of Economics, Vienna, Austria
Richard Traunmüller
Affiliation:
Department of Social Sciences, Goethe University Frankfurt am Main, Germany
*
*Corresponding author. Email: thomas.pluemper@wu.ac.at

Abstract

This article evaluates the reliability of sensitivity tests. Using Monte Carlo methods we show that, first, the definition of robustness exerts a large influence on the robustness of variables. Second and more importantly, our results also demonstrate that inferences based on sensitivity tests are most likely to be valid if determinants and confounders are almost uncorrelated and if the variables included in the true model exert a strong influence on outcomes. Third, no definition of robustness reliably avoids both false positives and false negatives. We find that for a wide variety of data-generating processes, rarely used definitions of robustness perform better than the frequently used model averaging rule suggested by Sala-i-Martin. Fourth, our results also suggest that Leamer’s extreme bounds analysis and Bayesian model averaging are extremely unlikely to generate false positives. Thus, if based on these inferential criteria a variable is robust, it is almost certain to belong into the empirical model. Fifth and finally, we also show that researchers should avoid drawing inferences based on lack of robustness.

Type
Research Notes
Copyright
Copyright © The European Political Science Association 2018 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Bartels, LM (1997) Specification Uncertainty and Model Averaging. American Journal of Political Science 41, 641674.CrossRefGoogle Scholar
Cederman, LE, Wimmer, A Min, B (2010) Why do Ethnic Groups Rebel. New Data and Analysis. World Politics 62(1), 87119.CrossRefGoogle Scholar
Gleditsch, KS (2007) Transnational Dimensions of Civil War. Journal of Peace Research 44(3), 293309.CrossRefGoogle Scholar
Granger, CWJ Uhlig, HF (1990) Reasonable Extreme Bounds Analysis. Journal of Econometrics 44, 159170.CrossRefGoogle Scholar
Hafner-Burton, EM (2005) Right or Robust. The Sensitive Nature of Repression to Globalization. Journal of Peace Research 42(6), 679698.CrossRefGoogle Scholar
Hegre, H Sambanis, N (2006) Sensitivity Analysis of Empirical Results on Civil War Onset. Journal of Conflict Resolution 50(4), 508535.CrossRefGoogle Scholar
Hoeting, JA, Madigan, D, Raftery, AE Volinsky, CT (1999) Bayesian Model Averaging: A Tutorial. Statistical Science 14, 382401.Google Scholar
Leamer, EE (1978) Specification Searches: Ad Hoc Inference with Nonexperimental Data. New York: Wiley.Google Scholar
Leamer, EE (1983) Let’s Take the Con Out of Econometrics. American Economic Review 73(1), 3143.Google Scholar
Leamer, EE (1985) Sensitivity Analysis Would Help. American Economic Review 57(3), 308313.Google Scholar
Leamer, EE (2010) Tantalus on the Road to Asymptopia. The Journal of Economic Perspectives 24(2), 3146.CrossRefGoogle Scholar
Leamer, E Leonard, H (1983) Reporting the Fragility of Regression Estimates. The Review of Economics and Statistics 65(2), 306317.CrossRefGoogle Scholar
Levine, R Renelt, D (1992) A Sensitivity Analysis of Cross-Country Growth Regressions. American Economic Review 82(4), 942963.Google Scholar
Neumayer, E Plümper, T (2017) Robustness Tests for Quantitative Research. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Raftery, AE (1995) Bayesian Model Selection in Social Research. Sociological Methodology 25, 111163.CrossRefGoogle Scholar
Sala-i-Martin, X (1997) I Just Ran Two Million Regressions. American Economic Review 87(2), 178183.Google Scholar
Sturm, JE, Berger, H de Haan, J (2005) Which Variables Explain Decisions on IMF Credit. An Extreme Bounds Analysis. Economics & Politics 17(2), 177213.CrossRefGoogle Scholar
Sturm, JE de Haan, J (2005) Determinants of Long-Term Growth: New Results Applying Estimation and Extreme Bounds Analysis. Empirical Economics 30(3), 597617.CrossRefGoogle Scholar
Supplementary material: Link

Plümper and Traunmüller Dataset

Link
2
Cited by

Save article to Kindle

To save this article to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

The sensitivity of sensitivity analysis
Available formats
×

Save article to Dropbox

To save this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about saving content to Dropbox.

The sensitivity of sensitivity analysis
Available formats
×

Save article to Google Drive

To save this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about saving content to Google Drive.

The sensitivity of sensitivity analysis
Available formats
×
×

Reply to: Submit a response

Please enter your response.

Your details

Please enter a valid email address.

Conflicting interests

Do you have any conflicting interests? *