Skip to main content Accessibility help
×
Home

A Quantitative Method for Substantive Robustness Assessment

  • Justin Esarey and Nathan Danneman

Abstract

Empirical political science is not simply about reporting evidence; it is also about coming to conclusions on the basis of that evidence and acting on those conclusions. But whether a result is substantively significant––strong and certain enough to justify acting upon the belief that the null hypothesis is false––is difficult to objectively pin down, in part because different researchers have different standards for interpreting evidence. Instead, this article advocates judging results according to their “substantive robustness,” the degree to which a community with heterogeneous standards for interpreting evidence would agree that the result is substantively significant. This study illustrates how this can be done using Bayesian statistical decision techniques. Judging results in this way yields a tangible benefit: false positives are reduced without decreasing the power of the test, thus decreasing the error rate in published results.

Copyright

Footnotes

Hide All
Justin Esarey is Assistant Professor of Political Science, Rice University, PO Box 1892, Houston, TX 77251-1892, USA (justin@justinesarey.com). Nathan Danneman is Data Scientist, L-3 Data Tactics, 7901 Jones Branch Dr. No. 700, McLean, VA, USA (ndanneman@gmail.com). This research was supported in part by a University Research Committee grant from Emory University. We thank Bill Berry, Jeff Staton, Tom Clark, Drew Linzer, Emily Ritter, John Gasper, Stu Jordan, Tim Salmon, David Siegel, Jackie DeMeritt, Deirdre McCloskey, Andrew Martin and respondents at presentations of this research at Emory University, Florida State University, Georgia State University, the University of Rochester, the 2010 Annual Meeting of the Society for Political Methodology, the 2010 Annual Meeting of the Midwest Political Science Association, and the 2009 Annual Meeting of the American Political Science Association for helpful comments and criticism. To view supplementary material for this article, please visit http://dx.doi.org/10.1017/psrm.2014.14.

Footnotes

References

Hide All
Achen, Chris. 1982. Interpreting and Using Regression. Quantitative Applications in the Social Sciences. Sage University Press.
Brambor, Thomas, Clark, William, and Golder, Matt. 2006. ‘Understanding Interaction Models: Improving Empirical Analyses’. Political Analysis 14:6382.
Clark, W.R., and Golder, Matthew. 2006. ‘Rehabilitating Duverger’s Theory’. Comparative Political Studies 39(6):679708.
Clinton, Joshua. 2006. ‘Representation in Congress: Constituents and Roll Calls in the 106th House’. Journal of Politics 68(2):397409.
Clinton, Joshua, Jackman, Simon, and Rivers, Douglas. 2004. ‘The Statistical Analysis of Roll Call Data’. American Political Science Review 98:355370.
DeGroot, Morris. 2004 {1970}. Optimal Statistical Decisions. Hoboken, NJ: Wiley Interscience.
French, Simon, and Insua, David Rios. 2000. Statistical Decision Theory. London and New York: Oxford University Press.
Gelfand, Alan E., and Ghosh, Malay. 2000. ‘Generalized Linear Models: A Bayesian View’. In Generalized Linear Models: A Bayesian Perspective, edited by Sujit K. Ghosh and Bani K. Mallick, 322. New York: Marcel Dekker.
Gelman, Andrew, Carlin, John B., Stern, Hal S., and Rubin, Donald B.. 2003. Bayesian Data Analysis, Second Edition. Boca Raton, FL: Chapman & Hall/CRC.
Gill, Jeff. 1999. ‘The Insignificance of Null Hypothesis Significance Testing’. Political Research Quarterly 52(3):647674.
Hunter, John E. 1997. ‘Needed: A Ban on the Significance Test’. Psychological Science 8:37.
Lunt, Peter. 2004. ‘The Significance of the Significance Test Controversy: Comments on ‘Size Matters’’. The Journal of Socio-Economics 33:559564.
Manski, Charles F. 2007. Identification for Prediction and Decision. Cambridge, MA: Harvard University Press.
McCloskey, Deirdre. 1998. The Rhetoric of Economics. Ann Arbor: University of Wisconsin Press.
Miller, Jane E. 2008. ‘Interpreting the Substantive Significance of Multivariable Regression Coefficients’. In 2008 Proceedings of the American Statistical Association, Statistical Education Section. Available at http://policy.rutgers.edu/faculty/miller/2008regression_ coefficients.pdf
Pratt, John W., Raiffa, Howard, and Schlaifer, Robert. 1996. Introduction to Statistical Decision Theory. Cambridge, MA: MIT Press.
Siegfried, Tom. 2010. ‘Odds Are, It’s Wrong: Science Fails to Face the Shortcomings of Statistics’. Science News 177:2629.
Wald, Abraham. 1950. Statistical Decision Functions. Oxford: Wiley.
Ziliak, Steven, and McCloskey, Deirdre. 2004. ‘Size Matters: The Standard Error of Regressions in the American Economic Review’. The Journal of Socio-Economics 33:527546.
Ziliak, Steven, and McCloskey, Deirdre. 2008. The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives. Ann Arbor: University of Michigan Press.

Related content

Powered by UNSILO
Type Description Title
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (613 bytes)
613 bytes
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (8 KB)
8 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (4 KB)
4 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (6 KB)
6 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (11 KB)
11 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (8 KB)
8 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (8 KB)
8 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (101 KB)
101 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (8 KB)
8 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (8 KB)
8 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (690 bytes)
690 bytes
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (11 KB)
11 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (8 KB)
8 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (8 KB)
8 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (10 KB)
10 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (25 KB)
25 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (102 KB)
102 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (1 KB)
1 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (9 KB)
9 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (11 KB)
11 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (2 KB)
2 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (10 KB)
10 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (8 KB)
8 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (8 KB)
8 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (6 KB)
6 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (1 KB)
1 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (5 KB)
5 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (1 KB)
1 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (1 KB)
1 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (3 KB)
3 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (6 KB)
6 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (1 KB)
1 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (6 KB)
6 KB
UNKNOWN
Supplementary materials

Esarey and Danneman Supplementary Material
Supplementary Material

 Unknown (1 KB)
1 KB

A Quantitative Method for Substantive Robustness Assessment

  • Justin Esarey and Nathan Danneman

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed.