Skip to main content Accessibility help
×
Home
Hostname: page-component-99c86f546-4hcbs Total loading time: 0.141 Render date: 2021-11-29T00:52:40.983Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "metricsAbstractViews": false, "figures": true, "newCiteModal": false, "newCitedByModal": true, "newEcommerce": true, "newUsageEvents": true }

A Quantitative Method for Substantive Robustness Assessment

Published online by Cambridge University Press:  23 September 2014

Abstract

Empirical political science is not simply about reporting evidence; it is also about coming to conclusions on the basis of that evidence and acting on those conclusions. But whether a result is substantively significant––strong and certain enough to justify acting upon the belief that the null hypothesis is false––is difficult to objectively pin down, in part because different researchers have different standards for interpreting evidence. Instead, this article advocates judging results according to their “substantive robustness,” the degree to which a community with heterogeneous standards for interpreting evidence would agree that the result is substantively significant. This study illustrates how this can be done using Bayesian statistical decision techniques. Judging results in this way yields a tangible benefit: false positives are reduced without decreasing the power of the test, thus decreasing the error rate in published results.

Type
Original Articles
Copyright
Copyright © The European Political Science Association 2014 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Justin Esarey is Assistant Professor of Political Science, Rice University, PO Box 1892, Houston, TX 77251-1892, USA (justin@justinesarey.com). Nathan Danneman is Data Scientist, L-3 Data Tactics, 7901 Jones Branch Dr. No. 700, McLean, VA, USA (ndanneman@gmail.com). This research was supported in part by a University Research Committee grant from Emory University. We thank Bill Berry, Jeff Staton, Tom Clark, Drew Linzer, Emily Ritter, John Gasper, Stu Jordan, Tim Salmon, David Siegel, Jackie DeMeritt, Deirdre McCloskey, Andrew Martin and respondents at presentations of this research at Emory University, Florida State University, Georgia State University, the University of Rochester, the 2010 Annual Meeting of the Society for Political Methodology, the 2010 Annual Meeting of the Midwest Political Science Association, and the 2009 Annual Meeting of the American Political Science Association for helpful comments and criticism. To view supplementary material for this article, please visit http://dx.doi.org/10.1017/psrm.2014.14.

References

Achen, Chris. 1982. Interpreting and Using Regression. Quantitative Applications in the Social Sciences. Sage University Press.CrossRefGoogle Scholar
Brambor, Thomas, Clark, William, and Golder, Matt. 2006. ‘Understanding Interaction Models: Improving Empirical Analyses’. Political Analysis 14:6382.CrossRefGoogle Scholar
Clark, W.R., and Golder, Matthew. 2006. ‘Rehabilitating Duverger’s Theory’. Comparative Political Studies 39(6):679708.CrossRefGoogle Scholar
Clinton, Joshua. 2006. ‘Representation in Congress: Constituents and Roll Calls in the 106th House’. Journal of Politics 68(2):397409.CrossRefGoogle Scholar
Clinton, Joshua, Jackman, Simon, and Rivers, Douglas. 2004. ‘The Statistical Analysis of Roll Call Data’. American Political Science Review 98:355370.CrossRefGoogle Scholar
DeGroot, Morris. 2004 {1970}. Optimal Statistical Decisions. Hoboken, NJ: Wiley Interscience.CrossRefGoogle Scholar
French, Simon, and Insua, David Rios. 2000. Statistical Decision Theory. London and New York: Oxford University Press.Google Scholar
Gelfand, Alan E., and Ghosh, Malay. 2000. ‘Generalized Linear Models: A Bayesian View’. In Generalized Linear Models: A Bayesian Perspective, edited by Sujit K. Ghosh and Bani K. Mallick, 322. New York: Marcel Dekker.Google Scholar
Gelman, Andrew, Carlin, John B., Stern, Hal S., and Rubin, Donald B.. 2003. Bayesian Data Analysis, Second Edition. Boca Raton, FL: Chapman & Hall/CRC.Google Scholar
Gill, Jeff. 1999. ‘The Insignificance of Null Hypothesis Significance Testing’. Political Research Quarterly 52(3):647674.CrossRefGoogle Scholar
Hunter, John E. 1997. ‘Needed: A Ban on the Significance Test’. Psychological Science 8:37.CrossRefGoogle Scholar
Lunt, Peter. 2004. ‘The Significance of the Significance Test Controversy: Comments on ‘Size Matters’’. The Journal of Socio-Economics 33:559564.CrossRefGoogle Scholar
Manski, Charles F. 2007. Identification for Prediction and Decision. Cambridge, MA: Harvard University Press.Google Scholar
McCloskey, Deirdre. 1998. The Rhetoric of Economics. Ann Arbor: University of Wisconsin Press.Google Scholar
Miller, Jane E. 2008. ‘Interpreting the Substantive Significance of Multivariable Regression Coefficients’. In 2008 Proceedings of the American Statistical Association, Statistical Education Section. Available at http://policy.rutgers.edu/faculty/miller/2008regression_ coefficients.pdfGoogle Scholar
Pratt, John W., Raiffa, Howard, and Schlaifer, Robert. 1996. Introduction to Statistical Decision Theory. Cambridge, MA: MIT Press.Google Scholar
Siegfried, Tom. 2010. ‘Odds Are, It’s Wrong: Science Fails to Face the Shortcomings of Statistics’. Science News 177:2629.CrossRefGoogle Scholar
Wald, Abraham. 1950. Statistical Decision Functions. Oxford: Wiley.Google Scholar
Ziliak, Steven, and McCloskey, Deirdre. 2004. ‘Size Matters: The Standard Error of Regressions in the American Economic Review’. The Journal of Socio-Economics 33:527546.CrossRefGoogle Scholar
Ziliak, Steven, and McCloskey, Deirdre. 2008. The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives. Ann Arbor: University of Michigan Press.Google Scholar
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 613 Bytes
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 8 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 4 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 6 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 11 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 8 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 8 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 101 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 8 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 8 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 690 Bytes
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 11 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 8 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 8 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 10 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 25 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 102 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 1 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 9 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 11 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 2 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 10 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 8 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 8 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 6 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 1 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 5 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 1 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 1 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 3 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 6 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 1 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 6 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 1 KB
6
Cited by

Send article to Kindle

To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

A Quantitative Method for Substantive Robustness Assessment
Available formats
×

Send article to Dropbox

To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

A Quantitative Method for Substantive Robustness Assessment
Available formats
×

Send article to Google Drive

To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

A Quantitative Method for Substantive Robustness Assessment
Available formats
×
×

Reply to: Submit a response

Please enter your response.

Your details

Please enter a valid email address.

Conflicting interests

Do you have any conflicting interests? *