Skip to main content
×
×
Home

How Robust Standard Errors Expose Methodological Problems They Do Not Fix, and What to Do About It

  • Gary King (a1) and Margaret E. Roberts (a2)
Abstract

“Robust standard errors” are used in a vast array of scholarship to correct standard errors for model misspecification. However, when misspecification is bad enough to make classical and robust standard errors diverge, assuming that it is nevertheless not so bad as to bias everything else requires considerable optimism. And even if the optimism is warranted, settling for a misspecified model, with or without robust standard errors, will still bias estimators of all but a few quantities of interest. The resulting cavernous gap between theory and practice suggests that considerable gains in applied statistics may be possible. We seek to help researchers realize these gains via a more productive way to understand and use robust standard errors; a new general and easier-to-use “generalized information matrix test” statistic that can formally assess misspecification (based on differences between robust and classical variance estimates); and practical illustrations via simulations and real examples from published research. How robust standard errors are used needs to change, but instead of jettisoning this popular tool we show how to use it to provide effective clues about model misspecification, likely biases, and a guide to considerably more reliable, and defensible, inferences. Accompanying this article is software that implements the methods we describe.

Copyright
Corresponding author
e-mail: king@harvard.edu (corresponding author)
Footnotes
Hide All

Authors' Note: Our thanks to Neal Beck, Tim Büthe, Andrew Hall, Helen Milner, Eric Neumayer, Rich Nielsen, Brandon Stewart, and Megan Westrum for many helpful comments, and David Zhang for expert research assistance. All data and information necessary to replicate our work are available in a Dataverse replication file at King and Roberts (2014).

Footnotes
References
Hide All
Abadie, Alberto, Imbens, Guido W., and Zheng, Fanyin. 2011. Robust inference for misspecified models conditional on covariates. The National Bureau of Economic Research Working Paper.
Arceneaux, Kevin, and Nickerson, David W. 2009. Modeling certainty with clustered data: A comparison of methods. Political Analysis 17(2): 177–90.
Arellano, Manuel. 1987. Computing robust standard errors for within-groups estimators. Oxford Bulletin of Economics and Statistics 49(4): 431–4.
Beck, Nathaniel, and Katz, Jonathan. 1995. What to do (and not to do) with time-series-cross-section data. American Political Science Review 89:634–47.
Bertrand, Marianne, Duflo, Esther, and Mullainathan, Sendhil. 2004. How much should we trust differences-in-differences estimates? Quarterly Journal of Economics 119(1): 249–75.
Breusch, T. S., and Pagan, A. R. 1979. Simple test for heteroscedasticity and random coefficient variation. Econometrica 47(5): 1287–94.
Büthe, Tim, and Milner, Helen V. 2008. The politics of foreign direct investment into developing countries: Increasing FDI through international trade agreements? American Journal of Political Science 52(4): 741–62.
Chesher, Andrew, and Spady, Richard. 1991. Asymptotic expansions of the information matrix test statistic. Econometrica: Journal of the Econometric Society 59:787815.
Davidson, Russell, and MacKinnon, James G. 1992. A new form of the information matrix test. Econometrica: Journal of the Econometric Society 54:145–57.
Dean, C., and Lawless, J. F. 1989. Tests for detecting overdispersion in Poisson regression models. Journal of the American Statistical Association 84(406): 467–72.
Dhaene, Geert, and Hoorelbeke, Dirk. 2004. The information matrix test with bootstrap-based covariance matrix estimation. Economics Letters 82(3): 341–47.
Dreher, Axel, and Jensen, Nathan M. 2007. Independent actor or agent? An empirical analysis of the impact of U.S. interests on international monetary fund conditions. Journal of Law and Economics 50(1): 105–24.
Driscoll, John C., and Kraay, Aart C. 1998. Consistent covariance matrix estimation with spatially dependent panel data. Review of Economics and Statistics 80:549–60.
Eicker, F. 1963. Asymptotic normality and consistency of the least squares estimators for families of linear regressions. Annals of Mathematical Statistics 34:447–56.
Fisher, Ronald A. 1935. The design of experiments. London: Oliver and Boyd.
Freedman, David A. 2006. On the so-called “Huber sandwich estimator” and “robust standard errors.” American Statistician 60(4): 299302.
Gartzke, Erik A., and Skrede Gleditsch, Kristian. 2008. The ties that bias: Specifying and operationalizing components of dyadic dependence in international conflict. Working Paper.
Goldberger, Arthur. 1991. A course in econometrics. Cambridge, MA: Harvard University Press.
Green, Donald P., and Vavreck, Lynn. 2008. Analysis of cluster-randomized experiments: A comparison of alternative estimation approaches. Political Analysis 15(2): 138–52.
Hoff, Peter D., and Ward, Michael D. 2004. Modeling dependencies in international relations networks. Political Analysis 12(2): 160–75.
Huber, Peter J. 1967. The behavior of maximum likelihood estimates under nonstandard conditions. Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability 1:221233.
Imai, Kosuke, King, Gary, and Lau, Olivia. 2008. Toward a common framework for statistical analysis and development. Journal of Computational Graphics and Statistics 17(4): 122.
Kiefer, Nicholas M. 1980. Estimation of fixed effect models for time series of cross-sections with arbitrary intertemporal covariance. Journal of Econometrics 14(2): 195202.
King, Gary. 1989a. Unifying political methodology: The likelihood theory of statistical inference. Ann Arbor: Michigan University Press.
King, Gary. 1989b. Variance specification in event count models: From restrictive assumptions to a generalized estimator. American Journal of Political Science 33(3): 762–84.
King, Gary, and Zeng, Langche. 2006. The dangers of extreme counterfactuals. Political Analysis 14(2): 131–59.
King, Gary, and Roberts, Margaret. 2014. Replication data for: How robust standard errors expose methodological problems they do not fix, and what to do about it. UNF:5:BclyVsbYLpjnS0Bx6FDnNA== http://dx.doi.org/10.7910/DVN/26935 IQSS Dataverse Network [Distributor].
King, Gary, Tomz, Michael, and Wittenberg, Jason. 2000. Making the most of statistical analyses: Improving interpretation and presentation. American Journal of Political Science 44(2): 341–55.
King, Gary, Keohane, Robert O., and Verba, Sidney. 1994. Designing social inquiry: Scientific inference in qualitative research. Princeton, NJ: Princeton University Press.
Koenker, Roger. 1981. A note on studentizing a test for heteroscedasticity. Journal of Econometrics 17(1): 107–12.
Koenker, Roger, and Bassett, Gilber. 1982. Robust for heteroscedasticity based on regression quantiles. Econometrica 50(1): 4361.
Lancaster, Tony. 1984. The covariance matrix of the information matrix test. Econometrica: Journal of the Econometric Society 52:1051–53.
Leamer, Edward E. 2010. Tantalus on the road to asymptopia. Journal of Economic Perspectives 24(2): 3146.
Moulton, Brent R. 1986. Random group effects and the precision of regression estimates. Journal of Econometrics 32:385–97.
Neumayer, Eric. 2003. The determinants of aid allocation by regional multilateral development banks and United Nations agencies. International Studies Quarterly 47(1): 101–22.
Newey, Whitney K., and West, Kenneth D. 1987. A simple, positive definite, heteroskedasticity and autocorrelation consistent covariance matrix. Econometrica 55(3): 703–8.
Neyman, J. 1923. On the application of probability theory to agricultural experiments. Essay on principles. Section 9. Statistical Science 5:465–80.
Orme, Chris. 1988. The calculation of the information matrix test for binary data models. Manchester School 56(4): 370–6.
Orme, Chris. 1990. The small-sample performance of the information-matrix test. Journal of Econometrics 46(3): 309–31.
Petersen, Mitchell. 2009. Estimating standard errors in finance panel data sets: Comparing approaches. Review of Financial Studies 22(1): 435–80.
Taylor, Larry W. 1987. The size bias of White's information matrix test. Economics Letters 24(1): 6367.
White, Halbert. 1980. A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica 48(4): 817–38.
White, Halbert. 1996. Estimation, inference, and specification analysis. New York: Cambridge University Press.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
MathJax

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed