Hostname: page-component-6766d58669-7fx5l Total loading time: 0 Render date: 2026-05-22T10:57:10.333Z Has data issue: false hasContentIssue false
  • English
  • Français

How Robust Standard Errors Expose Methodological Problems They Do Not Fix, and What to Do About It

Published online by Cambridge University Press:  04 January 2017

Gary King  and
Margaret E. Roberts
Gary King*
Affiliation:
Institute for Quantitative Social Science, 1737 Cambridge Street, Harvard University, Cambridge, MA 02138
Margaret E. Roberts
Affiliation:
Department of Political Science, 9500 Gilman Drive, #0521, University of California San Diego, La Jolla, CA 92093, e-mail: molly.e.roberts@gmail.com
*
e-mail: king@harvard.edu (corresponding author)
Get access Rights & Permissions [Opens in a new window]

Abstract

“Robust standard errors” are used in a vast array of scholarship to correct standard errors for model misspecification. However, when misspecification is bad enough to make classical and robust standard errors diverge, assuming that it is nevertheless not so bad as to bias everything else requires considerable optimism. And even if the optimism is warranted, settling for a misspecified model, with or without robust standard errors, will still bias estimators of all but a few quantities of interest. The resulting cavernous gap between theory and practice suggests that considerable gains in applied statistics may be possible. We seek to help researchers realize these gains via a more productive way to understand and use robust standard errors; a new general and easier-to-use “generalized information matrix test” statistic that can formally assess misspecification (based on differences between robust and classical variance estimates); and practical illustrations via simulations and real examples from published research. How robust standard errors are used needs to change, but instead of jettisoning this popular tool we show how to use it to provide effective clues about model misspecification, likely biases, and a guide to considerably more reliable, and defensible, inferences. Accompanying this article is software that implements the methods we describe.

Information

Type
Articles
Information
Political Analysis , Volume 23 , Issue 2 , Spring 2015 , pp. 159 - 179
Copyright
Copyright © The Author 2014. Published by Oxford University Press on behalf of the Society for Political Methodology 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

Authors' Note: Our thanks to Neal Beck, Tim Büthe, Andrew Hall, Helen Milner, Eric Neumayer, Rich Nielsen, Brandon Stewart, and Megan Westrum for many helpful comments, and David Zhang for expert research assistance. All data and information necessary to replicate our work are available in a Dataverse replication file at King and Roberts (2014).

References

Abadie, Alberto, Imbens, Guido W., and Zheng, Fanyin. 2011. Robust inference for misspecified models conditional on covariates. The National Bureau of Economic Research Working Paper.10.3386/w17442CrossRefGoogle Scholar
Arceneaux, Kevin, and Nickerson, David W. 2009. Modeling certainty with clustered data: A comparison of methods. Political Analysis 17(2): 177–90.CrossRefGoogle Scholar
Arellano, Manuel. 1987. Computing robust standard errors for within-groups estimators. Oxford Bulletin of Economics and Statistics 49(4): 431–4.10.1111/j.1468-0084.1987.mp49004006.xCrossRefGoogle Scholar
Beck, Nathaniel, and Katz, Jonathan. 1995. What to do (and not to do) with time-series-cross-section data. American Political Science Review 89:634–47.10.2307/2082979CrossRefGoogle Scholar
Bertrand, Marianne, Duflo, Esther, and Mullainathan, Sendhil. 2004. How much should we trust differences-in-differences estimates? Quarterly Journal of Economics 119(1): 249–75.CrossRefGoogle Scholar
Breusch, T. S., and Pagan, A. R. 1979. Simple test for heteroscedasticity and random coefficient variation. Econometrica 47(5): 1287–94.10.2307/1911963CrossRefGoogle Scholar
Büthe, Tim, and Milner, Helen V. 2008. The politics of foreign direct investment into developing countries: Increasing FDI through international trade agreements? American Journal of Political Science 52(4): 741–62.10.1111/j.1540-5907.2008.00340.xCrossRefGoogle Scholar
Chesher, Andrew, and Spady, Richard. 1991. Asymptotic expansions of the information matrix test statistic. Econometrica: Journal of the Econometric Society 59:787–815.10.2307/2938228CrossRefGoogle Scholar
Davidson, Russell, and MacKinnon, James G. 1992. A new form of the information matrix test. Econometrica: Journal of the Econometric Society 54:145–57.Google Scholar
Dean, C., and Lawless, J. F. 1989. Tests for detecting overdispersion in Poisson regression models. Journal of the American Statistical Association 84(406): 467–72.10.1080/01621459.1989.10478792CrossRefGoogle Scholar
Dhaene, Geert, and Hoorelbeke, Dirk. 2004. The information matrix test with bootstrap-based covariance matrix estimation. Economics Letters 82(3): 341–47.10.1016/j.econlet.2003.09.002CrossRefGoogle Scholar
Dreher, Axel, and Jensen, Nathan M. 2007. Independent actor or agent? An empirical analysis of the impact of U.S. interests on international monetary fund conditions. Journal of Law and Economics 50(1): 105–24.Google Scholar
Driscoll, John C., and Kraay, Aart C. 1998. Consistent covariance matrix estimation with spatially dependent panel data. Review of Economics and Statistics 80:549–60.CrossRefGoogle Scholar
Eicker, F. 1963. Asymptotic normality and consistency of the least squares estimators for families of linear regressions. Annals of Mathematical Statistics 34:447–56.10.1214/aoms/1177704156CrossRefGoogle Scholar
Fisher, Ronald A. 1935. The design of experiments. London: Oliver and Boyd.Google Scholar
Freedman, David A. 2006. On the so-called “Huber sandwich estimator” and “robust standard errors.” American Statistician 60(4): 299–302.10.1198/000313006X152207CrossRefGoogle Scholar
Gartzke, Erik A., and Skrede Gleditsch, Kristian. 2008. The ties that bias: Specifying and operationalizing components of dyadic dependence in international conflict. Working Paper.Google Scholar
Goldberger, Arthur. 1991. A course in econometrics. Cambridge, MA: Harvard University Press.Google Scholar
Green, Donald P., and Vavreck, Lynn. 2008. Analysis of cluster-randomized experiments: A comparison of alternative estimation approaches. Political Analysis 15(2): 138–52.Google Scholar
Hoff, Peter D., and Ward, Michael D. 2004. Modeling dependencies in international relations networks. Political Analysis 12(2): 160–75.10.1093/pan/mph012CrossRefGoogle Scholar
Huber, Peter J. 1967. The behavior of maximum likelihood estimates under nonstandard conditions. Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability 1:221–233.Google Scholar
Imai, Kosuke, King, Gary, and Lau, Olivia. 2008. Toward a common framework for statistical analysis and development. Journal of Computational Graphics and Statistics 17(4): 1–22.Google Scholar
Kiefer, Nicholas M. 1980. Estimation of fixed effect models for time series of cross-sections with arbitrary intertemporal covariance. Journal of Econometrics 14(2): 195–202.10.1016/0304-4076(80)90090-1CrossRefGoogle Scholar
King, Gary. 1989a. Unifying political methodology: The likelihood theory of statistical inference. Ann Arbor: Michigan University Press.Google Scholar
King, Gary. 1989b. Variance specification in event count models: From restrictive assumptions to a generalized estimator. American Journal of Political Science 33(3): 762–84.10.2307/2111071CrossRefGoogle Scholar
King, Gary, and Zeng, Langche. 2006. The dangers of extreme counterfactuals. Political Analysis 14(2): 131–59.10.1093/pan/mpj004CrossRefGoogle Scholar
King, Gary, and Roberts, Margaret. 2014. Replication data for: How robust standard errors expose methodological problems they do not fix, and what to do about it. UNF:5:BclyVsbYLpjnS0Bx6FDnNA== http://dx.doi.org/10.7910/DVN/26935 IQSS Dataverse Network [Distributor].CrossRefGoogle Scholar
King, Gary, Tomz, Michael, and Wittenberg, Jason. 2000. Making the most of statistical analyses: Improving interpretation and presentation. American Journal of Political Science 44(2): 341–55.CrossRefGoogle Scholar
King, Gary, Keohane, Robert O., and Verba, Sidney. 1994. Designing social inquiry: Scientific inference in qualitative research. Princeton, NJ: Princeton University Press.10.1515/9781400821211CrossRefGoogle Scholar
Koenker, Roger. 1981. A note on studentizing a test for heteroscedasticity. Journal of Econometrics 17(1): 107–12.10.1016/0304-4076(81)90062-2CrossRefGoogle Scholar
Koenker, Roger, and Bassett, Gilber. 1982. Robust for heteroscedasticity based on regression quantiles. Econometrica 50(1): 43–61.10.2307/1912528CrossRefGoogle Scholar
Lancaster, Tony. 1984. The covariance matrix of the information matrix test. Econometrica: Journal of the Econometric Society 52:1051–53.10.2307/1911198CrossRefGoogle Scholar
Leamer, Edward E. 2010. Tantalus on the road to asymptopia. Journal of Economic Perspectives 24(2): 31–46.10.1257/jep.24.2.31CrossRefGoogle Scholar
Moulton, Brent R. 1986. Random group effects and the precision of regression estimates. Journal of Econometrics 32:385–97.10.1016/0304-4076(86)90021-7CrossRefGoogle Scholar
Neumayer, Eric. 2003. The determinants of aid allocation by regional multilateral development banks and United Nations agencies. International Studies Quarterly 47(1): 101–22.10.1111/1468-2478.4701005CrossRefGoogle Scholar
Newey, Whitney K., and West, Kenneth D. 1987. A simple, positive definite, heteroskedasticity and autocorrelation consistent covariance matrix. Econometrica 55(3): 703–8.10.2307/1913610CrossRefGoogle Scholar
Neyman, J. 1923. On the application of probability theory to agricultural experiments. Essay on principles. Section 9. Statistical Science 5:465–80.Google Scholar
Orme, Chris. 1988. The calculation of the information matrix test for binary data models. Manchester School 56(4): 370–6.10.1111/j.1467-9957.1988.tb01339.xCrossRefGoogle Scholar
Orme, Chris. 1990. The small-sample performance of the information-matrix test. Journal of Econometrics 46(3): 309–31.10.1016/0304-4076(90)90012-ICrossRefGoogle Scholar
Petersen, Mitchell. 2009. Estimating standard errors in finance panel data sets: Comparing approaches. Review of Financial Studies 22(1): 435–80.10.1093/rfs/hhn053CrossRefGoogle Scholar
Taylor, Larry W. 1987. The size bias of White's information matrix test. Economics Letters 24(1): 63–67.10.1016/0165-1765(87)90182-0CrossRefGoogle Scholar
White, Halbert. 1980. A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica 48(4): 817–38.10.2307/1912934CrossRefGoogle Scholar
White, Halbert. 1996. Estimation, inference, and specification analysis. New York: Cambridge University Press.Google Scholar