Skip to main content
×
Home
    • Aa
    • Aa

How Robust Standard Errors Expose Methodological Problems They Do Not Fix, and What to Do About It

  • Gary King (a1) and Margaret E. Roberts (a2)
Abstract

“Robust standard errors” are used in a vast array of scholarship to correct standard errors for model misspecification. However, when misspecification is bad enough to make classical and robust standard errors diverge, assuming that it is nevertheless not so bad as to bias everything else requires considerable optimism. And even if the optimism is warranted, settling for a misspecified model, with or without robust standard errors, will still bias estimators of all but a few quantities of interest. The resulting cavernous gap between theory and practice suggests that considerable gains in applied statistics may be possible. We seek to help researchers realize these gains via a more productive way to understand and use robust standard errors; a new general and easier-to-use “generalized information matrix test” statistic that can formally assess misspecification (based on differences between robust and classical variance estimates); and practical illustrations via simulations and real examples from published research. How robust standard errors are used needs to change, but instead of jettisoning this popular tool we show how to use it to provide effective clues about model misspecification, likely biases, and a guide to considerably more reliable, and defensible, inferences. Accompanying this article is software that implements the methods we describe.

Copyright
Corresponding author
e-mail: king@harvard.edu (corresponding author)
Footnotes
Hide All

Authors' Note: Our thanks to Neal Beck, Tim Büthe, Andrew Hall, Helen Milner, Eric Neumayer, Rich Nielsen, Brandon Stewart, and Megan Westrum for many helpful comments, and David Zhang for expert research assistance. All data and information necessary to replicate our work are available in a Dataverse replication file at King and Roberts (2014).

Footnotes
Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

Alberto Abadie , Guido W. Imbens , and Fanyin Zheng . 2011. Robust inference for misspecified models conditional on covariates. The National Bureau of Economic Research Working Paper.

Alberto Abadie , Guido W. Imbens , and Fanyin Zheng . 2011. Robust inference for misspecified models conditional on covariates. The National Bureau of Economic Research Working Paper.

Kevin Arceneaux , and David W. Nickerson 2009. Modeling certainty with clustered data: A comparison of methods. Political Analysis 17(2): 177–90.

Kevin Arceneaux , and David W. Nickerson 2009. Modeling certainty with clustered data: A comparison of methods. Political Analysis 17(2): 177–90.

Nathaniel Beck , and Jonathan Katz . 1995. What to do (and not to do) with time-series-cross-section data. American Political Science Review 89:634–47.

Nathaniel Beck , and Jonathan Katz . 1995. What to do (and not to do) with time-series-cross-section data. American Political Science Review 89:634–47.

Marianne Bertrand , Esther Duflo , and Sendhil Mullainathan . 2004. How much should we trust differences-in-differences estimates? Quarterly Journal of Economics 119(1): 249–75.

Marianne Bertrand , Esther Duflo , and Sendhil Mullainathan . 2004. How much should we trust differences-in-differences estimates? Quarterly Journal of Economics 119(1): 249–75.

T. S. Breusch , and A. R. Pagan 1979. Simple test for heteroscedasticity and random coefficient variation. Econometrica 47(5): 1287–94.

T. S. Breusch , and A. R. Pagan 1979. Simple test for heteroscedasticity and random coefficient variation. Econometrica 47(5): 1287–94.

Tim Büthe , and Helen V. Milner 2008. The politics of foreign direct investment into developing countries: Increasing FDI through international trade agreements? American Journal of Political Science 52(4): 741–62.

Tim Büthe , and Helen V. Milner 2008. The politics of foreign direct investment into developing countries: Increasing FDI through international trade agreements? American Journal of Political Science 52(4): 741–62.

Andrew Chesher , and Richard Spady . 1991. Asymptotic expansions of the information matrix test statistic. Econometrica: Journal of the Econometric Society 59:787815.

Andrew Chesher , and Richard Spady . 1991. Asymptotic expansions of the information matrix test statistic. Econometrica: Journal of the Econometric Society 59:787815.

C. Dean , and J. F. Lawless 1989. Tests for detecting overdispersion in Poisson regression models. Journal of the American Statistical Association 84(406): 467–72.

C. Dean , and J. F. Lawless 1989. Tests for detecting overdispersion in Poisson regression models. Journal of the American Statistical Association 84(406): 467–72.

Geert Dhaene , and Dirk Hoorelbeke . 2004. The information matrix test with bootstrap-based covariance matrix estimation. Economics Letters 82(3): 341–47.

Geert Dhaene , and Dirk Hoorelbeke . 2004. The information matrix test with bootstrap-based covariance matrix estimation. Economics Letters 82(3): 341–47.

Axel Dreher , and Nathan M. Jensen 2007. Independent actor or agent? An empirical analysis of the impact of U.S. interests on international monetary fund conditions. Journal of Law and Economics 50(1): 105–24.

Axel Dreher , and Nathan M. Jensen 2007. Independent actor or agent? An empirical analysis of the impact of U.S. interests on international monetary fund conditions. Journal of Law and Economics 50(1): 105–24.

John C. Driscoll , and Aart C. Kraay 1998. Consistent covariance matrix estimation with spatially dependent panel data. Review of Economics and Statistics 80:549–60.

John C. Driscoll , and Aart C. Kraay 1998. Consistent covariance matrix estimation with spatially dependent panel data. Review of Economics and Statistics 80:549–60.

F. Eicker 1963. Asymptotic normality and consistency of the least squares estimators for families of linear regressions. Annals of Mathematical Statistics 34:447–56.

F. Eicker 1963. Asymptotic normality and consistency of the least squares estimators for families of linear regressions. Annals of Mathematical Statistics 34:447–56.

David A. Freedman 2006. On the so-called “Huber sandwich estimator” and “robust standard errors.” American Statistician 60(4): 299302.

David A. Freedman 2006. On the so-called “Huber sandwich estimator” and “robust standard errors.” American Statistician 60(4): 299302.

Peter D. Hoff , and Michael D. Ward 2004. Modeling dependencies in international relations networks. Political Analysis 12(2): 160–75.

Peter D. Hoff , and Michael D. Ward 2004. Modeling dependencies in international relations networks. Political Analysis 12(2): 160–75.

Nicholas M. Kiefer 1980. Estimation of fixed effect models for time series of cross-sections with arbitrary intertemporal covariance. Journal of Econometrics 14(2): 195202.

Nicholas M. Kiefer 1980. Estimation of fixed effect models for time series of cross-sections with arbitrary intertemporal covariance. Journal of Econometrics 14(2): 195202.

Gary King . 1989b. Variance specification in event count models: From restrictive assumptions to a generalized estimator. American Journal of Political Science 33(3): 762–84.

Gary King . 1989b. Variance specification in event count models: From restrictive assumptions to a generalized estimator. American Journal of Political Science 33(3): 762–84.

Gary King , and Langche Zeng . 2006. The dangers of extreme counterfactuals. Political Analysis 14(2): 131–59.

Gary King , and Langche Zeng . 2006. The dangers of extreme counterfactuals. Political Analysis 14(2): 131–59.

Gary King , and Margaret Roberts . 2014. Replication data for: How robust standard errors expose methodological problems they do not fix, and what to do about it. UNF:5:BclyVsbYLpjnS0Bx6FDnNA== http://dx.doi.org/10.7910/DVN/26935 IQSS Dataverse Network [Distributor].

Gary King , and Margaret Roberts . 2014. Replication data for: How robust standard errors expose methodological problems they do not fix, and what to do about it. UNF:5:BclyVsbYLpjnS0Bx6FDnNA== http://dx.doi.org/10.7910/DVN/26935 IQSS Dataverse Network [Distributor].

Gary King , Michael Tomz , and Jason Wittenberg . 2000. Making the most of statistical analyses: Improving interpretation and presentation. American Journal of Political Science 44(2): 341–55.

Gary King , Michael Tomz , and Jason Wittenberg . 2000. Making the most of statistical analyses: Improving interpretation and presentation. American Journal of Political Science 44(2): 341–55.

Roger Koenker . 1981. A note on studentizing a test for heteroscedasticity. Journal of Econometrics 17(1): 107–12.

Roger Koenker . 1981. A note on studentizing a test for heteroscedasticity. Journal of Econometrics 17(1): 107–12.

Roger Koenker , and Gilber Bassett . 1982. Robust for heteroscedasticity based on regression quantiles. Econometrica 50(1): 4361.

Roger Koenker , and Gilber Bassett . 1982. Robust for heteroscedasticity based on regression quantiles. Econometrica 50(1): 4361.

Tony Lancaster . 1984. The covariance matrix of the information matrix test. Econometrica: Journal of the Econometric Society 52:1051–53.

Tony Lancaster . 1984. The covariance matrix of the information matrix test. Econometrica: Journal of the Econometric Society 52:1051–53.

Edward E. Leamer 2010. Tantalus on the road to asymptopia. Journal of Economic Perspectives 24(2): 3146.

Edward E. Leamer 2010. Tantalus on the road to asymptopia. Journal of Economic Perspectives 24(2): 3146.

Brent R. Moulton 1986. Random group effects and the precision of regression estimates. Journal of Econometrics 32:385–97.

Brent R. Moulton 1986. Random group effects and the precision of regression estimates. Journal of Econometrics 32:385–97.

Eric Neumayer . 2003. The determinants of aid allocation by regional multilateral development banks and United Nations agencies. International Studies Quarterly 47(1): 101–22.

Eric Neumayer . 2003. The determinants of aid allocation by regional multilateral development banks and United Nations agencies. International Studies Quarterly 47(1): 101–22.

Whitney K. Newey , and Kenneth D. West 1987. A simple, positive definite, heteroskedasticity and autocorrelation consistent covariance matrix. Econometrica 55(3): 703–8.

Whitney K. Newey , and Kenneth D. West 1987. A simple, positive definite, heteroskedasticity and autocorrelation consistent covariance matrix. Econometrica 55(3): 703–8.

Chris Orme . 1988. The calculation of the information matrix test for binary data models. Manchester School 56(4): 370–6.

Chris Orme . 1988. The calculation of the information matrix test for binary data models. Manchester School 56(4): 370–6.

Chris Orme . 1990. The small-sample performance of the information-matrix test. Journal of Econometrics 46(3): 309–31.

Chris Orme . 1990. The small-sample performance of the information-matrix test. Journal of Econometrics 46(3): 309–31.

Mitchell Petersen . 2009. Estimating standard errors in finance panel data sets: Comparing approaches. Review of Financial Studies 22(1): 435–80.

Mitchell Petersen . 2009. Estimating standard errors in finance panel data sets: Comparing approaches. Review of Financial Studies 22(1): 435–80.

Larry W. Taylor 1987. The size bias of White's information matrix test. Economics Letters 24(1): 6367.

Larry W. Taylor 1987. The size bias of White's information matrix test. Economics Letters 24(1): 6367.

Halbert White . 1980. A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica 48(4): 817–38.

Halbert White . 1980. A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica 48(4): 817–38.

Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
MathJax

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 46 *
Loading metrics...

Abstract views

Total abstract views: 216 *
Loading metrics...

* Views captured on Cambridge Core between 4th January 2017 - 23rd July 2017. This data will be updated every 24 hours.