Skip to main content
×
×
Home

Assessing Fit Quality and Testing for Misspecification in Binary-Dependent Variable Models

  • Justin Esarey (a1) and Andrew Pierce (a2)
Abstract

In this article, we present a technique and critical test statistic for assessing the fit of a binary-dependent variable model (e.g., a logit or probit). We examine how closely a model's predicted probabilities match the observed frequency of events in the data set, and whether these deviations are systematic or merely noise. Our technique allows researchers to detect problems with a model's specification that obscure substantive understanding of the underlying data-generating process, such as missing interaction terms or unmodeled nonlinearities. We also show that these problems go undetected by the fit statistics most commonly used in political science.

Copyright
Corresponding author
e-mail: justin@justinesarey.com (corresponding author)
Footnotes
Hide All

Authors' note: We thank Drew Linzer, Mike Ward, Jacqueline H. R. Demeritt, Jeff Staton, John Freeman, Neal Beck, Patrick Brandt, Phil Schrodt, Teppei Yamamoto, Kevin Clarke, and Will H. Moore for their comments, suggestions, and conversations about previous iterations of the article. Replication materials for all our simulations and data analysis can be found online at the Political Analysis dataverse: http://hdl.handle.net/1902.1/18399. Supplementary materials for the article are available on the Political Analysis Web site.

Footnotes
References
Hide All
Achen, Christopher H. 2002. Toward a new methodology: Microfoundations and ART. Annual Reviews of Political Science 5: 423–50.
Ai, Chunrong, and Norton, Edward C. 2003. Interaction terms in logit and probit models. Economics Letters 80: 123–9.
Azzalini, A., Bowman, A. W., and Hardle, W. 1989. On the use of nonparametric regression for model checking. Biometrika 76: 111.
Beck, Nathaniel, and Jackman, Simon. 1998. Beyond linearity by default: Generalized additive models. American Journal of Political Science 42: 596627.
Bowman, Adrian W., and Azzalini, Adelchi. 1997. Applied smoothing techniques for data analysis. Oxford: Oxford University Press.
Brambor, Thomas, Clark, William, and Golder, Matt. 2006. Understanding interaction models: Improving empirical analyses. Political Analysis 14: 6382.
Brown, Scott, and Heathcote, Andrew. 2002. On the use of nonparametric regression in assessing parametric regression models. Journal of Mathematical Psychology 46: 716–30.
Carter, David B., and Signorino, Curtis S. 2010. Back to the future: Modeling time dependence in binary data. Political Analysis 18: 271–92.
Cleveland, William S., and Loader, Clive. 1996. Smoothing by local regression: Principles and methods. In Statistical Theory and Computational Aspects of Smoothing, eds. Hardle, W., and Schimek, M. G., 1049. Heidelberg, Germany: Springer.
Cleveland, William S., and Devlin, Susan J. 1988. Locally weighted regression: An approach to regression analysis by local fitting. Journal of the American Statistical Association 83: 596610.
Copas, J. B. 1983. Plotting p against x. Journal of the Royal Statistical Society, Series C 32: 2531.
Craven, Peter, and Wahba, Grace. 1979. Smoothing noisy data with spline functions. Numerische Mathematik 31: 377403.
Firth, D., Glosup, J., and Hinkley, D. V. 1991. Model checking with nonparametric curves. Biometrika 78: 245–52.
Franzese, Robert J., and Kam, Cindy D. 2007. Modeling and interpreting interactive hypotheses in regression analysis. Ann Arbor: University of Michigan Press.
Gartzke, Eric. 1999. War is in the error term. International Organization 53: 567–87.
Gelman, Andrew, Carlin, John B., Stern, Hal, and Rubin, Donald B. 2004. Bayesian data analysis. Boca Raton, FL: Chapman and Hill/CRC.
Greenhill, Brian, Ward, Michael D., and Sacks, Audrey. 2011. The separation plot: A new visual method for evaluating the fit of binary models. American Journal of Political Science 55: 9911002.
Hardle, Wolfgang, Muller, Marlene, Sperlich, Stefan, and Werwatz, Alex. 2004. Nonparametric and semiparametric models. Berlin: Springer.
Hart, Jeffrey D. 1997. Nonparametric smoothing and lack-of-fit tests. New York: Springer.
Herron, Michael. 1999. Postestimation uncertainty in limited dependent variable models. Political Analysis 8: 8398.
Hosmer, D. W., Hosmer, T., Le Cessie, S., and Lemeshow, S. 1997. A comparison of goodness-of-fit tests for the logistic regression model. Statistics in Medicine 16: 965–80.
Hosmer, David W., and Lemeshow, Stanley. 1980. A goodness-of-fit test for the multiple logistic regression model. Communications in Statistics A10: 1043–69.
Hosmer, David W., and Lemeshow, Stanley. 2000. Applied logistic regression. New York: Wiley Interscience.
Hurvich, Clifford M., Simonoff, Jeffrey S., and Tsai, Chih-Ling. 1998. Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion. Journal of the Royal Statistical Society, Series B 60: 271–93.
Klein, James P., Goetz, Gary, and Diehl, Paul F. 2006. The new rivalry data set: Procedures and patterns. Journal of Peace Research 43: 331–48.
le Cessie, S., and van Houwelingen, J. C. 1991. A goodness-of-fit test for binary regression models, based on smoothing methods. Biometrics 47: 1267–82.
Lemeshow, Stanley, and Hosmer, David W. 1982. The use of goodness-of-fit statistics in the development of logistic regression models. American Journal of Epidemiology 115: 92106.
Macdonald, Peter D. M. 2011. R functions for ROC curves and the Hosmer-Lemeshow test (accessed 7 August 2012).
Morey, Daniel. 2011. When war brings peace: A dynamic model of the rivalry process. American Journal of Political Science 55: 263–75.
Savun, Burcu, and Tirone, Daniel C. 2011. Foreign aid, democratization, and civil conflict: How does democracy aid civil conflict? American Journal of Political Science 55: 233–46.
Ward, Michael D., Greenhill, Brian, and Bakke, Kristin. 2010. The perils of policy by p-value: Predicting civil conflicts. Journal of Peace Research 46: 363–75.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
MathJax

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed