Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Home
Hostname: page-component-768dbb666b-jrcft Total loading time: 0.336 Render date: 2023-02-06T20:46:23.105Z Has data issue: true Feature Flags: { "useRatesEcommerce": false } hasContentIssue true
  • English
Political Analysis Political Analysis

Article contents

Assessing Fit Quality and Testing for Misspecification in Binary-Dependent Variable Models

Published online by Cambridge University Press:  04 January 2017

Justin Esarey  and
Andrew Pierce
Justin Esarey*
Affiliation:
Department of Political Science, Rice University
Andrew Pierce
Affiliation:
Department of Political Science, Emory University. e-mail: awpierc@emory.edu
*
e-mail: justin@justinesarey.com (corresponding author)
Get access Rights & Permissions[Opens in a new window]

Abstract

In this article, we present a technique and critical test statistic for assessing the fit of a binary-dependent variable model (e.g., a logit or probit). We examine how closely a model's predicted probabilities match the observed frequency of events in the data set, and whether these deviations are systematic or merely noise. Our technique allows researchers to detect problems with a model's specification that obscure substantive understanding of the underlying data-generating process, such as missing interaction terms or unmodeled nonlinearities. We also show that these problems go undetected by the fit statistics most commonly used in political science.

Type
Research Article
Information
Political Analysis , Volume 20 , Issue 4 , Autumn 2012 , pp. 480 - 500
Copyright
Copyright © The Author 2012. Published by Oxford University Press on behalf of the Society for Political Methodology 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Authors' note: We thank Drew Linzer, Mike Ward, Jacqueline H. R. Demeritt, Jeff Staton, John Freeman, Neal Beck, Patrick Brandt, Phil Schrodt, Teppei Yamamoto, Kevin Clarke, and Will H. Moore for their comments, suggestions, and conversations about previous iterations of the article. Replication materials for all our simulations and data analysis can be found online at the Political Analysis dataverse: http://hdl.handle.net/1902.1/18399. Supplementary materials for the article are available on the Political Analysis Web site.

References

Achen, Christopher H. 2002. Toward a new methodology: Microfoundations and ART. Annual Reviews of Political Science 5: 423–50.CrossRefGoogle Scholar
Ai, Chunrong, and Norton, Edward C. 2003. Interaction terms in logit and probit models. Economics Letters 80: 123–9.CrossRefGoogle Scholar
Azzalini, A., Bowman, A. W., and Hardle, W. 1989. On the use of nonparametric regression for model checking. Biometrika 76: 1–11.CrossRefGoogle Scholar
Beck, Nathaniel, and Jackman, Simon. 1998. Beyond linearity by default: Generalized additive models. American Journal of Political Science 42: 596–627.CrossRefGoogle Scholar
Bowman, Adrian W., and Azzalini, Adelchi. 1997. Applied smoothing techniques for data analysis. Oxford: Oxford University Press.Google Scholar
Brambor, Thomas, Clark, William, and Golder, Matt. 2006. Understanding interaction models: Improving empirical analyses. Political Analysis 14: 63–82.CrossRefGoogle Scholar
Brown, Scott, and Heathcote, Andrew. 2002. On the use of nonparametric regression in assessing parametric regression models. Journal of Mathematical Psychology 46: 716–30.CrossRefGoogle Scholar
Carter, David B., and Signorino, Curtis S. 2010. Back to the future: Modeling time dependence in binary data. Political Analysis 18: 271–92.CrossRefGoogle Scholar
Cleveland, William S., and Loader, Clive. 1996. Smoothing by local regression: Principles and methods. In Statistical Theory and Computational Aspects of Smoothing, eds. Hardle, W., and Schimek, M. G., 10–49. Heidelberg, Germany: Springer.CrossRefGoogle Scholar
Cleveland, William S., and Devlin, Susan J. 1988. Locally weighted regression: An approach to regression analysis by local fitting. Journal of the American Statistical Association 83: 596–610.CrossRefGoogle Scholar
Copas, J. B. 1983. Plotting p against x. Journal of the Royal Statistical Society, Series C 32: 25–31.Google Scholar
Craven, Peter, and Wahba, Grace. 1979. Smoothing noisy data with spline functions. Numerische Mathematik 31: 377–403.CrossRefGoogle Scholar
Firth, D., Glosup, J., and Hinkley, D. V. 1991. Model checking with nonparametric curves. Biometrika 78: 245–52.CrossRefGoogle Scholar
Franzese, Robert J., and Kam, Cindy D. 2007. Modeling and interpreting interactive hypotheses in regression analysis. Ann Arbor: University of Michigan Press.CrossRefGoogle Scholar
Gartzke, Eric. 1999. War is in the error term. International Organization 53: 567–87.CrossRefGoogle Scholar
Gelman, Andrew, Carlin, John B., Stern, Hal, and Rubin, Donald B. 2004. Bayesian data analysis. Boca Raton, FL: Chapman and Hill/CRC.Google Scholar
Greenhill, Brian, Ward, Michael D., and Sacks, Audrey. 2011. The separation plot: A new visual method for evaluating the fit of binary models. American Journal of Political Science 55: 991–1002.CrossRefGoogle Scholar
Hardle, Wolfgang, Muller, Marlene, Sperlich, Stefan, and Werwatz, Alex. 2004. Nonparametric and semiparametric models. Berlin: Springer.CrossRefGoogle Scholar
Hart, Jeffrey D. 1997. Nonparametric smoothing and lack-of-fit tests. New York: Springer.CrossRefGoogle Scholar
Herron, Michael. 1999. Postestimation uncertainty in limited dependent variable models. Political Analysis 8: 83–98.CrossRefGoogle Scholar
Hosmer, D. W., Hosmer, T., Le Cessie, S., and Lemeshow, S. 1997. A comparison of goodness-of-fit tests for the logistic regression model. Statistics in Medicine 16: 965–80.3.0.CO;2-O>CrossRefGoogle ScholarPubMed
Hosmer, David W., and Lemeshow, Stanley. 1980. A goodness-of-fit test for the multiple logistic regression model. Communications in Statistics A10: 1043–69.Google Scholar
Hosmer, David W., and Lemeshow, Stanley. 2000. Applied logistic regression. New York: Wiley Interscience.CrossRefGoogle Scholar
Hurvich, Clifford M., Simonoff, Jeffrey S., and Tsai, Chih-Ling. 1998. Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion. Journal of the Royal Statistical Society, Series B 60: 271–93.CrossRefGoogle Scholar
Klein, James P., Goetz, Gary, and Diehl, Paul F. 2006. The new rivalry data set: Procedures and patterns. Journal of Peace Research 43: 331–48.CrossRefGoogle Scholar
le Cessie, S., and van Houwelingen, J. C. 1991. A goodness-of-fit test for binary regression models, based on smoothing methods. Biometrics 47: 1267–82.CrossRefGoogle Scholar
Lemeshow, Stanley, and Hosmer, David W. 1982. The use of goodness-of-fit statistics in the development of logistic regression models. American Journal of Epidemiology 115: 92–106.CrossRefGoogle ScholarPubMed
Macdonald, Peter D. M. 2011. R functions for ROC curves and the Hosmer-Lemeshow test (accessed 7 August 2012).Google Scholar
Morey, Daniel. 2011. When war brings peace: A dynamic model of the rivalry process. American Journal of Political Science 55: 263–75.CrossRefGoogle Scholar
Savun, Burcu, and Tirone, Daniel C. 2011. Foreign aid, democratization, and civil conflict: How does democracy aid civil conflict? American Journal of Political Science 55: 233–46.CrossRefGoogle Scholar
Ward, Michael D., Greenhill, Brian, and Bakke, Kristin. 2010. The perils of policy by p-value: Predicting civil conflicts. Journal of Peace Research 46: 363–75.Google Scholar
11
Cited by

Save article to Kindle

To save this article to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Assessing Fit Quality and Testing for Misspecification in Binary-Dependent Variable Models
Available formats
×

Save article to Dropbox

To save this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about saving content to Dropbox.

Assessing Fit Quality and Testing for Misspecification in Binary-Dependent Variable Models
Available formats
×

Save article to Google Drive

To save this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about saving content to Google Drive.

Assessing Fit Quality and Testing for Misspecification in Binary-Dependent Variable Models
Available formats
×
×

Reply to: Submit a response

Please enter your response.

Your details

Please enter a valid email address.

Conflicting interests

Do you have any conflicting interests? *