Hostname: page-component-76fb5796d-9pm4c Total loading time: 0 Render date: 2024-04-26T22:35:23.324Z Has data issue: false hasContentIssue false
  • English
  • Français

What Do We Learn from Graduate Admissions Committees? A Multiple Rater, Latent Variable Model, with Incomplete Discrete and Continuous Indicators

Published online by Cambridge University Press:  04 January 2017

Simon Jackman
Simon Jackman*
Affiliation:
Department of Political Science, Stanford University, Stanford, CA 94305–6044. e-mail: jackman@leland.stanford.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

What do we really know about applicants to graduate school? How much information is in an applicant's file? What do we learn by having graduate admissions committees read and score applicant files? In this article, I develop a statistical model for measuring applicant quality, combining the information in the committee members' ordinal ratings with the information in applicants' GRE scores. The model produces estimates of applicant quality purged of the influence of committee members' preferences over ostensibly extraneous applicant characteristics, such as gender and intended field of study. An explicitly Bayesian approach is adopted for estimation and inference, making it straightforward to obtain confidence intervals not only on latent applicant quality but over rank orderings of applicants and the probability of belonging in a set of likely admittees. Using data from applications to a highly ranked political science graduate program, I show that there is considerable uncertainty in estimates of applicant quality, making it impossible to make authoritative distinctions as to quality among large portions of the applicant pool. The multiple rater model I develop here is extremely flexible and has applications in fields as diverse as judicial politics, legislative politics, international relations, and public opinion.

Type
Research Article
Information
Political Analysis , Volume 12 , Issue 4: Special Issue on Bayesian Methods , Autumn 2004 , pp. 400 - 424
Copyright
Copyright © Society for Political Methodology 2004 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Aldrich, John H., and McKelvey, Richard D. 1977. “A Method of Scaling with Applications to the 1968 and 1972 Presidential Elections.” American Political Science Review 71: 111–130.CrossRefGoogle Scholar
Castles, Francis G., and Mair, Peter. 1984. “Left-Right Political Scales: Some ‘Expert’ Judgements.” European Journal of Political Research 12: 73–88.Google Scholar
Clinton, Joshua, Jackman, Simon, and Rivers, Douglas. Forthcoming. “The Statistical Analysis of Roll Call Data.” American Political Science Review 98 (2).Google Scholar
Clinton, Joshua D., and Lapinski, John S. 2004. “Measuring Significant Legislation: Assessing Congressional Output from 1877–1948.” Presented at the Annual Meetings of the Midwestern Political Science Association, Chicago, IL.Google Scholar
Cox, N. R. 1974. “Estimation of the Correlation between a Continuous and a Discrete Variable.” Biometrics 30: 171–178.Google Scholar
Edwards, Yancy D., and Allenby, Greg M. 2003. “Multivariate Analysis of Multiple Response Data.” Journal of Marketing Research 40: 321–334.Google Scholar
Erikson, Robert S. 1990. “Roll Calls, Reputations, and Representation in the U.S. Senate.” Legislative Studies Quarterly 15: 623–642.CrossRefGoogle Scholar
Franklin, Charles. 1991. “Eschewing Obfuscation? Campaigns and the Perception of U.S. Senate Incumbents.” American Political Science Review 85: 1193–1214.CrossRefGoogle Scholar
Hoff, Peter, Raftery, Adrian E., and Handcock, Mark S. 2002. “Latent Space Approaches to Social Network Analysis.” Journal of the American Statistical Association 97: 1090–1098.Google Scholar
Huber, John, and Inglehart, Ronald. 1995. “Expert Interpretations of Party Space and Party Locations in 42 Societies.” Party Politics 1: 73–111.Google Scholar
Jöreskog, Karl G., and Sörbom, Dag. 1993. New Features in LISREL 8. Chicago: Scientific Software International.Google Scholar
Johnson, Valen E., and Albert, James H. 1999. Ordinal Data Modeling. New York: Springer-Verlag.Google Scholar
King, Gary, Bruce, John M., and Gilligan, Michael. 1993. “The Science of Political Science Graduate Admissions.” PS: Political Science and Politics 26: 772–778.Google Scholar
Laver, Michael, and Hunt, Ben W. 1992. Policy and Party Competition. London: Routledge.Google Scholar
Law, David S. 2004. Strategic Judicial Lawmaking: An Empirical Investigation of Ideology and Publication on the U.S. Court of Appeals for the Ninth Circuit PhD thesis Department of Political Science, Stanford University.Google Scholar
Loader, C. 1999. Local Regression and Likelihood. New York: Springer.Google Scholar
Martin, Andrew D., and Quinn, Kevin M. 2002. “Dynamic Ideal Point Estimation via Markov Chain Monte Carlo for the U.S. Supreme Court, 1953–1999.” Political Analysis 10: 134–153.Google Scholar
Neal, Radford M. 1997. “Markov Chain Monte Carlo Methods Based on ‘Slicing’ the Density Function.” Technical Report No. 9722. Department of Statistics, University of Toronto.Google Scholar
Noel, Hans. 2004. “The Spatial Structure of Ideological Discourse in America: A Hierarchical Model for Estimating Ideal Points with a Paucity of Data.” Presented at the Annual Meetings of the Midwestern Political Science Association, Chicago, IL.Google Scholar
Olsson, Ulf, Drasgow, Fritz, and Dorans, Neil J. 1982. “The Polyserial Correlation Coefficient.” Psychometrika 47: 337–347.CrossRefGoogle Scholar
Poole, Keith T., and Rosenthal, Howard. 1997. Congress: A Political-Economic History of Roll Call Voting. New York: Oxford University Press.Google Scholar
Rivers, Douglas. 2003. “Identification of Multidimensional Item-Response Models.” Typescript. Department of Political Science, Stanford University.Google Scholar
Spiegelhalter, David J., and Lauritzen, S. L. 1990. “Sequential Updating of Conditional Probabilities on Directed Graphical Structures.” Networks 20: 579–605.CrossRefGoogle Scholar
Spiegelhalter, David J., Thomas, Andrew, and Best, Nicky G. 1996. “Computation on Bayesian Graphical Models.” In Bayesian Statistics 5, eds. Bernardo, J. M., Berger, J. O., Dawid, A. P., and Smith, A. F. M. Oxford: Oxford University Press, pp. 407–425.CrossRefGoogle Scholar
Spiegelhalter, David J., Thomas, Andrew, Best, Nicky, and Lunn, Dave. 2003. WinBUGS User Manual Version 1.4. Cambridge, UK: MRC Biostatistics Unit.Google Scholar