Skip to main content

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
Cancel
×
×
Home
  • Home
Article
  • Access

Computerized Adaptive Testing for Public Opinion Surveys

  • Jacob M. Montgomery (a1) and Josh Cutler (a2)
Abstract

Survey researchers avoid using large multi-item scales to measure latent traits due to both the financial costs and the risk of driving up nonresponse rates. Typically, investigators select a subset of available scale items rather than asking the full battery. Reduced batteries, however, can sharply reduce measurement precision and introduce bias. In this article, we present computerized adaptive testing (CAT) as a method for minimizing the number of questions each respondent must answer while preserving measurement accuracy and precision. CAT algorithms respond to individuals' previous answers to select subsequent questions that most efficiently reveal respondents' positions on a latent dimension. We introduce the basic stages of a CAT algorithm and present the details for one approach to item selection appropriate for public opinion research. We then demonstrate the advantages of CAT via simulation and empirically comparing dynamic and static measures of political knowledge.

    •
      • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Computerized Adaptive Testing for Public Opinion Surveys
      Available formats
      ×
      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      Computerized Adaptive Testing for Public Opinion Surveys
      Available formats
      ×
      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      Computerized Adaptive Testing for Public Opinion Surveys
      Available formats
      ×
Copyright
COPYRIGHT: © The Author 2013. Published by Oxford University Press on behalf of the Society for Political Methodology 
Corresponding author
e-mail: jacob.montgomery@wustl.edu (corresponding author)
Footnotes
Hide All

Authors' note: We are grateful for helpful comments provided by Martin Elff, Sunshine Hillygus, Walter Mebane, Brendan Nyhan, and two anonymous reviewers. A previous version of this article was presented at the 2012 Annual Meeting of the Midwest Political Science Association, the 2012 Saint Louis Area Methods Meeting, and the 2012 Summer Methods Meeting. Supplementary materials for this article are available on the Political Analysis Web site.

Footnotes
References
Hide All
Anderson, A., Basilevsky, A., and Hum, D. 1983. Missing data: A review of the literature. In Handboook of survey research, eds. Rossi, P. H., Wright, J. D., and Anderson, A. B., 415–81. New York: Academic Press.
Bafumi, J., Gelman, A., Park, D. K., and Kaplan, N. 2005. Practical issues in implementing and understanding Bayesian ideal point estimation. Political Analysis 13(2): 171–87.
Bafumi, J., and Herron, M. C. 2010. Leapfrog representation and extremism: A study of American voters and their members in Congress. American Political Science Review 104(3): 519–42.
Bailey, M. A. 2007. Comparable preference estimates across time and institutions for the court, Congress, and presidency. American Journal of Political Science 51(3): 433–48.
Baker, F. B., and Kim, S.-H. 2004. Item response theory: Parameter estimation techniques. New York: Marcel Dekker.
Barabas, J. 2002. Another look at the measurement of political knowledge. Political Analysis 10(2): 209–22.
Berinsky, A. J., Huber, G. A., and Lenz, G. S. 2012. Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk. Political Analysis 20(3): 329–50.
Brewer, P. R. 2003. Values, political knowledge, and public opinion about gay rights. Public Opinion Quarterly 67(3): 173–201.
Burchell, B., and Marsh, C. 1992. The effect of questionnaire length on survey response. Quality & Quantity 26(3): 233–44.
Cacioppo, J. T., and Petty, R. E. 1984. The efficient assessment of need for cognition. Journal of Personality Assessment 48(3): 306–7.
Choi, S. W., and Swartz, R. J. 2009. Comparison of CAT item selection criteria for polytomous items. Applied Psychological Measurement 33(6): 419–40.
Clinton, J. D., and Meirowitz, A. 2001. Agenda constrained legislator ideal points and the spatial voting model. Political Analysis 9(3): 242–59.
Clinton, J. D., and Meirowitz, A. 2003. Integrating voting theory and roll call analysis: A framework. Political Analysis 11(4): 381–96.
Clinton, J., Jackman, S., and Rivers, D. 2004. The statistical analysis of roll call voting: A unified approach. American Political Science Review 98(2): 355–70.
Crawford, S. D., Couper, M. P., and Lamias, M. J. 2001. Web surveys: Perceptions of burden. Social Science Computer Review 19(2): 146–62.
DeBell, M. 2012. Harder than it looks: Coding political knowledge on the ANES. Paper presented at the 2012 meeting of the Midwest Political Science Association, Chicago, IL.
Delli Carpini, M. X., and Keeter, S. 1993. Measuring political knowledge: Putting first things first. American Journal of Political Science 37(4): 1179–206.
Delli Carpini, M. X., and Keeter, S. 1996. What Americans know about politics and why it matters. New Haven, CT: Yale University Press.
Dodd, B. G., De Ayala, R., and Koch, W. R. 1995. Computerized adaptive testing with polytomous items. Applied Psychological Measurement 19(1): 5–22.
Embretson, S. E., and Reise, S. P. 2000. Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.
Feldman, S., and Huddy, L. 2005. Racial resentment and white opposition to race-conscious programs: Principles or prejudice? American Journal of Political Science 49(1): 168–83.
Forbey, J. D., and Ben-Porath, Y. S. 2007. Computerized adaptive personality testing: A review and illustration with the MMPI-2 computerized adaptive version. Psychological Assessment 19(1): 14–24.
Galesic, M., and Bosnjak, M. 2009. Effects of questionnaire length on participation and indicators of response quality in Web surveys. Public Opinion Quarterly 73(2): 349–60.
Gerber, A. S., Huber, G. A., Doherty, D., Dowling, C. M., and Ha, S. E. 2010. Personality and political attitudes: Relationships across issue domains and political contexts. American Political Science Review 104(01): 111–33.
Gibson, J. L., and Caldeira, G. A. 2009. Knowing the Supreme Court? A reconsideration of public ignorance of the high court. Journal of Politics 71(2): 429–41.
Gillion, D. Q. 2012. Re-defining political participation through item response theory. Unpublished paper.
Gomez, B. T., and Wilson, J. M. 2001. Political sophistication and economic voting in the American electorate: A theory of heterogeneous attribution. American Journal of Political Science 45(4): 899–914.
Gosling, S. D., Rentfrow, P. J., and Swann, W. B. 2003. A very brief measure of the big-five personality domains. Journal of Research in Personality 37(6): 504–28.
Heberlein, T. A., and Baumgartner, R. 1978. Factors affecting response rates to mailed questionnaires: A quantitative analysis of the published literature. American Sociological Review 43(4): 447–62.
Herzog, A. R., and Bachman, J. G. 1981. Effects of questionnaire length on response quality. Public Opinion Quarterly 45(4): 549–59.
Hol, A. M., Vorst, H. C., and Mellenbergh, G. J. 2007. Computerized adaptive testing for polytomous motivation items: Administration mode effects and a comparison with short forms. Applied Psychological Measurement 31(5): 412–29.
Jackman, S. 2001. Multidimensional analysis of roll call data via Bayesian simulation: Identification, estimation, inference, and model checking. Political Analysis 9(3): 227–41.
Kingsbury, G., and Weiss, D. J. 1983. A comparison of IRT-based adaptive mastery testing and a sequential mastery testing procedure. In New horizons in testing: Latent trait test theory and computerized adaptive testing, ed. Weiss, D. J. New York: Academic Press.
Krosnick, J. A. 1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology 5(3): 213–36.
Krosnick, J. A. 1999. Survey research. Annual Review of Psychology 50: 537–67.
Krosnick, J. A., Holbrook, A. L., Berent, M. K., Carson, R. A. B. T., Hanemann, W., Kopp, R. J., Mitchell, C., Cameron, R., Presser, S., Ruud, P. A., Smith, V., Moody, W. R., Green, M. C., and Conaway, M. 2002. The impact of “no opinion“ response options on data quality: Non-attitude reduction or an invitation to satisfice? Public Opinion Quarterly 66(3): 371–403.
Lord, F. M. 1980. Applications of item response theory to practical testing problems. Hillsdale, NJ: L. Erlbaum Associates.
Lord, F., and Novick, M. R. 1968. Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
Lupia, A. 2006. How elitism undermines the study of voter competence. Critical Review 18 (1–3): 217–32.
Lupia, A. 2008. Procedural transparency and the credibility of election surveys. Electoral Studies 27(4): 732–9.
Luskin, R. C. 1987. Measuring political sophistication. American Journal of Political Science 31(4): 856–99.
Luskin, R. C., and Bullock, J. G. 2011. “Don't know” means “don't know”: DK responses and the public's level of political knowledge. Journal of Politics 73(2): 547–57.
Martin, A. D., and Quinn, K. M. 2002. Dynamic ideal point estimation via Markov chain Monte Carlo for the US Supreme Court, 1953–1999. Political Analysis 10(2): 134–53.
Matthews, R. A., Kath, L. M., and Barnes-Farrell, J. L. 2010. A short, valid, predictive measure of work-family conflict: Item selection and scale validation. Journal of Occupational Health Psychology 15(1): 75–90.
Mondak, J. J., and Anderson, M. R. 2004. The knowledge gap: A reexamination of gender-based differences in political knowledge. Journal of Politics 66(2): 492–512.
Mondak, J. J., and Davis, B. C. 2001. Asked and answered: Knowledge levels when we will not take “don't know” for an answer. Political Behavior 23(3): 199–224.
Mondak, J. J. 2001. Developing valid knowledge scales. American Journal of Political Science 45(1): 224–38.
Montgomery, J. M., and Cutler, J. 2012. Replication data for: Computerized adaptive testing for public opinion surveys. http://hdl.handle.net/1902.1/19381 IQSS Dataverse Network.
Piazza, T., Sniderman, P. M., and Tetlock, P. E. 1989. Analysis of the dynamics of political reasoning: A general-purpose computer-assisted methodology. Political Analysis 1(1): 99–119.
Podsakoff, P. M., and MacKenzie, S. B. 1994. An examination of the psychometric properties and nomological validity of some revised and reduced substitutes for leadership scales. Journal of Applied Psychology 79(5): 702–13.
Poole, K. T. 2005. Spatial models of parliamentary voting. New York: Cambridge University Press.
Prior, M., and Lupia, A. 2008. Money, time, and political knowledge: Distinguishing quick recall and political learning skills. American Journal of Political Science 52(1): 19–183.
Prior, M. 2012. Visual political knowledge: A different road to competence. Unpublished paper.
Richins, M. L. 2004. The material values scale: Measurement properties and development of a short form. Journal of Consumer Research 31(1): 209–19.
Russell, S. S., Spitzmüller, C., Lin, L. F., Stanton, J. M., Smith, P. C., and Ironson, G. H. 2004. Shorter can also be better: The abridged job in general scale. Educational and Psychological Measurement 64(5): 878–93.
Segall, D. O. 2002. Confirmatory item factor analysis using Markov chain Monte Carlo estimation with applications to online calibration in CAT. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.
Segall, D. O. 2005. Computerized adaptive testing. In Encyclopedia of social measurement, ed. Kempf-Leonard, K., Vol. 1, 429–38. Oxford, UK: Elsevier.
Segall, D. O. 2010. Principles of multidemensional adaptive testing. In Elements of adaptive testing, eds. van der Linden, W. J. and Glas, C. A. W., 57–76. New York: Springer.
Sheatsley, P. 1983. Questionnaire construction and item writing. In Handboook of survey research, eds. Rossi, P. H., Wright, J. D., and Anderson, A. B., 195–230. New York: Academic Press.
Singh, J., Howell, R. D., and Rhoads, G. K. 2007. Designs for Likert-type data: An approach for implementing marketing surveys. Journal of Marketing Research 19(1): 12–24.
Sniderman, P. M., Brody, R. A., and Tetlock, P. E. 1991. Reasoning and choice: Explorations in political psychology. New York: Cambridge University Press.
Sniderman, P. M., Piazza, T., Tetlock, P. E., and Kendrick, A. 1991. The new racism. American Journal of Political Science 35(2): 423–47.
Stanton, J. M., Sinar, E. F., Balzer, W. K., and Smith, P. C. 2002. Issues and strategies for reducing the length of self-report scales. Personnel Psychology 55(1): 167–94.
Thompson, E. R. 2012. A brief index of affective job satisfaction. Group & Organization Management 37(3): 275–307.
Treier, S., and Hillygus, D. S. 2009. The nature of political ideology in the contemporary electorate. Public Opinion Quarterly 73(4): 679–703.
Treier, S., and Jackman, S. 2008. Democracy as a latent variable. American Journal of Political Science 52(1): 201–17.
van der Linden, W. J. 1998. Bayesian item selection criteria for adaptive testing. Psychometrika 63(2): 201–16.
van der Linden, W. J. 1999. Empirical initialization of the trait estimator in adaptive testing. Applied Psychological Measurement 23(1): 21–29.
van der Linden, W. J. 2008. Using response times for item selection in adaptive testing. Journal of Educational and Behavioral Statistics 33(1): 5–20.
van der Linden, W. J. 2010. Constrained adaptive testing with shadow tests. In Elements of adaptive testing, eds. van der Linden, W. J. and Glas, C. A. W., 31–56. New York: Springer.
van der Linden, W. J., and Pashley, P. J. 2010. Elements of adaptive testing. New York: Springer.
Verba, S., Schlozman, K. L., and Brady, H. E. 1995. Voice and equality: Civic voluntarism in American politics. Cambridge, MA: Harvard University Press.
Wainer, H. 1990. Introduction and history. In Computerized Adaptive Testing: A Primer, ed. Wainer, H. Hillsdale, NJ: Lawrence Erlbaum Associates.
Waller, N. G., and Reise, S. P. 1989. Computerized adaptive personality assessment: An illustration with the absorption scale. Journal of Personality and Social Psychology 57(6): 1051.
Weiss, D. J. 1982. Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement 6(4): 473–92.
Weiss, D. J., and Kingsbury, G. G. 1984. Application of computerized adaptive testing to educational problems. Journal of Educational Measurement 21(4): 361–75.
Xu, X., and Douglas, J. 2006. Computerized adaptive testing under nonparamteric IRT models. Psychometrika 71(1): 121–37.
Yammarino, F. J., Skinner, S. J., and Childers, T. L. 1991. Understanding mail survey response behavior: A meta-analysis. Public Opinion Quarterly 55(4): 613–639.
Zaller, J. R. 1992. The nature and origins of mass opinion. New York: Cambridge University Press.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *

CAPTCHA *

Skip to the audio challenge
×
MathJax
Type Description Title
PDF
Supplementary materials

Montgomery and Cutler supplementary material
Appendix

 PDF (52 KB)
52 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed

Cancel
Confirm
×