Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Home
  • Home
EnglishFrançais
Political Analysis Political Analysis

Article contents

Computerized Adaptive Testing for Public Opinion Surveys

Published online by Cambridge University Press:  04 January 2017

Jacob M. Montgomery  and
Josh Cutler
Jacob M. Montgomery
Affiliation:
Department of Political Science, Washington University in St. Louis, Campus Box 1063, One Brookings Drive, St Louis, MO 63130-4899
Josh Cutler
Affiliation:
Department of Political Science, Duke University, 326 Perkins Library, Box 90204, Durham, NC 27708
Corresponding
E-mail address:
jacob.montgomery@wustl.edu
Rights & Permissions[Opens in a new window]

Abstract

Survey researchers avoid using large multi-item scales to measure latent traits due to both the financial costs and the risk of driving up nonresponse rates. Typically, investigators select a subset of available scale items rather than asking the full battery. Reduced batteries, however, can sharply reduce measurement precision and introduce bias. In this article, we present computerized adaptive testing (CAT) as a method for minimizing the number of questions each respondent must answer while preserving measurement accuracy and precision. CAT algorithms respond to individuals' previous answers to select subsequent questions that most efficiently reveal respondents' positions on a latent dimension. We introduce the basic stages of a CAT algorithm and present the details for one approach to item selection appropriate for public opinion research. We then demonstrate the advantages of CAT via simulation and empirically comparing dynamic and static measures of political knowledge.

Type
Research Article
Information
Political Analysis , Volume 21 , Issue 2 , Spring 2013 , pp. 172 - 192
Copyright
Copyright © The Author 2013. Published by Oxford University Press on behalf of the Society for Political Methodology 

Footnotes

Authors' note: We are grateful for helpful comments provided by Martin Elff, Sunshine Hillygus, Walter Mebane, Brendan Nyhan, and two anonymous reviewers. A previous version of this article was presented at the 2012 Annual Meeting of the Midwest Political Science Association, the 2012 Saint Louis Area Methods Meeting, and the 2012 Summer Methods Meeting. Supplementary materials for this article are available on the Political Analysis Web site.

References

Anderson, A., Basilevsky, A., and Hum, D. 1983. Missing data: A review of the literature. In Handboook of survey research, eds. Rossi, P. H., Wright, J. D., and Anderson, A. B., 415–81. New York: Academic Press.Google Scholar
Bafumi, J., Gelman, A., Park, D. K., and Kaplan, N. 2005. Practical issues in implementing and understanding Bayesian ideal point estimation. Political Analysis 13(2): 171–87.CrossRefGoogle Scholar
Bafumi, J., and Herron, M. C. 2010. Leapfrog representation and extremism: A study of American voters and their members in Congress. American Political Science Review 104(3): 519–42.CrossRefGoogle Scholar
Bailey, M. A. 2007. Comparable preference estimates across time and institutions for the court, Congress, and presidency. American Journal of Political Science 51(3): 433–48.CrossRefGoogle Scholar
Baker, F. B., and Kim, S.-H. 2004. Item response theory: Parameter estimation techniques. New York: Marcel Dekker.Google Scholar
Barabas, J. 2002. Another look at the measurement of political knowledge. Political Analysis 10(2): 209–22.CrossRefGoogle Scholar
Berinsky, A. J., Huber, G. A., and Lenz, G. S. 2012. Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk. Political Analysis 20(3): 329–50.CrossRefGoogle Scholar
Brewer, P. R. 2003. Values, political knowledge, and public opinion about gay rights. Public Opinion Quarterly 67(3): 173–201.CrossRefGoogle Scholar
Burchell, B., and Marsh, C. 1992. The effect of questionnaire length on survey response. Quality & Quantity 26(3): 233–44.CrossRefGoogle Scholar
Cacioppo, J. T., and Petty, R. E. 1984. The efficient assessment of need for cognition. Journal of Personality Assessment 48(3): 306–7.CrossRefGoogle ScholarPubMed
Choi, S. W., and Swartz, R. J. 2009. Comparison of CAT item selection criteria for polytomous items. Applied Psychological Measurement 33(6): 419–40.CrossRefGoogle ScholarPubMed
Clinton, J. D., and Meirowitz, A. 2001. Agenda constrained legislator ideal points and the spatial voting model. Political Analysis 9(3): 242–59.CrossRefGoogle Scholar
Clinton, J. D., and Meirowitz, A. 2003. Integrating voting theory and roll call analysis: A framework. Political Analysis 11(4): 381–96.CrossRefGoogle Scholar
Clinton, J., Jackman, S., and Rivers, D. 2004. The statistical analysis of roll call voting: A unified approach. American Political Science Review 98(2): 355–70.CrossRefGoogle Scholar
Crawford, S. D., Couper, M. P., and Lamias, M. J. 2001. Web surveys: Perceptions of burden. Social Science Computer Review 19(2): 146–62.CrossRefGoogle Scholar
DeBell, M. 2012. Harder than it looks: Coding political knowledge on the ANES. Paper presented at the 2012 meeting of the Midwest Political Science Association, Chicago, IL.Google Scholar
Delli Carpini, M. X., and Keeter, S. 1993. Measuring political knowledge: Putting first things first. American Journal of Political Science 37(4): 1179–206.Google Scholar
Delli Carpini, M. X., and Keeter, S. 1996. What Americans know about politics and why it matters. New Haven, CT: Yale University Press.Google Scholar
Dodd, B. G., De Ayala, R., and Koch, W. R. 1995. Computerized adaptive testing with polytomous items. Applied Psychological Measurement 19(1): 5–22.CrossRefGoogle Scholar
Embretson, S. E., and Reise, S. P. 2000. Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.Google Scholar
Feldman, S., and Huddy, L. 2005. Racial resentment and white opposition to race-conscious programs: Principles or prejudice? American Journal of Political Science 49(1): 168–83.CrossRefGoogle Scholar
Forbey, J. D., and Ben-Porath, Y. S. 2007. Computerized adaptive personality testing: A review and illustration with the MMPI-2 computerized adaptive version. Psychological Assessment 19(1): 14–24.CrossRefGoogle ScholarPubMed
Galesic, M., and Bosnjak, M. 2009. Effects of questionnaire length on participation and indicators of response quality in Web surveys. Public Opinion Quarterly 73(2): 349–60.CrossRefGoogle Scholar
Gerber, A. S., Huber, G. A., Doherty, D., Dowling, C. M., and Ha, S. E. 2010. Personality and political attitudes: Relationships across issue domains and political contexts. American Political Science Review 104(01): 111–33.CrossRefGoogle Scholar
Gibson, J. L., and Caldeira, G. A. 2009. Knowing the Supreme Court? A reconsideration of public ignorance of the high court. Journal of Politics 71(2): 429–41.CrossRefGoogle Scholar
Gillion, D. Q. 2012. Re-defining political participation through item response theory. Unpublished paper.Google Scholar
Gomez, B. T., and Wilson, J. M. 2001. Political sophistication and economic voting in the American electorate: A theory of heterogeneous attribution. American Journal of Political Science 45(4): 899–914.CrossRefGoogle Scholar
Gosling, S. D., Rentfrow, P. J., and Swann, W. B. 2003. A very brief measure of the big-five personality domains. Journal of Research in Personality 37(6): 504–28.CrossRefGoogle Scholar
Heberlein, T. A., and Baumgartner, R. 1978. Factors affecting response rates to mailed questionnaires: A quantitative analysis of the published literature. American Sociological Review 43(4): 447–62.CrossRefGoogle Scholar
Herzog, A. R., and Bachman, J. G. 1981. Effects of questionnaire length on response quality. Public Opinion Quarterly 45(4): 549–59.CrossRefGoogle Scholar
Hol, A. M., Vorst, H. C., and Mellenbergh, G. J. 2007. Computerized adaptive testing for polytomous motivation items: Administration mode effects and a comparison with short forms. Applied Psychological Measurement 31(5): 412–29.CrossRefGoogle Scholar
Jackman, S. 2001. Multidimensional analysis of roll call data via Bayesian simulation: Identification, estimation, inference, and model checking. Political Analysis 9(3): 227–41.CrossRefGoogle Scholar
Kingsbury, G., and Weiss, D. J. 1983. A comparison of IRT-based adaptive mastery testing and a sequential mastery testing procedure. In New horizons in testing: Latent trait test theory and computerized adaptive testing, ed. Weiss, D. J. New York: Academic Press.Google Scholar
Krosnick, J. A. 1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology 5(3): 213–36.CrossRefGoogle Scholar
Krosnick, J. A. 1999. Survey research. Annual Review of Psychology 50: 537–67.CrossRefGoogle ScholarPubMed
Krosnick, J. A., Holbrook, A. L., Berent, M. K., Carson, R. A. B. T., Hanemann, W., Kopp, R. J., Mitchell, C., Cameron, R., Presser, S., Ruud, P. A., Smith, V., Moody, W. R., Green, M. C., and Conaway, M. 2002. The impact of “no opinion“ response options on data quality: Non-attitude reduction or an invitation to satisfice? Public Opinion Quarterly 66(3): 371–403.CrossRefGoogle Scholar
Lord, F. M. 1980. Applications of item response theory to practical testing problems. Hillsdale, NJ: L. Erlbaum Associates.Google Scholar
Lord, F., and Novick, M. R. 1968. Statistical theories of mental test scores. Reading, MA: Addison-Wesley.Google Scholar
Lupia, A. 2006. How elitism undermines the study of voter competence. Critical Review 18 (1–3): 217–32.CrossRefGoogle Scholar
Lupia, A. 2008. Procedural transparency and the credibility of election surveys. Electoral Studies 27(4): 732–9.CrossRefGoogle Scholar
Luskin, R. C. 1987. Measuring political sophistication. American Journal of Political Science 31(4): 856–99.CrossRefGoogle Scholar
Luskin, R. C., and Bullock, J. G. 2011. “Don't know” means “don't know”: DK responses and the public's level of political knowledge. Journal of Politics 73(2): 547–57.CrossRefGoogle Scholar
Martin, A. D., and Quinn, K. M. 2002. Dynamic ideal point estimation via Markov chain Monte Carlo for the US Supreme Court, 1953–1999. Political Analysis 10(2): 134–53.CrossRefGoogle Scholar
Matthews, R. A., Kath, L. M., and Barnes-Farrell, J. L. 2010. A short, valid, predictive measure of work-family conflict: Item selection and scale validation. Journal of Occupational Health Psychology 15(1): 75–90.CrossRefGoogle Scholar
Mondak, J. J., and Anderson, M. R. 2004. The knowledge gap: A reexamination of gender-based differences in political knowledge. Journal of Politics 66(2): 492–512.CrossRefGoogle Scholar
Mondak, J. J., and Davis, B. C. 2001. Asked and answered: Knowledge levels when we will not take “don't know” for an answer. Political Behavior 23(3): 199–224.CrossRefGoogle Scholar
Mondak, J. J. 2001. Developing valid knowledge scales. American Journal of Political Science 45(1): 224–38.CrossRefGoogle Scholar
Montgomery, J. M., and Cutler, J. 2012. Replication data for: Computerized adaptive testing for public opinion surveys. http://hdl.handle.net/1902.1/19381 IQSS Dataverse Network.Google Scholar
Piazza, T., Sniderman, P. M., and Tetlock, P. E. 1989. Analysis of the dynamics of political reasoning: A general-purpose computer-assisted methodology. Political Analysis 1(1): 99–119.CrossRefGoogle Scholar
Podsakoff, P. M., and MacKenzie, S. B. 1994. An examination of the psychometric properties and nomological validity of some revised and reduced substitutes for leadership scales. Journal of Applied Psychology 79(5): 702–13.CrossRefGoogle Scholar
Poole, K. T. 2005. Spatial models of parliamentary voting. New York: Cambridge University Press.CrossRefGoogle Scholar
Prior, M., and Lupia, A. 2008. Money, time, and political knowledge: Distinguishing quick recall and political learning skills. American Journal of Political Science 52(1): 19–183.CrossRefGoogle Scholar
Prior, M. 2012. Visual political knowledge: A different road to competence. Unpublished paper.Google Scholar
Richins, M. L. 2004. The material values scale: Measurement properties and development of a short form. Journal of Consumer Research 31(1): 209–19.CrossRefGoogle Scholar
Russell, S. S., Spitzmüller, C., Lin, L. F., Stanton, J. M., Smith, P. C., and Ironson, G. H. 2004. Shorter can also be better: The abridged job in general scale. Educational and Psychological Measurement 64(5): 878–93.CrossRefGoogle Scholar
Segall, D. O. 2002. Confirmatory item factor analysis using Markov chain Monte Carlo estimation with applications to online calibration in CAT. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.Google Scholar
Segall, D. O. 2005. Computerized adaptive testing. In Encyclopedia of social measurement, ed. Kempf-Leonard, K., Vol. 1, 429–38. Oxford, UK: Elsevier.Google Scholar
Segall, D. O. 2010. Principles of multidemensional adaptive testing. In Elements of adaptive testing, eds. van der Linden, W. J. and Glas, C. A. W., 57–76. New York: Springer.Google Scholar
Sheatsley, P. 1983. Questionnaire construction and item writing. In Handboook of survey research, eds. Rossi, P. H., Wright, J. D., and Anderson, A. B., 195–230. New York: Academic Press.CrossRefGoogle Scholar
Singh, J., Howell, R. D., and Rhoads, G. K. 2007. Designs for Likert-type data: An approach for implementing marketing surveys. Journal of Marketing Research 19(1): 12–24.Google Scholar
Sniderman, P. M., Brody, R. A., and Tetlock, P. E. 1991. Reasoning and choice: Explorations in political psychology. New York: Cambridge University Press.CrossRefGoogle Scholar
Sniderman, P. M., Piazza, T., Tetlock, P. E., and Kendrick, A. 1991. The new racism. American Journal of Political Science 35(2): 423–47.CrossRefGoogle Scholar
Stanton, J. M., Sinar, E. F., Balzer, W. K., and Smith, P. C. 2002. Issues and strategies for reducing the length of self-report scales. Personnel Psychology 55(1): 167–94.CrossRefGoogle Scholar
Thompson, E. R. 2012. A brief index of affective job satisfaction. Group & Organization Management 37(3): 275–307.CrossRefGoogle Scholar
Treier, S., and Hillygus, D. S. 2009. The nature of political ideology in the contemporary electorate. Public Opinion Quarterly 73(4): 679–703.CrossRefGoogle Scholar
Treier, S., and Jackman, S. 2008. Democracy as a latent variable. American Journal of Political Science 52(1): 201–17.CrossRefGoogle Scholar
van der Linden, W. J. 1998. Bayesian item selection criteria for adaptive testing. Psychometrika 63(2): 201–16.CrossRefGoogle Scholar
van der Linden, W. J. 1999. Empirical initialization of the trait estimator in adaptive testing. Applied Psychological Measurement 23(1): 21–29.CrossRefGoogle Scholar
van der Linden, W. J. 2008. Using response times for item selection in adaptive testing. Journal of Educational and Behavioral Statistics 33(1): 5–20.CrossRefGoogle Scholar
van der Linden, W. J. 2010. Constrained adaptive testing with shadow tests. In Elements of adaptive testing, eds. van der Linden, W. J. and Glas, C. A. W., 31–56. New York: Springer.CrossRefGoogle Scholar
van der Linden, W. J., and Pashley, P. J. 2010. Elements of adaptive testing. New York: Springer.CrossRefGoogle Scholar
Verba, S., Schlozman, K. L., and Brady, H. E. 1995. Voice and equality: Civic voluntarism in American politics. Cambridge, MA: Harvard University Press.Google Scholar
Wainer, H. 1990. Introduction and history. In Computerized Adaptive Testing: A Primer, ed. Wainer, H. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
Waller, N. G., and Reise, S. P. 1989. Computerized adaptive personality assessment: An illustration with the absorption scale. Journal of Personality and Social Psychology 57(6): 1051.CrossRefGoogle ScholarPubMed
Weiss, D. J. 1982. Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement 6(4): 473–92.CrossRefGoogle Scholar
Weiss, D. J., and Kingsbury, G. G. 1984. Application of computerized adaptive testing to educational problems. Journal of Educational Measurement 21(4): 361–75.CrossRefGoogle Scholar
Xu, X., and Douglas, J. 2006. Computerized adaptive testing under nonparamteric IRT models. Psychometrika 71(1): 121–37.CrossRefGoogle Scholar
Yammarino, F. J., Skinner, S. J., and Childers, T. L. 1991. Understanding mail survey response behavior: A meta-analysis. Public Opinion Quarterly 55(4): 613–639.CrossRefGoogle Scholar
Zaller, J. R. 1992. The nature and origins of mass opinion. New York: Cambridge University Press.CrossRefGoogle Scholar

Montgomery and Cutler supplementary material

Appendix

[Opens in a new window]
PDF 52 KB

Altmetric attention score

Full text views

Full text views reflects PDF downloads, PDFs sent to Google Drive, Dropbox and Kindle and HTML full text views.

Total number of HTML views: 0
Total number of PDF views: 224 *
View data table for this chart

* Views captured on Cambridge Core between 04th January 2017 - 3rd December 2020. This data will be updated every 24 hours.

Access
Hostname: page-component-79f79cbf67-nqzjh Total loading time: 0.545 Render date: 2020-12-03T01:03:47.288Z Query parameters: { "hasAccess": "1", "openAccess": "0", "isLogged": "0", "lang": "en" } Feature Flags last update: Thu Dec 03 2020 00:07:04 GMT+0000 (Coordinated Universal Time) Feature Flags: { "metrics": true, "metricsAbstractViews": false, "peerReview": true, "crossMark": true, "comments": true, "relatedCommentaries": true, "subject": true, "clr": false, "languageSwitch": true }

Send article to Kindle

To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Computerized Adaptive Testing for Public Opinion Surveys
Available formats
×

Send article to Dropbox

To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

Computerized Adaptive Testing for Public Opinion Surveys
Available formats
×

Send article to Google Drive

To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

Computerized Adaptive Testing for Public Opinion Surveys
Available formats
×
×

Reply to: Submit a response

Your details

Conflicting interests

Do you have any conflicting interests? *