Skip to main content

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
Cancel
×
Home
  • Home
Article
    • Aa
    • Aa
  • Cited by 5
  • Cited by
    Crossref Citations
    Crossref logo
    This article has been cited by the following publications. This list is generated based on data provided by CrossRef.

    Loose, Krista Hou, Yue and Berinsky, Adam J. 2017. Achieving Efficiency Without Losing Accuracy: Strategies for Scale Reduction with an Application to Risk Attitudes and Racial Resentment*. Social Science Quarterly,

    Hollibaugh, Gary E. 2017. Presidential Appointments and Public Trust. Presidential Studies Quarterly, Vol. 46, Issue. 3, p. 618.

    Marinova, Dani M. 2017. On the Use of Crowdsourcing Labor Markets in Research. Perspectives on Politics, Vol. 14, Issue. 02, p. 422.

    Motta, Matthew P. Callaghan, Timothy H. and Smith, Brianna 2017. Looking for Answers: Identifying Search Behavior and Improving Knowledge-Based Data Quality in Online Surveys. International Journal of Public Opinion Research, p. edw027.

    Salganik, Matthew J. Levy, Karen E. C. and Helleringer, Stephane 2017. Wiki Surveys: Open and Quantifiable Social Data Collection. PLOS ONE, Vol. 10, Issue. 5, p. e0123483.

    ×
  • Get access
    Add to cart $25.00
    Check if you have access via personal or institutional login
    Log in Register Recommend to librarian

Computerized Adaptive Testing for Public Opinion Surveys

  • Jacob M. Montgomery (a1) and Josh Cutler (a2)
Abstract

Survey researchers avoid using large multi-item scales to measure latent traits due to both the financial costs and the risk of driving up nonresponse rates. Typically, investigators select a subset of available scale items rather than asking the full battery. Reduced batteries, however, can sharply reduce measurement precision and introduce bias. In this article, we present computerized adaptive testing (CAT) as a method for minimizing the number of questions each respondent must answer while preserving measurement accuracy and precision. CAT algorithms respond to individuals' previous answers to select subsequent questions that most efficiently reveal respondents' positions on a latent dimension. We introduce the basic stages of a CAT algorithm and present the details for one approach to item selection appropriate for public opinion research. We then demonstrate the advantages of CAT via simulation and empirically comparing dynamic and static measures of political knowledge.

Copyright
COPYRIGHT: © The Author 2013. Published by Oxford University Press on behalf of the Society for Political Methodology 
Corresponding author
e-mail: jacob.montgomery@wustl.edu (corresponding author)
Footnotes
Hide All

Authors' note: We are grateful for helpful comments provided by Martin Elff, Sunshine Hillygus, Walter Mebane, Brendan Nyhan, and two anonymous reviewers. A previous version of this article was presented at the 2012 Annual Meeting of the Midwest Political Science Association, the 2012 Saint Louis Area Methods Meeting, and the 2012 Summer Methods Meeting. Supplementary materials for this article are available on the Political Analysis Web site.

Footnotes
Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

A. Anderson , A. Basilevsky , and D. Hum 1983. Missing data: A review of the literature. In Handboook of survey research, eds. P. H. Rossi , J. D. Wright , and A. B. Anderson , 415–81. New York: Academic Press.
A. Anderson , A. Basilevsky , and D. Hum 1983. Missing data: A review of the literature. In Handboook of survey research, eds. P. H. Rossi , J. D. Wright , and A. B. Anderson , 415–81. New York: Academic Press.
J. Bafumi , A. Gelman , D. K. Park , and N. Kaplan 2005. Practical issues in implementing and understanding Bayesian ideal point estimation. Political Analysis 13(2): 171–87.
J. Bafumi , and M. C. Herron 2010. Leapfrog representation and extremism: A study of American voters and their members in Congress. American Political Science Review 104(3): 519–42.
M. A. Bailey 2007. Comparable preference estimates across time and institutions for the court, Congress, and presidency. American Journal of Political Science 51(3): 433–48.
J. Barabas 2002. Another look at the measurement of political knowledge. Political Analysis 10(2): 209–22.
A. J. Berinsky , G. A. Huber , and G. S. Lenz 2012. Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk. Political Analysis 20(3): 329–50.
P. R. Brewer 2003. Values, political knowledge, and public opinion about gay rights. Public Opinion Quarterly 67(3): 173–201.
B. Burchell , and C. Marsh 1992. The effect of questionnaire length on survey response. Quality & Quantity 26(3): 233–44.
J. T. Cacioppo , and R. E. Petty 1984. The efficient assessment of need for cognition. Journal of Personality Assessment 48(3): 306–7.
J. T. Cacioppo , and R. E. Petty 1984. The efficient assessment of need for cognition. Journal of Personality Assessment 48(3): 306–7.
S. W. Choi , and R. J. Swartz 2009. Comparison of CAT item selection criteria for polytomous items. Applied Psychological Measurement 33(6): 419–40.
J. D. Clinton , and A. Meirowitz 2001. Agenda constrained legislator ideal points and the spatial voting model. Political Analysis 9(3): 242–59.
J. D. Clinton , and A. Meirowitz 2003. Integrating voting theory and roll call analysis: A framework. Political Analysis 11(4): 381–96.
J. Clinton , S. Jackman , and D. Rivers 2004. The statistical analysis of roll call voting: A unified approach. American Political Science Review 98(2): 355–70.
S. D. Crawford , M. P. Couper , and M. J. Lamias 2001. Web surveys: Perceptions of burden. Social Science Computer Review 19(2): 146–62.
B. G. Dodd , R. De Ayala , and W. R. Koch 1995. Computerized adaptive testing with polytomous items. Applied Psychological Measurement 19(1): 5–22.
S. Feldman , and L. Huddy 2005. Racial resentment and white opposition to race-conscious programs: Principles or prejudice? American Journal of Political Science 49(1): 168–83.
S. Feldman , and L. Huddy 2005. Racial resentment and white opposition to race-conscious programs: Principles or prejudice? American Journal of Political Science 49(1): 168–83.
J. D. Forbey , and Y. S. Ben-Porath 2007. Computerized adaptive personality testing: A review and illustration with the MMPI-2 computerized adaptive version. Psychological Assessment 19(1): 14–24.
M. Galesic , and M. Bosnjak 2009. Effects of questionnaire length on participation and indicators of response quality in Web surveys. Public Opinion Quarterly 73(2): 349–60.
A. S. Gerber , G. A. Huber , D. Doherty , C. M. Dowling , and S. E. Ha 2010. Personality and political attitudes: Relationships across issue domains and political contexts. American Political Science Review 104(01): 111–33.
J. L. Gibson , and G. A. Caldeira 2009. Knowing the Supreme Court? A reconsideration of public ignorance of the high court. Journal of Politics 71(2): 429–41.
B. T. Gomez , and J. M. Wilson 2001. Political sophistication and economic voting in the American electorate: A theory of heterogeneous attribution. American Journal of Political Science 45(4): 899–914.
S. D. Gosling , P. J. Rentfrow , and W. B. Swann 2003. A very brief measure of the big-five personality domains. Journal of Research in Personality 37(6): 504–28.
T. A. Heberlein , and R. Baumgartner 1978. Factors affecting response rates to mailed questionnaires: A quantitative analysis of the published literature. American Sociological Review 43(4): 447–62.
A. R. Herzog , and J. G. Bachman 1981. Effects of questionnaire length on response quality. Public Opinion Quarterly 45(4): 549–59.
A. M. Hol , H. C. Vorst , and G. J. Mellenbergh 2007. Computerized adaptive testing for polytomous motivation items: Administration mode effects and a comparison with short forms. Applied Psychological Measurement 31(5): 412–29.
S. Jackman 2001. Multidimensional analysis of roll call data via Bayesian simulation: Identification, estimation, inference, and model checking. Political Analysis 9(3): 227–41.
J. A. Krosnick 1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology 5(3): 213–36.
J. A. Krosnick 1999. Survey research. Annual Review of Psychology 50: 537–67.
J. A. Krosnick , A. L. Holbrook , M. K. Berent , R. A. B. T. Carson , W. Hanemann , R. J. Kopp , C. Mitchell , R. Cameron , S. Presser , P. A. Ruud , V. Smith , W. R. Moody , M. C. Green , and M. Conaway 2002. The impact of “no opinion“ response options on data quality: Non-attitude reduction or an invitation to satisfice? Public Opinion Quarterly 66(3): 371–403.
A. Lupia 2006. How elitism undermines the study of voter competence. Critical Review 18 (1–3): 217–32.
A. Lupia 2008. Procedural transparency and the credibility of election surveys. Electoral Studies 27(4): 732–9.
R. C. Luskin 1987. Measuring political sophistication. American Journal of Political Science 31(4): 856–99.
R. C. Luskin , and J. G. Bullock 2011. “Don't know” means “don't know”: DK responses and the public's level of political knowledge. Journal of Politics 73(2): 547–57.
A. D. Martin , and K. M. Quinn 2002. Dynamic ideal point estimation via Markov chain Monte Carlo for the US Supreme Court, 1953–1999. Political Analysis 10(2): 134–53.
R. A. Matthews , L. M. Kath , and J. L. Barnes-Farrell 2010. A short, valid, predictive measure of work-family conflict: Item selection and scale validation. Journal of Occupational Health Psychology 15(1): 75–90.
J. J. Mondak , and M. R. Anderson 2004. The knowledge gap: A reexamination of gender-based differences in political knowledge. Journal of Politics 66(2): 492–512.
J. J. Mondak , and B. C. Davis 2001. Asked and answered: Knowledge levels when we will not take “don't know” for an answer. Political Behavior 23(3): 199–224.
J. J. Mondak 2001. Developing valid knowledge scales. American Journal of Political Science 45(1): 224–38.
P. M. Podsakoff , and S. B. MacKenzie 1994. An examination of the psychometric properties and nomological validity of some revised and reduced substitutes for leadership scales. Journal of Applied Psychology 79(5): 702–13.
K. T. Poole 2005. Spatial models of parliamentary voting. New York: Cambridge University Press.
M. Prior , and A. Lupia 2008. Money, time, and political knowledge: Distinguishing quick recall and political learning skills. American Journal of Political Science 52(1): 19–183.
M. L. Richins 2004. The material values scale: Measurement properties and development of a short form. Journal of Consumer Research 31(1): 209–19.
M. L. Richins 2004. The material values scale: Measurement properties and development of a short form. Journal of Consumer Research 31(1): 209–19.
S. S. Russell , C. Spitzmüller , L. F. Lin , J. M. Stanton , P. C. Smith , and G. H. Ironson 2004. Shorter can also be better: The abridged job in general scale. Educational and Psychological Measurement 64(5): 878–93.
D. O. Segall 2005. Computerized adaptive testing. In Encyclopedia of social measurement, ed. K. Kempf-Leonard , Vol. 1, 429–38. Oxford, UK: Elsevier.
P. Sheatsley 1983. Questionnaire construction and item writing. In Handboook of survey research, eds. P. H. Rossi , J. D. Wright , and A. B. Anderson , 195–230. New York: Academic Press.
P. M. Sniderman , R. A. Brody , and P. E. Tetlock 1991. Reasoning and choice: Explorations in political psychology. New York: Cambridge University Press.
P. M. Sniderman , T. Piazza , P. E. Tetlock , and A. Kendrick 1991. The new racism. American Journal of Political Science 35(2): 423–47.
J. M. Stanton , E. F. Sinar , W. K. Balzer , and P. C. Smith 2002. Issues and strategies for reducing the length of self-report scales. Personnel Psychology 55(1): 167–94.
E. R. Thompson 2012. A brief index of affective job satisfaction. Group & Organization Management 37(3): 275–307.
S. Treier , and D. S. Hillygus 2009. The nature of political ideology in the contemporary electorate. Public Opinion Quarterly 73(4): 679–703.
S. Treier , and S. Jackman 2008. Democracy as a latent variable. American Journal of Political Science 52(1): 201–17.
W. J. van der Linden 1998. Bayesian item selection criteria for adaptive testing. Psychometrika 63(2): 201–16.
W. J. van der Linden 1999. Empirical initialization of the trait estimator in adaptive testing. Applied Psychological Measurement 23(1): 21–29.
W. J. van der Linden 2008. Using response times for item selection in adaptive testing. Journal of Educational and Behavioral Statistics 33(1): 5–20.
W. J. van der Linden , and P. J. Pashley 2010. Elements of adaptive testing. New York: Springer.
N. G. Waller , and S. P. Reise 1989. Computerized adaptive personality assessment: An illustration with the absorption scale. Journal of Personality and Social Psychology 57(6): 1051.
D. J. Weiss 1982. Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement 6(4): 473–92.
D. J. Weiss 1982. Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement 6(4): 473–92.
D. J. Weiss , and G. G. Kingsbury 1984. Application of computerized adaptive testing to educational problems. Journal of Educational Measurement 21(4): 361–75.
X. Xu , and J. Douglas 2006. Computerized adaptive testing under nonparamteric IRT models. Psychometrika 71(1): 121–37.
F. J. Yammarino , S. J. Skinner , and T. L. Childers 1991. Understanding mail survey response behavior: A meta-analysis. Public Opinion Quarterly 55(4): 613–639.
J. R. Zaller 1992. The nature and origins of mass opinion. New York: Cambridge University Press.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *

CAPTCHA *

Skip to the audio challenge
×
MathJax
Type Description Title
PDF
Supplementary Materials

Montgomery and Cutler supplementary material
Appendix

 PDF (52 KB)
52 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 9 *
Loading metrics...

Abstract views

Total abstract views: 54 *
Loading metrics...

* Views captured on Cambridge Core between 4th January 2017 - 19th September 2017. This data will be updated every 24 hours.

Cancel
Confirm
×