Skip to main content

An Analysis of ANES Items and Their Use in the Construction of Political Knowledge Scales

  • Matthew T. Pietryka (a1) and Randall C. MacIntosh (a2)

Valid comparisons of group scores on additive measures such as political knowledge scales require that the conditional response probabilities for individuals on the observed items be invariant across groups after controlling for their overall level of the latent trait of interest. Using a multi-group confirmatory factor analysis of knowledge items drawn from American National Election Studies, we find that the scales used in recent research are not sufficiently invariant for valid comparisons across a host of theoretically important grouping variables. We demonstrate that it is possible to construct valid invariant scales using a subset of items and show the impact of invariance by comparing results from the valid and invalid scales. We provide an analysis of differential item functioning based on grouping variables commonly used in political science research to explore the utility of each item in the construction of valid knowledge scales. An application of the VTT suggests it is more appropriate to conceive of these items as effects of a latent variable rather than cause or formative indicators. These results suggest that models attempting to explain apparent knowledge gaps between subgroups have been unsuccessful because previously constructed scales were validated by fiat.

    • Send article to Kindle

      To send this article to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the or variations. ‘’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      An Analysis of ANES Items and Their Use in the Construction of Political Knowledge Scales
      Available formats
      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      An Analysis of ANES Items and Their Use in the Construction of Political Knowledge Scales
      Available formats
      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      An Analysis of ANES Items and Their Use in the Construction of Political Knowledge Scales
      Available formats
Corresponding author
e-mail: (corresponding author)
Hide All

Authors' note: We thank Debra Leiter and Ron Rapoport for helpful comments and Jay Dow for sharing replication materials. Supplementary materials for this article are available on the Political Analysis Web site (Pietryka and Macintosh 2013).

Hide All
Ackerman, Terry A. 1992. A didactic explanation of item bias, item impact, and item validity from a multidimensional perspective. Journal of Educational Measurement 29(1): 6791.
Ahn, T. K., Huckfeldt, Robert, and Ryan, John Barry. 2010. Communication, influence, and informational asymmetries among voters. Political Psychology 31: 763–87.
Alba, Joseph W., and Hasher, Lynn. 1983. Is memory schematic? Psychological Bulletin 93: 203–31.
Alvarez, R. Michael, and Nagler, Jonathan. 2004. Party system compactness: Measurement and consequences. Political Analysis 12(1): 4662.
American National Election Studies ( 2010. Time Series Cumulative Data File [dataset]. Stanford University and the University of Michigan [producers and distributors].
Arbuckle, James L. 2009. Amos 18 User's Guide [Computer software]. Crawfordville, FL: Amos Development Corporation.
Asparouhov, Tihomir, and Muthen, Bengt. 2006. Robust chi square difference testing with mean and variance adjusted test statistics (Web Notes: No. 10, May 26, 2006). (accessed July 12, 2010).
Axelrod, Robert. 1973. Schema theory: An information processing model of perception and cognition. American Political Science Review 67(4): 1248–66.
Bagozzi, Richard P. 2011. Measurement and meaning in information systems and organizational research: Methodological and philosophical foundations. MIS Quarterly 35(2): 261–92.
Bartels, Larry M. 1996. Uninformed votes: Information effects in presidential elections. American Journal of Political Science 40(1): 194230.
Blalock, Hubert M. 1964. Causal inferences in nonexperimental research. Chapel Hill: University of North Carolina Press. (accessed January 16, 2013).
Bollen, Kenneth A. 2011. Evaluating effect, composite and causal indicators in structural equation models. MIS Quarterly 35(2): 359–72.
Bollen, Kenneth A., and Ting, Kwok-Fai. 1993. Confirmatory tetrad analysis. Sociological Methodology 23: 147–75.
Bollen, Kenneth A., and Ting, Kwok-Fai. 2000. A tetrad test for causal indicators. Psychological Methods 5(1): 322.
Bollen, Kenneth, and Lennox, Richard. 1991. Conventional wisdom on measurement: A structural equation perspective. Psychological Bulletin 110(2): 305–14.
Bollen, Kenneth A., and Bauldry, Shawn. 2011. Three cs in measurement models: Causal indicators, composite indicators and covariates. Psychological Methods 16(3): 265–84.
Borsboom, Denny. 2006. The attack of the psychometricians. Psychometrika 71(3): 425–40.
Brown, Timothy. 2006. Confirmatory factor analysis for applied research. New York: Gilford Press.
Byrne, Barbara, Shavelson, Richard, and Muthen, Bengt. 1989. Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin 105: 456–66.
Clinton, Joshua, Jackman, Simon, and Rivers, Douglas. 2004. The statistical analysis of roll call data. American Political Science Review 98(02): 355–70.
Conover, Pamela Johnston, and Feldman, Stanley. 1984. How people organize the political world: A schematic model. American Journal of Political Science 28(1): 95126.
Davidov, Eldad. 2009. Measurement equivalence of nationalism and constructive patriotism in the ISSP: 34 countries in a comparative perspective. Political Analysis 17(1): 6482.
Davis, Darren, and Silver, Brian D. 2003. Stereotype threat and race of interviewer effects in a survey on political knowledge. American Journal of Political Science 47(1): 3345.
Carpini, Delli, Michael, X., and Keeter, Scott. 1993. Measuring political knowledge: Putting first things first. American Journal of Political Science 37(4): 1179–206.
Carpini, Delli, Michael, X., and Keeter, Scott. 1996. What Americans know about politics and why it matters. New Haven, CT: Yale University Press.
Carpini, Delli, Michael, X., and Keeter, Scott. 2000. Gender and political knowledge. In Gender and American politics: Women, men and the political process, eds. Tolleson-Rinehart, Sue and Josephson, Jyl. J., 2152. Armonk, NY: M.E. Sharpe.
Diamantopoulos, Adamantios, and Winklhofer, Heidi M. 2001. Index construction with formative indicators: An alternative to scale development. Journal of Marketing Research 38(2): 269–77.
Dolan, Kathleen. 2011. Do women and men know different things? Measuring gender differences in political knowledge. Journal of Politics 73(01): 97107.
Dow, Jay K. 2009. Gender differences in political knowledge: Distinguishing characteristics-based and returns-based differences. Political Behavior 31(1): 117–36.
Downs, Anthony. 1957. An economic theory of democracy. New York: Harper and Row.
Edwards, Jeffrey R. 2001. Multidimensional constructs in organizational behavior research: An integrative analytical framework. Organizational Research Methods 4(14): 144–92.
Edwards, Jeffrey R. 2011. The fallacy of formative measurement. Organizational Research Methods 14(2): 370–88.
Edwards, Jeffrey R., and Bagozzi, Richard P. 2000. On the nature and direction of relationships between constructs and measures. Psychological Methods 5(2): 155–74.
Fornell, Claes, and Bookstein, Fred. 1982. Two structural equation models: LISREL and PLS applied to consumer exit-voice theory. Journal of Marketing Research 19(4): 440–52.
Franz, Michael M., Freedman, Paul B., Goldstein, Kenneth M., and Ridout, Travis N. 2007. Campaign Advertising and American Democracy. Philadelphia: Temple University Press.
Frazer, Elizabeth, and Macdonald, Kenneth. 2003. Sex differences in political knowledge in Britain. Political Studies 51(1): 6783.
Freedman, Paul B., Franz, Michael M., and Goldstein, Kenneth M. 2004. Campaign advertising and democratic citizenship. American Journal of Political Science 48(4): 723–41.
Galston, William A. 2001. Political knowledge, political engagement, and civic education. Annual Review of Political Science 4(1): 217–34.
Gibson, James L., and Caldeira, Gregory A. 2009. Knowing the Supreme Court? A reconsideration of public ignorance of the High Court. Journal of Politics 71(02): 429–41.
Hauser, Robert M., and Goldberger, Arthur S. 1971. The treatment of unobservable variables in path analysis. Sociological Methodology 3: 81117.
Highton, Benjamin. 2009. Revisiting the relationship between educational attainment and political sophistication. Journal of Politics 71(04): 1564–76.
Hipp, John R., Bauer, Daniel J., and Bollen, Kenneth A. 2005. Conducting tetrad tests of model fit and contrasts of tetrad-nested models: A new SAS macro. Structural Equation Modeling: A Multidisciplinary Journal 12(1): 7693.
Hipp, John R., and Bollen, Kenneth A. 2003. Model fit in structural equation models with censored, ordinal, and dichotomous variables: Testing vanishing tetrads. Sociological Methodology 33(1): 267305.
Horn, John, and McArdle, John. 1992. A practical and theoretical guide to measurement invariance in aging research. Experimental Aging Research 18: 117–44.
Huckfeldt, Robert. 2001. The social communication of political expertise. American Journal of Political Science 45(2): 425–38.
Huckfeldt, R., and Sprague, J. 1995. Citizens, politics, and social communication: Information and influence in an election campaign. New York: Cambridge University Press.
Iacobucci, Dawn. 2010. Structural equations modeling: Fit indices, sample size, and advanced topics. Journal of Consumer Psychology 20(1): 90.
Jackman, Simon. 2008. Measurement. In The Oxford handbook of political methodology, eds. Box-Steffensmeier, Janet M., Brady, Henry E., and Collier, David, 119–51. New York: Oxford University Press.
Jöreskog, Karl, and Sörbom, Dag. 2006. LISREL 8.8 for Windows [Computer software]. Lincolnwood, IL: Scientific Software International, Inc.
Kolen, Michael J., and Brennan, Robert L. 2004. Test equating, scaling and linking: Methods and practices, 2nd ed. New York: Springer.
Krosnick, Jon A., Lupia, Arthur, DeBell, Matthew, and Donakowski, Darrell. 2008. Problems with ANES questions measuring political knowledge. (accessed June 22, 2010).
Lambert, Ronald D., Curtis, James E., Kay, Barry J., and Brown, Steven D. 1988. The social sources of political knowledge. Canadian Journal of Political Science 21(2): 359–74.
Lau, Richard R., and Redlawsk, David P. 2001. Advantages and disadvantages of cognitive heuristics in political decision making. American Journal of Political Science 45(4): 951–71.
Lau, Richard R., and Redlawsk, David P. 2008. Older but wiser? Effects of age on political cognition. Journal of Politics 70(1): 168185.
Lizotte, Mary-Kate, and Sidman, Andrew H. 2009. Explaining the gender gap in political knowledge. Politics & Gender 5(2): 127–51.
Lodge, Milton, and Taber, Charles S. 2000. Three steps toward a theory of motivated political reasoning. In Elements of reason: Cognition, choice, and the bounds of rationality, eds. Lupia, Arthur, McCubbins, Matthew D., and Popkin, Samuel L., 183213. New York, NY: Cambridge University Press.
Lodge, Milton, McGraw, Kathleen, and Stroh, Patrick. 1989. An impression-driven model of candidate evaluation. American Political Science Review 83(2): 399419.
Lupia, Arthur. 1994. Shortcuts versus encyclopedias: Information and voting behavior in California insurance reform elections. American Political Science Review 88(1): 6376.
Luskin, Robert C. 1987. Measuring political sophistication. American Journal of Political Science 31(4): 856–99.
Luskin, Robert C. 1990. Explaining political sophistication. Political Behavior 12(4): 331–61.
MacCallum, Robert C., and Browne, Michael W. 1993. The use of causal indicators in covariance structure models: Some practical issues. Psychological Bulletin 114(3): 533–41.
Macdonald, Stuart Elaine, Rabinowitz, George, and Listhaug, Ola. 1995. Political sophistication and models of issue voting. British Journal of Political Science 25(4): 453–83.
MacKenzie, Scott B., Podsakoff, Philip M., and Jarvis, Cheryl Burke. 2005. The problem of measurement model misspecification in behavioral and organizational research and some recommended solutions. Journal of Applied Psychology 90(4): 710–30.
McGlone, Matthew S., Aronson, Joshua, and Kobrynowicz, Diane. 2006. Stereotype threat and the gender gap in political knowledge. Psychology of Women Quarterly 30(4): 392–98.
Meredith, William. 1993. Measurement invariance, factor analysis and factorial invariance. Psychometrika 58: 525–43.
Miller, Warren E., Kinder, Donald R., Rosenstone, Steven J., and the National Election Studies. American national election studies, 1992 time series study [dataset]. Ann Arbor, MI: University of Michigan, Center for Political Studies [producer and distributor], 1999.
Millsap, Roger, and Yun-Tein, Jenn. 2004. Assessing factorial invariance in ordered-categorical measures. Multivariate Behavioral Research 39: 479515.
Mondak, Jeffery J. 1999. Reconsidering the measurement of political knowledge. Political Analysis 8(1): 5782.
Mondak, Jeffery J. 2001. Developing valid knowledge scales. American Journal of Political Science 45(1): 224–38.
Mondak, Jeffery J., and Anderson, Mary R. 2004. The knowledge gap: A reexamination of gender-based differences in political knowledge. Journal of Politics 66(2): 492512.
Morehouse Mendez, Jeanette, and Tracy Osborn. 2010. Gender and the perception of knowledge in political discussion. Political Research Quarterly 63(2): 270–80.
Muthen, Bengt, and Christoffersson, Anders. 1981. Simultaneous factor analysis of dichotomous variables in several groups. Psychometrika 46: 407–19.
Muthen, Bengt, and Asparouhov, Tihomir. 2002. Latent variable analysis with categorical outcomes: Multiple-group and growth modeling in Mplus (Mplus Web Notes: No. 4, Version 5, December 9, 2002). (accessed May 6, 2010).
Muthen, Linda, and Muthen, Bengt. 2010. Mplus User's Guide. 6th ed. (1998–2010). Los Angeles, CA: Muthen & Muthen.
Nunnally Jum, C., Ira, H., and Bernstein, . 1994. Psychometric theory. 3rd ed. NewYork: McGraw-Hill.
Palfrey, Thomas R., and Poole, Keith T. 1987. The relationship between information, ideology, and voting behavior. American Journal of Political Science 31(3): 511–30.
Pantoja, Adrian D., and Segura, Gary M. 2003. Fear and loathing in California: Contextual threat and political sophistication among Latino voters. Political Behavior 25(3): 265–86.
Pietryka, Matthew T., and MacIntosh, Randall C. 2013. Replication data for: An analysis of ANES items and their use in the construction of political knowledge scales. Dataverse Network [Distributor] V1 [Version].
Popkin, Samuel L., and Dimock, Michael A. 1999. Political knowledge and citizen competence. In Citizen competence and democratic institutions, eds. Elkin, Stephen L. and Edward Sołtan, Karol, 117–46. University Park, PA: Penn State Press.
Prior, Markus. 2005. News vs. entertainment: How increasing media choice widens gaps in political knowledge and turnout. American Journal of Political Science 49(3): 577–92.
Prior, Markus, and Lupia, Arthur. 2008. Money, time, and political knowledge: Distinguishing quick recall and political learning skills. American Journal of Political Science 52(1): 169–83.
Rapoport, Ronald B. 1979. What they don't know can hurt you. American Journal of Political Science 23(4): 805–15.
Satorra, Albert. 2000. Scaled and adjusted restricted tests in multi-sample analysis of moment structures. In Innovations in multivariate statistical analysis: A Festschrift for Heinz Neudecker, eds. Heijmans, Risto D. H., Pollock, D. S. G., and Satorra, Albert, 233–47. New York: Springer. (accessed July 14, 2010).
Satorra, Albert, and Bentler, Peter. 1999. A scaled difference chi-square test statistic for moment structure analysis. Technical Report, University of California, Los Angeles. (accessed September 15, 2011).
Seeman, Melvin. 1966. Alienation, membership, and political knowledge: A comparative study. Public Opinion Quarterly 30(3): 353–67.
Shadish, W., Cook, T., and Campbell, D. 2002. Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.
Smiley, M. 1999. Democratic citizenship: A question of competence. In Citizen competence and democratic institutions, eds. Elkin, Stephen L. and Edward Soltan, Karol, 371–83. University Park, PA: Pennsylvania State University Press.
Stegmueller, Daniel. 2011. Apples and oranges? The problem of equivalence in comparative research. Political Analysis 19(4): 471–87.
Stolle, Dietlind, and Gidengil, Elisabeth. 2010. What do women really know? A gendered analysis of varieties of political knowledge. Perspectives on Politics 8(1): 93.
Taber, Charles S., and Lodge, Milton. 2006. Motivated skepticism in the evaluation of political beliefs. American Journal of Political Science 50(3): 755–69.
The American National Election Studies ( The 1996 time series study [dataset]. Ann Arbor, MI: University of Michigan, Center for Political Studies [producer and distributor].
The American National Election Studies ( The 2000 time series study [dataset]. Ann Arbor, MI: University of Michigan, Center for Political Studies [producer and distributor].
The American National Election Studies ( Time series cumulative data file [dataset]. Stanford University and the University of Michigan [producers and distributors], 2010.
The National Election Studies ( The anes 2004 time series study [dataset]. Ann Arbor, MI: University of Michigan, Center for Political Studies [producer and distributor].
Vandenberg, Robert, and Lance, Charles. 2000. A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods 2: 469.
Verba, Sidney, Schlozman, Kay Lehman, and Brady, Henry E. 1995. Voice and equality: Civic voluntarism in American politics. Cambridge, MA: Harvard University Press.
Verba, Sidney, Burns, Nancy, and Schlozman, Kay Lehman. 1997. Knowing and caring about politics: Gender and political engagement. Journal of Politics 59(4): 1051–72.
Widman, Keith, and Reise, Steven. 1997. Exploring the measurement invariance of psychological instruments: Applications in the substance use domain. In The science of prevention: Methodological advances from alcohol and substance abuse research, eds. Bryant, K. J., Windle, M., and West, S. G., 281324. Washington, DC: American Psychological Association.
Young, Dannagal Goldthwaite. 2004. Late-night comedy in election 2000: Its influence on candidate trait ratings and the moderating effects of political knowledge and partisanship. Journal of Broadcasting & Electronic Media 48(1): 122.
Yum, June O., and Kendall, Kathleen E. 1988. Sources of political information in a presidential primary campaign. Journalism Quarterly 65: 148–51.
Zaller, John R. 1992. The nature and origins of mass opinion. New York, NY: Cambridge University Press.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
Type Description Title
Supplementary materials

Pietryka and MacIntosh supplementary material

 PDF (488 KB)
488 KB