Hostname: page-component-8448b6f56d-qsmjn Total loading time: 0 Render date: 2024-04-24T00:05:52.775Z Has data issue: false hasContentIssue false
  • English
  • Français

Harder Than It Looks: Coding Political Knowledge on the ANES

Published online by Cambridge University Press:  04 January 2017

Matthew DeBell
Matthew DeBell*
Affiliation:
Institute for Research in the Social Sciences, Stanford University, 30 Alta Road, Stanford, CA 94305 e-mail: debell@stanford.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

Political knowledge research faces a problem, perhaps even a crisis. For two decades, the American National Election Studies asked open-ended questions about political knowledge and coded answers using procedures that are neither reliable nor replicable and that were never shown to be optimally valid. Consequently, conclusions based on these widely used measures of the public's competence are in doubt. This article presents several new and overdue methodological improvements: coding knowledge data using formal and specific coding rules based on a substantive rationale for the validity of the codes, recognizing partially correct answers, using multiple coders working independently, using machine coding, and testing reliability and validity. The new methods are an improvement because they are transparent and replicable and they produce valid and extremely reliable knowledge data. Further, machine coding produces codes nearly identical to those from a team of human coders, at much lower cost.

Type
Research Article
Information
Political Analysis , Volume 21 , Issue 4 , Autumn 2013 , pp. 393 - 406
Copyright
Copyright © The Author 2013. Published by Oxford University Press on behalf of the Society for Political Methodology 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Abramowitz, Alan. 1995. It's abortion, stupid: Policy voting in the 1992 presidential election. Journal of Politics 57: 176–86.CrossRefGoogle Scholar
Barabas, Jason. 2002. Another look at the measurement of political knowledge. Political Analysis 10: 209–22.CrossRefGoogle Scholar
Brewer, Paul R. 2003. Values, political knowledge, and public opinion about gay rights. Public Opinion Quarterly 63: 173–201.CrossRefGoogle Scholar
Carmines, Edward G., and Stimson, James A. 1980. The two faces of issue voting. American Political Science Review 74: 78–91.Google Scholar
Cohen, Jacob. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20: 37–46.Google Scholar
DeBell, Matthew. 2010. How to analyze ANES survey data. ANES Technical Report Series, no. nes012492. http://electionstudies.org/resources/papers/nes012492.pdf (accessed September 28, 2010).Google Scholar
DeBell, Matthew. 2013. Replication data for validation: Harder than it looks: Coding political knowledge on the ANES [dataset], http://hdl.handle.net/1901.1/21558. IQSS Dataverse Network [Distributor] V1 [Version].Google Scholar
Carpini, Delli, Michael, X., and Keeter, Scott. 1996. What Americans know about politics and why it matters. New Haven, CT: Yale University Press.Google Scholar
Downs, Anthony. 1957. An economic theory of democracy. New York: Harper.Google Scholar
Esuli, Andrea, and Sebastiani, Fabrizio. 2010. Machines that learn how to code open-ended survey data. International Journal of Market Research 52: 775–800.Google Scholar
Fan, David P. 2008. Computer coding of 1992 ANES like/dislike and MIP responses. Presented at conference on Optimal Coding of Open-Ended Survey Data, Ann Arbor, MI.Google Scholar
Fleiss, Joseph L., Levin, Bruce, and Palik, Myunghee Cho. 2003. Statistical methods for rates and proportions. 3rd ed. New York: Wiley.Google Scholar
Gibson, James L., and Caldeira, Gregory A. 2009. Knowing the Supreme Court? A re-consideration of public ignorance of the high court. Journal of Politics 71: 429–41.CrossRefGoogle Scholar
Goren, Paul. 1997. Political expertise and issue voting in presidential elections. Political Research Quarterly 50: 387–412.Google Scholar
Karp, Jeffrey A. 1995. Explaining support for legislative term limits. Public Opinion Quarterly 59: 373–91.CrossRefGoogle Scholar
Krippendorff, Klaus. 2004. Content analysis: An introduction to its methodology. Thousand Oaks, CA: Sage.Google Scholar
Krosnick, Jon A., Lupia, Arthur, DeBell, Matthew, and Donakowski, Darrell. 2008. Problems with ANES questions measuring political knowledge. ANES memorandum. http://www.electionstudies.org/announce/newsltr/20080324PoliticalKnowledgeMemo.pdf (accessed March 24, 2008).Google Scholar
Ladd, Jonathan McDonald. 2007. Predispositions and public support for the president during the war on terrorism. Public Opinion Quarterly 71: 511–38.CrossRefGoogle Scholar
Lupia, Arthur. 2006. How elitism undermines the study of voter competence. Critical Review 18: 217–32.CrossRefGoogle Scholar
Lupia, Arthur. 2008. Procedural transparency and the credibility of election surveys. Electoral Studies 27: 732–39.CrossRefGoogle Scholar
Lupia, Arthur, Krosnick, Jon A., Luevano, Pat, DeBell, Matthew, and Donakowski, Darrell. 2009. User's guide to the ANES 2008 time series study. http://electionstudies.org/studypages/2008prepost/2008prepost_UsersGuide.pdf (accessed May 11, 2009).Google Scholar
Luskin, Robert C., and Bullock, John G. 2011. “Don't know” means “don't know”: DK responses and the public's level of political knowledge. Journal of Politics 73: 547–57.Google Scholar
Martinez, Michael D., and Craig, Stephen C. 2010. Dick Cheney? Didn't he shoot somebody? Measuring political knowledge in the 2008 American National Election Study. Presented at the annual meeting of the American Political Science Association, Washington, DC.Google Scholar
Marquis, Lionel. 2010. Understanding political knowledge and its influence on voting preferences in the 2007 federal election. Swiss Political Science Review 16: 425–56.Google Scholar
Mondak, Jeffery J. 2000. Reconsidering the measurement of political knowledge. Political Analysis 8: 57–82.Google Scholar
Mondak, Jeffery J. 2001. Developing valid knowledge scales. American Journal of Political Science 45: 224–38.Google Scholar
Mondak, Jeffery J., and Davis, Belinda Creel. 2001. Asked and answered: Knowledge levels when we will not take “don't know” for an answer. Political Behavior 23: 199–224.CrossRefGoogle Scholar
Nicholson, Stephen P., Pantoja, Adrian, and Segura, Gary M. 2006. Political knowledge and issue voting among the Latino electorate. Political Research Quarterly 59: 259–71.Google Scholar
Skalski, Paul. 2008. CATA options for the coding of open-ended survey data. Presented at conference on Optimal Coding of Open-Ended Survey Data, Ann Arbor, MI.Google Scholar
Subhawong, Aryn. 2008. A realistic look at the vice presidency: Why Dick Cheney is an “entity within the executive branch.” Saint Louis University Law Journal 53: 281–308.Google Scholar
Supplementary material: File

DeBell supplementary material

Appendices

Download DeBell supplementary material(File)
File 103.9 KB