Skip to main content Accessibility help

Online coders, open codebooks: New opportunities for content analysis of political communication

  • Nicholas J. G. Winter (a1), Adam G. Hughes (a2) and Lynn M. Sanders (a1)

Analyzing audiovisual communication is challenging because its content is highly symbolic and less rule-governed than verbal material. But audiovisual messages are important to understand: they amplify, enrich, and complicate the meaning of textual information. We describe a fully-reproducible approach to analyzing video content using minimally—but systematically—trained online workers. By aggregating the work of multiple coders, we achieve reliability, validity, and costs that equal those of traditional, intensively trained research assistants, with much greater speed, transparency, and replicability. We argue that measurement strategies relying on the “wisdom of the crowd” provide unique advantages for researchers analyzing complex and intricate audiovisual political content.

Corresponding author
*Corresponding author. Email:
Hide All
Armstrong, SL, Gleitman, LR and Gleitman, H (1999) What some concepts might not be. In Margolis, E and Laurence, S (eds). Concepts: Core Readings. Cambridge, MA: MIT Press, pp. 225–59.
Benoit, WL (2010) Content analysis in political communication. In Bucy, EP and Holbert, RL (eds). Sourcebook for Political Communication Research: Methods, Measures, and Analytic Techniques. New York: Taylor & Francis, pp. 268–79.
Benoit, K, Laver, M and Mikhaylov, S (2009) Treating words as data with error: uncertainty in text statements of policy positions. American Journal of Political Science 53, 495513.
Benoit, K, Conway, D, Lauderdale, BE, Laver, M and Mikhaylov, S (2016) Crowd-sourced text analysis: reproducible and agile production of political data. American Political Science Review 110, 278–95.
Bonica, A (2016) Database on Ideology, Money in Politics, and Elections: Public Version 2.0. [Computer file]. Stanford, CA: Stanford University Libraries. Available at
Brader, T (ed) (2006) Campaigning for Hearts and Minds: How Emotional Appeals in Political Ads Work. Chicago: University of Chicago Press.
Budak, C, Goel, S and Rao, JM (2016) Fair and balanced? Quantifying media bias through crowdsourced content analysis. Public Opinion Quarterly 80(S1), 250–71.
Carmines, EG and Zeller, RA (1979) Reliability and Validity Assessment. Thousand Oaks, CA: Sage.
Cohen, J (1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20, 3746.
Fowler, EF, Franz, MM and Ridout, TN (2014) Political Advertising in 2010. [Computer file]. Version 1.3. Middletown, CT: The Wesleyan Media Project; Department of Government at Wesleyan University.
Frijda, NH (1988) The laws of emotion. American Psychologist 43, 349–58.
Geer, JG (2006) In Defense of Negativity: Attack Ads in Presidential Campaigns. Chicago: University of Chicago Press.
Gelman, SA and Wellman, HM (1999) Insides and essences: early understandings of the non-obvious. In Eric, M and Laurence, S (eds). Concepts: Core Readings. Cambridge, MA: MIT Press, pp. 613–37.
Grabe, ME and Bucy, EP (2009) Image Bite Politics: News and the Visual Framing of Elections. New York: Oxford University Press.
Graber, DA (1987) Television news without pictures? Critical Studies in Mass Communication 4, 74–8.
Graber, DA and Smith, JM (2005) Political communication faces the 21st century. Journal of Communication 55, 479507.
Grimmer, J and Stewart, BM (2013) Text as data: the promise and pitfalls of automatic content analysis methods for political texts. Political Analysis 21, 267–97.
Gwet, KL (2014) Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement among Multiple Raters. Gaithersburg, MD: Advanced Analytics, LLC.
Hayes, D (2011) When gender and party collide: stereotyping in candidate trait attribution. Politics & Gender 7, 133–65.
Khatchadourian, H (1966) Common names and “Family Resemblances”. In Wittgenstein: The Philosophical Investigations, George, P (ed). Garden City, NY: Anchor Books, pp. 203–30.
Kinder, DR and Kiewiet., DR (1981) Sociotropic politics: the American case. British Journal of Political Science 11, 129–61.
Kinder, DR, Peters, MD, Abelson, RP and Fiske, ST (1980) Presidential prototypes. Political Behavior 2, 315–37.
Klein, D (2018) Implementing a general framework for assessing interrater agreement in stata. The Stata Journal 18, 871901.
Krippendorff, K (1970) Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement 30, 6170.
Lakoff, G (1987) Women, Fire, and Dangerous Things: What Categories Reveal About the Mind. Chicago: University of Chicago Press.
Lind, F, Gruber, M and Boomgaarden, HG (2017) Content analysis by the crowd: assessing the usability of crowdsourcing for coding latent constructs. Communication Methods and Measures 11, 191209.
Mann, R (2011) Daisy Petals and Mushroom Clouds: LBJ, Barry Goldwater, and the Ad That Changed American Politics. Baton Rouge: Louisiana State University Press.
Masters, RD and Sullivan, DG (1993) Nonverbal behavior and leadership: emotion and cognition in political information processing. In Shanto, I and McGuire, WJ (eds). Explorations in Political Psychology. Durham: Duke University Press, pp. 150–82.
Mendelberg, T (2001) The Race Card: Campaign Strategy, Implicit Messages, and the Norm of Equality. Princeton, NJ: Princeton University Press.
Messaris, P (1997) Visual Persuasion: The Role of Images in Advertising. Thousand Oaks, CA: Sage Publications.
Messaris, P and Abraham, L (2001) The role of images in framing news stories. In Stephen, DR, Gandy, OH and Grant, AE (eds). Framing Public Life: Perspectives on Media and Our Understanding of the Social World. Mahwah, NJ: Lawrence Erlbaum Associates, pp. 215–26.
Poole, KT and Rosenthal, H (2007) Ideology & Congress. 2nd rev. edn. New Brunswick: Transaction Publishers.
Potter, WJ and Levine-Donnerstein, D (1999) Rethinking validity and reliability in content analysis. Journal of Applied Communication Research 27, 258–84.
Reynolds, TJ and Whitlark, DB (1995) Applying laddering data to communications strategy and advertising practice. Journal of Advertising Research 35, 917.
Rosenberg, SW, Bohan, L, McCafferty, P and Harris., K (1986) The image and the vote: the effect of candidate presentation on voter preference. American Journal of Political Science 30, 108–27.
Schill, D (2012) The visual image and the political image: a review of visual communication research in the field of political communication. Review of Communication 12, 118–42.
Spielvogel, C (2005) You know where I stand: moral framing of the war on terrorism and the Iraq war in the 2004 presidential campaign. Rhetoric & Public Affairs 8, 549–69.
Strach, P, Zuber, K, Fowler, EF, Ridout, TN and Searles, K (2015) In a different voice? Explaining the use of men and women as voice-over announcers in political advertising. Political Communication 32, 183205.
Vakharia, D and Lease, M (2015) Beyond AMT: an analysis of paid crowd work platforms. iConference 2015 Proceedings. Available at
Weber, R, Mangus, JM, Huskey, R, Hopp, FR, Amir, O, et al. (2018) Extracting latent moral information from text narratives: relevance, challenges, and solutions. Communication Methods and Measures 12, 119–39.
Wittgenstein, L (1953) Philosophical Investigations. Oxford: B. Blackwell.
Woollacott, J (1982) Messages and meanings. In Tony, B, Curran, J, Gurevitch, M and Wollacott, J (eds). Culture, Society and the Media. New York: Routledge, pp. 87109.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Science Research and Methods
  • ISSN: 2049-8470
  • EISSN: 2049-8489
  • URL: /core/journals/political-science-research-and-methods
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Type Description Title
Supplementary materials

Winter et al. Dataset

Supplementary materials

Winter et al. supplementary material
Online Appendix

 PDF (6.7 MB)
6.7 MB


Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed