Hostname: page-component-77f85d65b8-g98kq Total loading time: 0 Render date: 2026-04-21T13:19:06.082Z Has data issue: false hasContentIssue false
  • English
  • Français

Online coders, open codebooks: New opportunities for content analysis of political communication

Published online by Cambridge University Press:  27 March 2019

Nicholas J. G. Winter Open the ORCID record for Nicholas J. G. Winter [Opens in a new window] ,
Adam G. Hughes  and
Lynn M. Sanders
Nicholas J. G. Winter*
Affiliation:
Department of Politics, University of Virginia, S185 Gibson Hall, 1540 Jefferson Park Ave, Charlottesville, VA
Adam G. Hughes
Affiliation:
Pew Research Center, Washington, D.C.
Lynn M. Sanders
Affiliation:
Department of Politics, University of Virginia, S185 Gibson Hall, 1540 Jefferson Park Ave, Charlottesville, VA
*
*Corresponding author. Email: nwinter@virginia.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

Analyzing audiovisual communication is challenging because its content is highly symbolic and less rule-governed than verbal material. But audiovisual messages are important to understand: they amplify, enrich, and complicate the meaning of textual information. We describe a fully-reproducible approach to analyzing video content using minimally—but systematically—trained online workers. By aggregating the work of multiple coders, we achieve reliability, validity, and costs that equal those of traditional, intensively trained research assistants, with much greater speed, transparency, and replicability. We argue that measurement strategies relying on the “wisdom of the crowd” provide unique advantages for researchers analyzing complex and intricate audiovisual political content.

Keywords

automated content analysiscrowdsourcingpolitical communicationtext and content analysis

Information

Type
Original Article
Information
Political Science Research and Methods , Volume 8 , Issue 4 , October 2020 , pp. 731 - 746
Copyright
Copyright © The European Political Science Association 2019

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Armstrong, SL, Gleitman, LR and Gleitman, H (1999) What some concepts might not be. In Margolis, E and Laurence, S (eds). Concepts: Core Readings. Cambridge, MA: MIT Press, pp. 225–59.Google Scholar
Benoit, WL (2010) Content analysis in political communication. In Bucy, EP and Holbert, RL (eds). Sourcebook for Political Communication Research: Methods, Measures, and Analytic Techniques. New York: Taylor & Francis, pp. 268–79.Google Scholar
Benoit, K, Laver, M and Mikhaylov, S (2009) Treating words as data with error: uncertainty in text statements of policy positions. American Journal of Political Science 53, 495–513.CrossRefGoogle Scholar
Benoit, K, Conway, D, Lauderdale, BE, Laver, M and Mikhaylov, S (2016) Crowd-sourced text analysis: reproducible and agile production of political data. American Political Science Review 110, 278–95.CrossRefGoogle Scholar
Bonica, A (2016) Database on Ideology, Money in Politics, and Elections: Public Version 2.0. [Computer file]. Stanford, CA: Stanford University Libraries. Available at http://data.stanford.edu/dime.Google Scholar
Brader, T (ed) (2006) Campaigning for Hearts and Minds: How Emotional Appeals in Political Ads Work. Chicago: University of Chicago Press.Google Scholar
Budak, C, Goel, S and Rao, JM (2016) Fair and balanced? Quantifying media bias through crowdsourced content analysis. Public Opinion Quarterly 80(S1), 250–71.CrossRefGoogle Scholar
Carmines, EG and Zeller, RA (1979) Reliability and Validity Assessment. Thousand Oaks, CA: Sage.CrossRefGoogle Scholar
Cohen, J (1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20, 37–46.CrossRefGoogle Scholar
Fowler, EF, Franz, MM and Ridout, TN (2014) Political Advertising in 2010. [Computer file]. Version 1.3. Middletown, CT: The Wesleyan Media Project; Department of Government at Wesleyan University.Google Scholar
Frijda, NH (1988) The laws of emotion. American Psychologist 43, 349–58.CrossRefGoogle ScholarPubMed
Geer, JG (2006) In Defense of Negativity: Attack Ads in Presidential Campaigns. Chicago: University of Chicago Press.CrossRefGoogle Scholar
Gelman, SA and Wellman, HM (1999) Insides and essences: early understandings of the non-obvious. In Eric, M and Laurence, S (eds). Concepts: Core Readings. Cambridge, MA: MIT Press, pp. 613–37.Google Scholar
Grabe, ME and Bucy, EP (2009) Image Bite Politics: News and the Visual Framing of Elections. New York: Oxford University Press.CrossRefGoogle Scholar
Graber, DA (1987) Television news without pictures? Critical Studies in Mass Communication 4, 74–8.CrossRefGoogle Scholar
Graber, DA and Smith, JM (2005) Political communication faces the 21st century. Journal of Communication 55, 479–507.CrossRefGoogle Scholar
Grimmer, J and Stewart, BM (2013) Text as data: the promise and pitfalls of automatic content analysis methods for political texts. Political Analysis 21, 267–97.CrossRefGoogle Scholar
Gwet, KL (2014) Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement among Multiple Raters. Gaithersburg, MD: Advanced Analytics, LLC.Google Scholar
Hayes, D (2011) When gender and party collide: stereotyping in candidate trait attribution. Politics & Gender 7, 133–65.CrossRefGoogle Scholar
Khatchadourian, H (1966) Common names and “Family Resemblances”. In Wittgenstein: The Philosophical Investigations, George, P (ed). Garden City, NY: Anchor Books, pp. 203–30.Google Scholar
Kinder, DR and Kiewiet., DR (1981) Sociotropic politics: the American case. British Journal of Political Science 11, 129–61.CrossRefGoogle Scholar
Kinder, DR, Peters, MD, Abelson, RP and Fiske, ST (1980) Presidential prototypes. Political Behavior 2, 315–37.CrossRefGoogle Scholar
Klein, D (2018) Implementing a general framework for assessing interrater agreement in stata. The Stata Journal 18, 871–901.CrossRefGoogle Scholar
Krippendorff, K (1970) Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement 30, 61–70.CrossRefGoogle Scholar
Lakoff, G (1987) Women, Fire, and Dangerous Things: What Categories Reveal About the Mind. Chicago: University of Chicago Press.CrossRefGoogle Scholar
Lind, F, Gruber, M and Boomgaarden, HG (2017) Content analysis by the crowd: assessing the usability of crowdsourcing for coding latent constructs. Communication Methods and Measures 11, 191–209.CrossRefGoogle ScholarPubMed
Mann, R (2011) Daisy Petals and Mushroom Clouds: LBJ, Barry Goldwater, and the Ad That Changed American Politics. Baton Rouge: Louisiana State University Press.Google Scholar
Masters, RD and Sullivan, DG (1993) Nonverbal behavior and leadership: emotion and cognition in political information processing. In Shanto, I and McGuire, WJ (eds). Explorations in Political Psychology. Durham: Duke University Press, pp. 150–82.CrossRefGoogle Scholar
Mendelberg, T (2001) The Race Card: Campaign Strategy, Implicit Messages, and the Norm of Equality. Princeton, NJ: Princeton University Press.Google Scholar
Messaris, P (1997) Visual Persuasion: The Role of Images in Advertising. Thousand Oaks, CA: Sage Publications.Google Scholar
Messaris, P and Abraham, L (2001) The role of images in framing news stories. In Stephen, DR, Gandy, OH and Grant, AE (eds). Framing Public Life: Perspectives on Media and Our Understanding of the Social World. Mahwah, NJ: Lawrence Erlbaum Associates, pp. 215–26.Google Scholar
Poole, KT and Rosenthal, H (2007) Ideology & Congress. 2nd rev. edn. New Brunswick: Transaction Publishers.Google Scholar
Potter, WJ and Levine-Donnerstein, D (1999) Rethinking validity and reliability in content analysis. Journal of Applied Communication Research 27, 258–84.CrossRefGoogle Scholar
Reynolds, TJ and Whitlark, DB (1995) Applying laddering data to communications strategy and advertising practice. Journal of Advertising Research 35, 9–17.Google Scholar
Rosenberg, SW, Bohan, L, McCafferty, P and Harris., K (1986) The image and the vote: the effect of candidate presentation on voter preference. American Journal of Political Science 30, 108–27.CrossRefGoogle Scholar
Schill, D (2012) The visual image and the political image: a review of visual communication research in the field of political communication. Review of Communication 12, 118–42.CrossRefGoogle Scholar
Spielvogel, C (2005) You know where I stand: moral framing of the war on terrorism and the Iraq war in the 2004 presidential campaign. Rhetoric & Public Affairs 8, 549–69.CrossRefGoogle Scholar
Strach, P, Zuber, K, Fowler, EF, Ridout, TN and Searles, K (2015) In a different voice? Explaining the use of men and women as voice-over announcers in political advertising. Political Communication 32, 183–205.CrossRefGoogle Scholar
Vakharia, D and Lease, M (2015) Beyond AMT: an analysis of paid crowd work platforms. iConference 2015 Proceedings. Available at http://hdl.handle.net/2142/73639.Google Scholar
Weber, R, Mangus, JM, Huskey, R, Hopp, FR, Amir, O, et al. (2018) Extracting latent moral information from text narratives: relevance, challenges, and solutions. Communication Methods and Measures 12, 119–39.CrossRefGoogle Scholar
Wittgenstein, L (1953) Philosophical Investigations. Oxford: B. Blackwell.Google Scholar
Woollacott, J (1982) Messages and meanings. In Tony, B, Curran, J, Gurevitch, M and Wollacott, J (eds). Culture, Society and the Media. New York: Routledge, pp. 87–109.Google Scholar
Supplementary material: Link

Winter et al. Dataset

Dataset

https://doi.org/10.7910/DVN/LWWGKY
Link
Supplementary material: PDF

Winter et al. supplementary material

Online Appendix

Download Winter et al. supplementary material(PDF)
PDF 6.7 MB