Skip to main content Accessibility help
×
Home
Hostname: page-component-559fc8cf4f-s5ss2 Total loading time: 1.021 Render date: 2021-02-27T02:28:22.707Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "metricsAbstractViews": false, "figures": false, "newCiteModal": false, "newCitedByModal": true }

Online coders, open codebooks: New opportunities for content analysis of political communication

Published online by Cambridge University Press:  27 March 2019

Nicholas J. G. Winter
Affiliation:
Department of Politics, University of Virginia, S185 Gibson Hall, 1540 Jefferson Park Ave, Charlottesville, VA
Adam G. Hughes
Affiliation:
Pew Research Center, Washington, D.C.
Lynn M. Sanders
Affiliation:
Department of Politics, University of Virginia, S185 Gibson Hall, 1540 Jefferson Park Ave, Charlottesville, VA
Corresponding
E-mail address:

Abstract

Analyzing audiovisual communication is challenging because its content is highly symbolic and less rule-governed than verbal material. But audiovisual messages are important to understand: they amplify, enrich, and complicate the meaning of textual information. We describe a fully-reproducible approach to analyzing video content using minimally—but systematically—trained online workers. By aggregating the work of multiple coders, we achieve reliability, validity, and costs that equal those of traditional, intensively trained research assistants, with much greater speed, transparency, and replicability. We argue that measurement strategies relying on the “wisdom of the crowd” provide unique advantages for researchers analyzing complex and intricate audiovisual political content.

Type
Original Article
Copyright
Copyright © The European Political Science Association 2019

Access options

Get access to the full version of this content by using one of the access options below.

References

Armstrong, SL, Gleitman, LR and Gleitman, H (1999) What some concepts might not be. In Margolis, E and Laurence, S (eds). Concepts: Core Readings. Cambridge, MA: MIT Press, pp. 225–59.Google Scholar
Benoit, WL (2010) Content analysis in political communication. In Bucy, EP and Holbert, RL (eds). Sourcebook for Political Communication Research: Methods, Measures, and Analytic Techniques. New York: Taylor & Francis, pp. 268–79.Google Scholar
Benoit, K, Laver, M and Mikhaylov, S (2009) Treating words as data with error: uncertainty in text statements of policy positions. American Journal of Political Science 53, 495513.CrossRefGoogle Scholar
Benoit, K, Conway, D, Lauderdale, BE, Laver, M and Mikhaylov, S (2016) Crowd-sourced text analysis: reproducible and agile production of political data. American Political Science Review 110, 278–95.CrossRefGoogle Scholar
Bonica, A (2016) Database on Ideology, Money in Politics, and Elections: Public Version 2.0. [Computer file]. Stanford, CA: Stanford University Libraries. Available at http://data.stanford.edu/dime.Google Scholar
Brader, T (ed) (2006) Campaigning for Hearts and Minds: How Emotional Appeals in Political Ads Work. Chicago: University of Chicago Press.Google Scholar
Budak, C, Goel, S and Rao, JM (2016) Fair and balanced? Quantifying media bias through crowdsourced content analysis. Public Opinion Quarterly 80(S1), 250–71.CrossRefGoogle Scholar
Carmines, EG and Zeller, RA (1979) Reliability and Validity Assessment. Thousand Oaks, CA: Sage.CrossRefGoogle Scholar
Cohen, J (1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20, 3746.CrossRefGoogle Scholar
Fowler, EF, Franz, MM and Ridout, TN (2014) Political Advertising in 2010. [Computer file]. Version 1.3. Middletown, CT: The Wesleyan Media Project; Department of Government at Wesleyan University.Google Scholar
Frijda, NH (1988) The laws of emotion. American Psychologist 43, 349–58.CrossRefGoogle ScholarPubMed
Geer, JG (2006) In Defense of Negativity: Attack Ads in Presidential Campaigns. Chicago: University of Chicago Press.CrossRefGoogle Scholar
Gelman, SA and Wellman, HM (1999) Insides and essences: early understandings of the non-obvious. In Eric, M and Laurence, S (eds). Concepts: Core Readings. Cambridge, MA: MIT Press, pp. 613–37.Google Scholar
Grabe, ME and Bucy, EP (2009) Image Bite Politics: News and the Visual Framing of Elections. New York: Oxford University Press.CrossRefGoogle Scholar
Graber, DA (1987) Television news without pictures? Critical Studies in Mass Communication 4, 74–8.CrossRefGoogle Scholar
Graber, DA and Smith, JM (2005) Political communication faces the 21st century. Journal of Communication 55, 479507.CrossRefGoogle Scholar
Grimmer, J and Stewart, BM (2013) Text as data: the promise and pitfalls of automatic content analysis methods for political texts. Political Analysis 21, 267–97.CrossRefGoogle Scholar
Gwet, KL (2014) Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement among Multiple Raters. Gaithersburg, MD: Advanced Analytics, LLC.Google Scholar
Hayes, D (2011) When gender and party collide: stereotyping in candidate trait attribution. Politics & Gender 7, 133–65.CrossRefGoogle Scholar
Khatchadourian, H (1966) Common names and “Family Resemblances”. In Wittgenstein: The Philosophical Investigations, George, P (ed). Garden City, NY: Anchor Books, pp. 203–30.Google Scholar
Kinder, DR and Kiewiet., DR (1981) Sociotropic politics: the American case. British Journal of Political Science 11, 129–61.CrossRefGoogle Scholar
Kinder, DR, Peters, MD, Abelson, RP and Fiske, ST (1980) Presidential prototypes. Political Behavior 2, 315–37.CrossRefGoogle Scholar
Klein, D (2018) Implementing a general framework for assessing interrater agreement in stata. The Stata Journal 18, 871901.CrossRefGoogle Scholar
Krippendorff, K (1970) Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement 30, 6170.CrossRefGoogle Scholar
Lakoff, G (1987) Women, Fire, and Dangerous Things: What Categories Reveal About the Mind. Chicago: University of Chicago Press.CrossRefGoogle Scholar
Lind, F, Gruber, M and Boomgaarden, HG (2017) Content analysis by the crowd: assessing the usability of crowdsourcing for coding latent constructs. Communication Methods and Measures 11, 191209.CrossRefGoogle ScholarPubMed
Mann, R (2011) Daisy Petals and Mushroom Clouds: LBJ, Barry Goldwater, and the Ad That Changed American Politics. Baton Rouge: Louisiana State University Press.Google Scholar
Masters, RD and Sullivan, DG (1993) Nonverbal behavior and leadership: emotion and cognition in political information processing. In Shanto, I and McGuire, WJ (eds). Explorations in Political Psychology. Durham: Duke University Press, pp. 150–82.CrossRefGoogle Scholar
Mendelberg, T (2001) The Race Card: Campaign Strategy, Implicit Messages, and the Norm of Equality. Princeton, NJ: Princeton University Press.Google Scholar
Messaris, P (1997) Visual Persuasion: The Role of Images in Advertising. Thousand Oaks, CA: Sage Publications.Google Scholar
Messaris, P and Abraham, L (2001) The role of images in framing news stories. In Stephen, DR, Gandy, OH and Grant, AE (eds). Framing Public Life: Perspectives on Media and Our Understanding of the Social World. Mahwah, NJ: Lawrence Erlbaum Associates, pp. 215–26.Google Scholar
Poole, KT and Rosenthal, H (2007) Ideology & Congress. 2nd rev. edn. New Brunswick: Transaction Publishers.Google Scholar
Potter, WJ and Levine-Donnerstein, D (1999) Rethinking validity and reliability in content analysis. Journal of Applied Communication Research 27, 258–84.CrossRefGoogle Scholar
Reynolds, TJ and Whitlark, DB (1995) Applying laddering data to communications strategy and advertising practice. Journal of Advertising Research 35, 917.Google Scholar
Rosenberg, SW, Bohan, L, McCafferty, P and Harris., K (1986) The image and the vote: the effect of candidate presentation on voter preference. American Journal of Political Science 30, 108–27.CrossRefGoogle Scholar
Schill, D (2012) The visual image and the political image: a review of visual communication research in the field of political communication. Review of Communication 12, 118–42.CrossRefGoogle Scholar
Spielvogel, C (2005) You know where I stand: moral framing of the war on terrorism and the Iraq war in the 2004 presidential campaign. Rhetoric & Public Affairs 8, 549–69.CrossRefGoogle Scholar
Strach, P, Zuber, K, Fowler, EF, Ridout, TN and Searles, K (2015) In a different voice? Explaining the use of men and women as voice-over announcers in political advertising. Political Communication 32, 183205.CrossRefGoogle Scholar
Vakharia, D and Lease, M (2015) Beyond AMT: an analysis of paid crowd work platforms. iConference 2015 Proceedings. Available at http://hdl.handle.net/2142/73639.Google Scholar
Weber, R, Mangus, JM, Huskey, R, Hopp, FR, Amir, O, et al. (2018) Extracting latent moral information from text narratives: relevance, challenges, and solutions. Communication Methods and Measures 12, 119–39.CrossRefGoogle Scholar
Wittgenstein, L (1953) Philosophical Investigations. Oxford: B. Blackwell.Google Scholar
Woollacott, J (1982) Messages and meanings. In Tony, B, Curran, J, Gurevitch, M and Wollacott, J (eds). Culture, Society and the Media. New York: Routledge, pp. 87109.Google Scholar

Winter et al. Dataset

Link

Winter et al. supplementary material

Online Appendix

PDF 7 MB

Altmetric attention score

Full text views

Full text views reflects PDF downloads, PDFs sent to Google Drive, Dropbox and Kindle and HTML full text views.

Total number of HTML views: 140
Total number of PDF views: 174 *
View data table for this chart

* Views captured on Cambridge Core between 27th March 2019 - 27th February 2021. This data will be updated every 24 hours.

Send article to Kindle

To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Online coders, open codebooks: New opportunities for content analysis of political communication
Available formats
×

Send article to Dropbox

To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

Online coders, open codebooks: New opportunities for content analysis of political communication
Available formats
×

Send article to Google Drive

To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

Online coders, open codebooks: New opportunities for content analysis of political communication
Available formats
×
×

Reply to: Submit a response


Your details


Conflicting interests

Do you have any conflicting interests? *