Skip to main content Accessibility help
Hostname: page-component-77c89778f8-rkxrd Total loading time: 0 Render date: 2024-07-19T22:53:50.009Z Has data issue: false hasContentIssue false

Images as Data for Social Science Research

An Introduction to Convolutional Neural Nets for Image Classification

Published online by Cambridge University Press:  17 July 2020

Nora Webb Williams
University of Illinois, Urbana-Champaign
Andreu Casas
Vrije Universiteit, Amsterdam
John D. Wilkerson
University of Washington


Images play a crucial role in shaping and reflecting political life. Digitization has vastly increased the presence of such images in daily life, creating valuable new research opportunities for social scientists. We show how recent innovations in computer vision methods can substantially lower the costs of using images as data. We introduce readers to the deep learning algorithms commonly used for object recognition, facial recognition, and visual sentiment analysis. We then provide guidance and specific instructions for scholars interested in using these methods in their own research.
Get access
Online ISBN: 9781108860741
Publisher: Cambridge University Press
Print publication: 13 August 2020

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)


A Neural Network Playground (2020). ningRate=0.03&regularizationRate=0&noise=0&networkShape=4,2&se ed=0.81811&showTestData=false&discretize=false&percTrainData=50 &x=true&y=true&xTimesY=fal (last accessed April 26, 2020).Google Scholar
ACMFAccT-Links (2020). (last accessed April 26, 2020).Google Scholar
Anastasopoulos, L. J. et al. (2016). “Photographic Home Styles in Congress: A Computer Vision Approach,” Approach,” 152. Scholar
Bennett, W. L., and Segerberg, A (2013). The Logic of Connective Action: Digital Media and the Personalization of Contentious Politics. New York: Cambridge University Press.CrossRefGoogle Scholar
Benoit, K., et al. (2016). “Crowd-sourced Text Analysis: Reproducible and Agile Production of Political Data.American Political Science Review 110(2), 278295.Google Scholar
Bimber, B., Flanagin, A. J., and Stohl, C (2005). “Reconceptualizing Collective Action in the Contemporary Media Environment.Communication Theory 15(4), 365388.–2885.2005.tb00340.x.CrossRefGoogle Scholar
Brantner, C., Lobinger, K, and Irmgard, W (2011). “Effects of Visual Framing on Emotional Responses and Evaluations of News Stories about the Gaza Conflict 2009.Journalism & Mass Communication Quarterly 88(3), 523540.Google Scholar
Britz, D. (2015). Understanding Convolutional Neural Networks for NLP – WildML. (last accessed April 26, 2020).Google Scholar
Broussard, M. (2018). Artificial Unintelligence: How Computers Misunderstand the World. Cambridge, MA: The Massachusetts Institute of Technology Press.Google Scholar
Budhiraja, A. (2016). Dropout in (Deep) Machine Learning-Amar Budhiraja – Medium. URL: ine-learning-74334da4bfc5 (last accessed April 26, 2020).Google Scholar
Buduma, N., and Locascio, N (2017). Fundamentals of Deep Learning: Designing Next-generation Machine Intelligence Algorithms. Sebastopol, CA: O’Reilly Media.Google Scholar
Buolamwini, J., and Gebru, T (2018). “Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification.In: Proceedings of Machine Learning Research. Vol. 81, pp. 115.Google Scholar
Burns, N., et al. (2011). “Sentiment Analysis of Customer Reviews: Balanced versus Unbalanced Datasets.” In: Knowledge-Based and Intelligent Information and Engineering Systems Ed. by Konig, A et al. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 161170.Google Scholar
Callen, M., and Long, J. D. (2015). “Institutional Corruption and Election Fraud: Evidence from a Field Experiment in Afghanistan.American Economic Review 105(1), 354381.CrossRefGoogle Scholar
Cantú, F. (2019). “The Fingerprints of Fraud: Evidence from Mexico’s 1988 Presidential Election.American Political Science Review. Vol. 113, Issue 3, pp. 710726.Google Scholar
Casas, A., and Webb Williams, N (2018). “Images that Matter: Online Protests and the Mobilizing Role of Pictures.Political Research Quarterly, Vol. 72, Issue 2, 360375.Google Scholar
Casas, A., et al. (2019). “Visual Clustering: A Technique for Drastically Reducing Image Annotation Tasks.” Presented at Annual Meeting of the International Communication Association, Washington, D.C.Google Scholar
Castells, M. (2012). Networks of Outrage and Hope: Social Movements in the Internet Age. Cambridge, UK; Malden, MA: Polity Press.Google Scholar
Cillizza, C. (2018). Melania Trump’s ’I Really Don’t Care. Do U?’ Jacket Was No Mistake- – CNNPolitics. (last accessed April 26, 2020).Google Scholar
Clarke, K., and Kocak, K (2018). Replication Data for: Launching Revolution: Social Media and the Egyptian Uprising’s First Movers. Scholar
COCO – Common Objects in Context (2020). (last accessed April 26, 2020).Google Scholar
Corrigall-Brown, C., and Wilkes, R (2012). “Picturing Protest: The Visual Framing of Collective Action by First Nations in Canada.American Behavioral Scientist 56(2), 223243.Google Scholar
CS231n Convolutional Neural Networks for Visual Recognition (2020). (last accessed April 25, 2020).Google Scholar
Dahmen, N. S. (2012). “Photographic Framing in the Stem Cell Debate.American Behavioral Scientist 56(2), 189203.Google Scholar
Dietrich, B.J. (2019). “Using Motion Detection to Measure Social Polarization in the U.S. House of Representatives.”Google Scholar
Domhan, T., Springenberg, J, and Hutter, F (2015). “Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves”. IJCAI’15: Proceedings of the 24th International Conference on Artificial Intelligence, pp. 34603468Google Scholar
Commission, European (2020). EU Data Protection Rules. info/priorities/justice-and-fundamental-rights/data-protection/2018-reform-eu-data-protection-rules/eu-data-protection-rules_en (last accessed April 26, 2020).Google Scholar
Geitgey, A. (2020). ageitgey/face_recognition: The World’s Simplest Facial Recognition API for Python and the Command Line. (last accessed April 26, 2020).Google Scholar
Gelman, A., and Hill, J (2007). Data Analysis Using Regression and Multilevel/Hierarchical Models. New York, NY: Cambridge University Press.Google Scholar
Gender Shades (2020). (last accessed April 26,2020).Google Scholar
Girshick, R. B. (2015). “Fast {R-CNN}.” CoRR abs/1504.0. Scholar
Girshick, R. B., et al. (2013). “Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation.” CoRR abs/1311.2. Scholar
Gitlin, T. (1980). The Whole World Is Watching: Mass Media in the Making and Unmaking of the New Left. Berkeley: University of California Press.Google Scholar
Goh, G. (2017). “Why Momentum Really Works.” Distill. Scholar
Golle, P. (2008). “Machine Learning Attacks against the Asirra CAPTCHA.” In: CCS ’08 Proceedings of the 15th ACM Conference on Computer and Communications Security, pp. 535542.Google Scholar
Grabe, M. E., and Bucy, E. P. (2009). Image Bite Politics: News and the Visual Framing of Elections. Oxford; New York: Oxford University Press.Google Scholar
Guo, Y., et al. (2016). “MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition.” In: European Conference on Computer Vision (ECCV). Scholar
Hamner, B. (2019). 2016 US Election | Kaggle. (last accessed April 26, 2020).Google Scholar
He, K., Gkioxari, G, et al. (2017). “Mask {R-CNN}.” CoRR abs/1703.0. Scholar
He, K., Zhang, X, et al. (2015). “Deep Residual Learning for Image Recognition.” In: arXiv:1512.03385.Google Scholar
Henderson, J. V., Storeygard, A, and Weil, D. N. (2012). “Measuring Economic Growth from Outer Space Author.American Economic Review 102(2), 9941028.Google Scholar
Horiuchi, Y., Komatsu, T, and Nakaya, F (2012). “Should Candidates Smile to Win Elections? An Application of Automated Face Recognition Technology.Political Psychology 33(6), 925933.Google Scholar
Howard, P. N., andHussain, M. M. (2013). Democracy’s Fourth Wave?: Digital Media and the Arab Spring. New York, NY: Oxford University Press.Google Scholar
Hwang, J., Imai, K, and Tarr, A (2019). “Automated Coding of Political Campaign Advertisement Videos: An Empirical Validation Study.” Kosuke Imai’s Homepage. Scholar
ImageNet (2020). (last accessed April 26, 2020).Google Scholar
Internet Live Stats – Internet Usage & Social Media Statistics (2020). (last accessed April 26, 2020).Google Scholar
Introna, L. D., and Wood, D (2004). “Picturing Algorithmic Surveillance: The Politics of Facial Recognition Systems.Surveillance and Society 2(2–3), 177198.Google Scholar
Iyer, A., and Oldmeadow, J (2006). “Picture This: Emotional and Political Responses to Photographs of the Kenneth Bigley Kidnapping.European Journal of Social Psychology 36(5), 635647.Google Scholar
Jean, N., et al. (2016). “Combining Satellite Imagery and Machine Learning to Predict Poverty.Science 353(6301), 790794. Scholar
Joo, J., Bucy, E. P., and Seidel, C (2019). “Automated Coding of Televised Leader Displays: Detecting Nonverbal Political Behavior with Computer Vision and Deep Learning.International Journal of Communication. Vol. 19, pp. 40444066.Google Scholar
Joo, J., Li, W, et al. (2014). “Visual Persuasion: Inferring Communicative Intents of Images.” In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, pp. 216223.CrossRefGoogle Scholar
Joo, J., Steen, F. F., and Zhu, S.-C. (2015). “Automated Facial Trait Judgment and Election Outcome Prediction: Social Dimensions of Face.” In: Proceedings of the IEEE International Conference on Computer Vision. IEEE, pp. 37123720.Google Scholar
Kaufman, A., King, G, and Komisarchik, M (2019). “How to Measure Legislative District Compactness If You Only Know It When You See it.”Google Scholar
Kearns, M., and Roth, A (2019). The Ethical Algorithm: The Science of Socially Aware Algorithm Design Oxford: Oxford University Press.Google Scholar
Kharroub, T., and Bas, O (2015). “Social Media and Protests: An Examination of Twitter Images of the 2011 Egyptian Revolution.New Media & Society 18(9): 1973 –1992. Scholar
Krizhevsky, A., Sutskever, I, and Hinton, G. E. (2012). “ImageNet Classification with Deep Convolutional Neural Networks.” In: Advances in Neural Information Processing Systems, pp. 11061114.Google Scholar
Kulkarni, G., et al. (2013). “BabyTalk: Understanding and Generating Simple Image Descriptions.IEEE Transactions on Pattern Analysis and Machine Intelligence 35(12), 28912903.Google Scholar
Lam, O.,etal. (2019). Men Appear Twice as Often as Women in News Photos on Facebook. Tech. rep. Pew Research Center. Scholar
University, Lancaster (2020). GDPR: What Researchers Need to Know | Lancaster University. (last accessed April 26, 2020).Google Scholar
LeCun, Y., Bengio, Y, and Hinton, G (2015). “Deep Learning.Nature 521(7553), 436444.CrossRefGoogle ScholarPubMed
Li, H., et al. (2015). “A Convolutional Neural Network Cascade for Face Detection.” In: 2015 IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
Lin, T. Y., et al. (2014). “Microsoft COCO: Common Objects in Context.” In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 8693 LNCS. PART 5. Springer Verlag, pp. 740755.CrossRefGoogle Scholar
Marcus, G. E., Neuman, W. R., and MacKuen, M (2000). Affective Intelligence and Political Judgement. Chicago and London: University of Chicago Press.Google Scholar
Mebane, W. R. J., et al. (2017). “Using Twitter to Observe Election Incidents in the United States.” (Paper presented at the 2016 Annual Meeting of the Midwest Political Science Association, Chicago, April 69, 2017)Google Scholar
Messaris, P., and Abraham, L (2001). “The Role of Images in Framing News Stories.” In: Framing Public Life: Perspectives on Media and Our Understanding of the Social World. Ed. by Reese, Stephen D., Gandy, Oscar H., and Grant, A. E.. Mahwah, NJ: Lawrence Erlbaum Associates Publishers, pp. 215226.Google Scholar
Metz, C. (2019). Facial Recognition Tech Is Growing Stronger, Thanks to Your Face. Scholar
Moreno, M. A., et al. (2013). “Ethics of Social Media Research: Common Concerns and Practical Considerations”. Cyberpsychology, Behavior, and Social Networking 16(9), 708713. URL: Scholar
Mountassir, A., Hourda, Benbrahim, and Berrada, I (2012). “An Empirical Study to Address the Problem of Unbalanced Data Sets in Sentiment Classification.” In: 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 32983303.CrossRefGoogle Scholar
MS-Celeb-1M: Challenge of Recognizing One Million Celebrities in the Real World-Microsoft Research (2020). (last accessed April 26, 2020).Google Scholar
Nanne, A., et al. (2019). “The Use of Computer Vision to Analyze Visual Brand-Related User Generated Content: A Comparison of YOLOV2, Google Cloud Vision, and Clarifai.Journal of Interactive Marketing 50, 156167.Google Scholar
Nelson, D. L., Reed, V. S., and Walling, J. R. (1976). “Pictorial Superiority Effect.Journal of Experimental Psychology: Human Learning and Memory 2(5), 523528.Google Scholar
O’Neil, C. (2017). Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy New York: Broadway Books.Google Scholar
Owen, D. (2018). “Should We Be Worried About Computerized Facial Recognition?” The New Yorker. Scholar
Paivio, A., Rogers, T. B., and Smythe, P. C. (1968). “Why Are Pictures Easier to Recall Than Words?Psychonomic Science 11(4), 137138.Google Scholar
Peng, K.-c., et al. (2015). “A Mixed Bag of Emotions: Model, Predict, and Transfer Emotion Distributions.” In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 19.Google Scholar
Peng, Y. (2018). “Same Candidates, Different Faces: Uncovering Media Bias in Visual Portrayals of Presidential Candidates with Computer Vision.Journal of Communication 68(5), 920941.Google Scholar
Peng, Y. (2020). COMPUTER VISION – YILANG PENG, computer-vision/ (last accessed April 26, 2020).Google Scholar
Philipp, H., Müller-Crepon, C, and Cederman, L.-E. (n.d.). “Roads to Rule, Roads to Rebel: Relational State Capacity and Conflict in Africa.”Google Scholar
Powell, T., et al. (2015). “A Clearer Picture: The Contribution of Visuals and Text to Framing Effects.Journal of Communication 65(6), 9971017.Google Scholar
Project Jupyter | Home (2020). (last accessed April 26, 2020).Google Scholar
Raiford, L. (2007). “World Together: SNCC and Photography of the Civil Rights Movement.American Quarterly 59(4), 11291157.Google Scholar
Redmon, J., Divvala, S, et al. (2016). “You Only Look Once: Unified, Real Time Object Detection.” In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779788.Google Scholar
Redmon, J., and Farhadi, A (2018). “YOLOv3: An Incremental Improvement.” Scholar
Ren, S., et al. (2015). “Faster {R-CNN:} Towards Real-Time Object Detection with Region Proposal Networks.” CoRR abs/1506.0. Scholar
Ribeiro, M. T., Singh, S, and Guestrin, C (2016). “ ‘Why Should I Trust You?’: Explaining the Predictions of Any Classifier.” In: Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16. New York, N: ACM, pp. 11351144. Scholar
Rosenberg, M., Confessore, N, and Cadwalladr, C (2018). “How Trump Consultants Exploited the Facebook Data of Millions.” The New York Times. Scholar
Rosenberg, S. W., et al. (1986). “The Image and the Vote: The Effect of Candidate Presentation on Voter Preference.American Journal of Political Science 30(1), 108127.Google Scholar
Ruder, S. (2016). “An Overview of Gradient Descent Optimization Algorithms.” CoRR abs/1609.0. Scholar
Russakovsky, O., et al. (2015). “ImageNet Large Scale Visual Recognition Challenge.International Journal of Computer Vision (IJCV) 115(3), 211252.Google Scholar
Saldaña, J. (2009). The Coding Manual for Qualitative Researchers. Thousand Oaks, CA: Sage Publications Ltd.Google Scholar
Schmidhuber, J. (2015). “Deep Learning in Neural Networks: An Overview”. Neural Networks 61, 85117.Google Scholar
Simonite, T. (2017). Machines Learn a Biased View of Women. www.wired .com/story/machines-taught-by-photos-learn-a-sexist-view-of-women/ (last accessed April 26, 2020).Google Scholar
Simonite, T. (2018). When It Comes to Gorillas, Google Photos Remains Blind www.wir (last accessed April 26, 2020).Google Scholar
Smith, L. N. (2018). “A Disciplined Approach to Neural Network Hyperparameters: Part 1 – Learning Rate, Batch Size, Momentum, and Weight Decay.” Scholar
Smith, R. (2007). “An Overview of the Tesseract OCR Engine.” In: Proceedings of the International Conference on Document Analysis and Recognition, ICDAR. Vol. 2, pp. 629633.Google Scholar
Sobolev, A., et al. (n.d.). “News and Geolocated Social Media Accurately Measure Protest Size.Cartography and Geographic Information Science 41(3).Google Scholar
Srivastava, N., et al. (2014). “Dropout: A Simple Way to Prevent Neural Networks from Overfitting.J. Mach. Learn. Res. 15(1), 19291958. Scholar
Steinert-Threlkeld, Z. C. (2018). Twitter as Data. Elements in Quantitative and Computational Methods for the Social Sciences. Cambridge: Cambridge University Press.Google Scholar
Steinert-Threlkeld, Z. C., and Joo, J (n.d.). “Event Data from Images.”Google Scholar
Suppe, R. (2018). “Orlando Police Decide to Keep Testing Controversial Amazon Facial Recognition Program.” USA Today. Scholar
Tai, L., and Liu, M (2016). “Mobile Robots Exploration through CNN- Based Reinforcement Learning.Robotics and Biomimetics 3(1), 24.–0055-x.Google Scholar
Tanksale, N. (2018). Finding Good Learning Rate and The One Cycle Policy. (last accessed April 26, 2020).Google Scholar
Taylor, L., and Nitschke, G (2017). “Improving Deep Learning using Generic Data Augmentation.” CoRR abs/1708.0. Scholar
Tesseract documentation | Tesseract OCR (2020). (last accessed April 26, 2020).Google Scholar
Todorov, A., et al. (2005). “Inferences of Competence from Faces Predict Election Outcomes.Science 308(5728), 16231626.Google Scholar
torchvision.modelsPyTorch Master Documentation (2020). (last accessed April 26, 2020).Google Scholar
Torres, M. (2019). “Give Me the Full Picture: Using Computer Vision to Understand Visual Frames and Political Communication.”Google Scholar
University of Oxford (2020). Responsibilities under GDPR | Research Support. (last accessed April 26, 2020).Google Scholar
Webb Williams, N., and Casas, A (2020). “norawebbwilliams/images_as_data: First Release.” Github.Google Scholar
Webb Williams, N., Casas, A, and Wilkerson, J (2020). Images as Data for Social Science Research. 879/tree/v1.Google Scholar
Wickham, H. (2017). tidyverse: Easily Install and Load the “Tidyverse.” Scholar
Williams, M. L., Burnap, P, and Sloan, L (2017). “Towards an Ethical Framework for Publishing Twitter Data in Social Research: Taking into Account Users’ Views, Online Context and Algorithmic Estimation.Sociology 51(6), 11491168.Google Scholar
Wilson, D. R., and Martinez, T. R. (2003). “The General Inefficiency of Batch Training for Gradient Descent Learning.Neural Networks 16(10), 14291451.–2.Google Scholar
Won, D., Steinert-Threlkeld, Z. C., and Joo, J (2017). “Protest Activity Detection and Perceived Violence Estimation from Social Media Images.” In: Proceedings of the 25th ACM International Conference on Multimedia. Scholar
You, Q. et al. (2015). “Robust Image Sentiment Analysis using Progressively Trained and Domain Transferred Deep Networks.” In: The Twenty-Ninth AAAI Conference, pp. 381388.Google Scholar
Zhang, H., and Pan, J (2019). “CASM: A Deep Learning Approach for Identifying Collective Action Events with Text and Image Data from Social Media.Sociological Methodology 49(1), 157.Google Scholar
Zhang, Q., and Zhu, S.-C. (2018). “Visual Interpretability for Deep Learning: a Survey.” Scholar
Zhao, J., et al. (2017). “Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-Level Constraints.” CoRR abs/1707.0. Scholar
Zhu, X., and Ramanan, D (2012). “Face Detection, Pose Estimation, and Landmark Estimation in the Wild.” In: International Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 28792886.Google Scholar

Save element to Kindle

To save this element to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the or variations. ‘’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Images as Data for Social Science Research
Available formats

Save element to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Images as Data for Social Science Research
Available formats

Save element to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Images as Data for Social Science Research
Available formats