Skip to main content Accessibility help
Hostname: page-component-59b7f5684b-s82fj Total loading time: 0.557 Render date: 2022-09-30T03:44:11.493Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "useRatesEcommerce": false, "displayNetworkTab": true, "displayNetworkMapGraph": false, "useSa": true } hasContentIssue true

Short of Suspension: How Suspension Warnings Can Reduce Hate Speech on Twitter

Published online by Cambridge University Press:  22 November 2021


Debates around the effectiveness of high-profile Twitter account suspensions and similar bans on abusive users across social media platforms abound. Yet we know little about the effectiveness of warning a user about the possibility of suspending their account as opposed to outright suspensions in reducing hate speech. With a pre-registered experiment, we provide causal evidence that a warning message can reduce the use of hateful language on Twitter, at least in the short term. We design our messages based on the literature on deterrence, and test versions that emphasize the legitimacy of the sender, the credibility of the message, and the costliness of being suspended. We find that the act of warning a user of the potential consequences of their behavior can significantly reduce their hateful language for one week. We also find that warning messages that aim to appear legitimate in the eyes of the target user seem to be the most effective. In light of these findings, we consider the policy implications of platforms adopting a more aggressive approach to warning users that their accounts may be suspended as a tool for reducing hateful speech online.

© The Author(s), 2021. Published by Cambridge University Press on behalf of the American Political Science Association

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)


A list of permanent links to Supplemental Materials provided by the authors precedes the References section.

Data replication sets are available in Harvard Dataverse at:


Beccaria, Cesare. 1963 [1764]. “On Crimes and Punishments.” Trans. H. Paolucci. Indianapolis, IN: Bobbs-Merrill.Google Scholar
Bodrunova, Svetlana S., Blekanov, Ivan, Smoliarova, Anna, and Litvinenko, Anna. 2019. “Beyond Left and Right: Real-World Political Polarization in Twitter Discussions on Inter-Ethnic Conflicts.” Media and Communication 7(3): 119–32. Scholar
Broockman, David, and Kalla, Joshua. 2016. “Durably Reducing Transphobia: A Field Experiment on Door-to-Door Canvassing.” Science 352(6282): 220–24.CrossRefGoogle ScholarPubMed
Chandrasekharan, Eshwar, Pavalanathan, Umashanthi, Srinivasan, Anirudh, Glynn, Adam, Eisenstein, Jacob, and Gilbert, Eric. 2017. “You Can't Stay Here: The Efficacy of Reddit's 2015 Ban Examined through Hate Speech.” Proceedings of the ACM on Human-Computer Interaction 1(CSCW): 122.CrossRefGoogle Scholar
Charnysh, Volha, Lucas, Christopher, and Singh, Prerna. 2015. “The Ties that Bind: National Identity Salience and Pro-Social Behavior toward the Ethnic Other.” Comparative Political Studies 48(3): 267300.CrossRefGoogle Scholar
Chowdhury, Farhan Asif, Allen, Lawrence, Yousuf, Mohammad, and Mueen, Abdullah. 2020. “On Twitter Purge: A Retrospective Analysis of Suspended Users.” In Companion Proceedings of the Web Conference 2020, 371-378. Scholar
Conzola, Vincent C., and Wogalter, Michael S.. 2001. “A Communication–Human Information Processing (C–HIP) Approach to Warning Effectiveness in the Workplace.” Journal of Risk Research 4(4): 309–22.CrossRefGoogle Scholar
Cusson, Maurice. 1993. “Situational Deterrence: Fear during the Criminal Event.” Crime Prevention Studies 1(3): 5568.Google Scholar
Franco, Annie, Malhotra, Neil, and Simonovits, Gabor. 2014. “Publication Bias in the Social Sciences: Unlocking the File Drawer.” Science 345(6203): 1502–505.CrossRefGoogle ScholarPubMed
Gagliardone, Iginio, Pohjonen, Matti, Beyene, Zenebe, Zerai, Abdissa, Aynekulu, Gerawork, Bekalu, Mesfin, Bright, Jonathan et al. 2016. “Mechachal: Online Debates and Elections in Ethiopia—From Hate Speech to Engagement in Social Media.” Available at SSRN 2831369 ( Scholar
Geerken, Michael R., and Gove, Walter R.. 1974. “Deterrence: Some Theoretical Considerations.” Law and Society Review 9(3): 497513.CrossRefGoogle Scholar
Gibbs, Jack P. 1968. “Crime, Punishment, and Deterrence.” Southwestern Social Science Quarterly 48(4): 515–30.Google Scholar
Guynn, Jessica. 2020. “Facebook Ranks Deleting Anti-Black and ‘Most Harmful’ Hate Speech over Comments about White People and Men.” USA Today, December 3. Retrieved March 1, 2021 ( Scholar
Guynn, Jessica. 2021.“Donald Trump Ruled Facebook, Twitter before He Was Banned. Will @realdonaldtrump Log into Gab or Somewhere Else?” USA Today, February 8. Retrieved March 1, 2021 ( Scholar
Jacobs, Bruce A. 2010. “Deterrence and Deterrability.” Criminology 48(2): 417–41.CrossRefGoogle Scholar
Kalla, Joshua L., and Broockman, David E.. 2020. “Reducing Exclusionary Attitudes through Interpersonal Conversation: Evidence from Three Field Experiments.” American Political Science Review 114(2): 410–25.CrossRefGoogle Scholar
Kiesler, Sara, Kraut, Robert, Resnick, Paul, and Kittur, Aniket. 2012. “Regulating Behavior in Online Communities.” In Building Successful Online Communities: Evidence-Based Social Design, Kraut, Robert E. and Resnick, Paul, 125178. Cambridge, MA: MIT Press. Scholar
Kumar, Sumit, and Pranesh, Raj Ratn. 2021. “TweetBLM: A Hate Speech Dataset and Analysis of Black Lives Matter-related Microblogs on Twitter.” ( Scholar
Livni, Ephrat. 2019. “Twitter, Facebook, and Insta Bans Send the Alt-Right to Gab and Telegram.” qz, May 12. Retrieved March 1, 2021 ( Scholar
Montanaro, Domenico. 2021. “Trump Teases Starting His Own Social Media Platform. Here's Why It'd Be Tough.” NPR, March 24. Retrieved June 1, 2021 ( -why-itd-be-tough).Google Scholar
Munger, Kevin. 2017. “Tweetment Effects on the Tweeted: Experimentally Reducing Racist Harassment.” Political Behavior 39(3): 629–49.CrossRefGoogle Scholar
Munger, Kevin. 2020. “Don’t @ Me: Experimentally Reducing Partisan Incivility on Twitter.” Journal of Experimental Political Science 8(2): 102–16. doi:10.1017/XPS.2020.14CrossRefGoogle Scholar
Müller, Karsten, and Schwarz, Carlo. 2018. “Fanning the Flames of Hate: Social Media and Hate Crime.” Journal of the European Economic Association 19(4): 2131–67.CrossRefGoogle Scholar
Müller, Karsten, and Schwarz, Carlo. 2020. “From Hashtag to Hate Crime: Twitter and Anti-Minority Sentiment.” Available at SSRN 3149103 ( Scholar
Nagin, Daniel S. 1998. “Criminal Deterrence Research at the Outset of the Twenty-First Century.” Crime and Justice 23: 142.CrossRefGoogle Scholar
Paluck, Elizabeth Levy, and Green, Donald P.. 2009a. “Deference, Dissent, and Dispute Resolution: An Experimental Intervention Using Mass Media to Change Norms and Behavior in Rwanda.” American Political Science Review 103(4): 622–44.CrossRefGoogle Scholar
Paluck, Elizabeth Levy, and Green, Donald P.. 2009b. “Prejudice Reduction: What Works? A Review and Assessment of Research and Practice.” Annual Review of Psychology 60(1): 339–67.CrossRefGoogle Scholar
Paternoster, Raymond. 1987. “The Deterrent Effect of the Perceived Certainty and Severity of Punishment: A Review of the Evidence and Issues.” Justice Quarterly 4(2): 173217.CrossRefGoogle Scholar
Pennycook, Gordon, Epstein, Ziv, Mosleh, Mohsen, Arechar, Antonio A., Eckles, Dean, and Rand, David G.. 2021. “Shifting Attention to Accuracy Can Reduce Misinformation Online.” Nature 592(7855): 590–95.CrossRefGoogle ScholarPubMed
Peters, Jay. 2020. “Twitter Now Bans Dehumanizing Remarks Based on Age, Disability, and Disease.” theverge, March 5. Retrieved March 1, 2021 ( Scholar
Pettigrew, Thomas F. 1998. “Intergroup Contact Ttheory.” Annual Review of Psychology 49(1): 6585.CrossRefGoogle Scholar
Rogers, Ronald W. 1975. “A Protection Motivation Theory of Fear Appeals and Attitude Change.” Journal of Psychology 91(1): 93114.CrossRefGoogle Scholar
Samii, Cyrus. 2013. “Perils or Promise of Ethnic Integration? Evidence from a Hard Case in Burundi.” American Political Science Review 107(3): 558–73.CrossRefGoogle Scholar
Sherman, Lawrence W. 1993. “Defiance, Deterrence, and Irrelevance: A Theory of the Criminal Sanction.” Journal of Research in Crime and Delinquency 30(4): 445–73.CrossRefGoogle Scholar
Siegel, Alexandra A., and Badaan, Vivienne. 2020. “# No2Sectarianism: Experimental Approaches to Reducing Sectarian Hate Speech Online.” American Political Science Review 114(3): 837–55.CrossRefGoogle Scholar
Silic, Mario, Silic, Dario, and Oblakovic, Goran. 2016. “Restrictive Deterrence: Impact of Warning Banner Messages on Repeated Low-Trust Software Use.” Presented at the 18th International Conference on Enterprise Information Systems (ICEIS 2016), April 25-28. Scholar
Simonovits, Gábor, Kezdi, Gabor, and Kardos, Peter. 2018. “Seeing the World through the Other's Eye: An Online Intervention Reducing Ethnic Prejudice.” American Political Science Review 112(1): 186–93.CrossRefGoogle Scholar
Spangler, Todd. 2020. “Reddit Finally Bans Hate Speech, Removes 2,000 Racist and Violent Forums Including The_Donald.” variety, June 29. Retrieved March 1, 2021 ( Scholar
Stafford, Mark C., and Warr, Mark. 1993. “A Reconceptualization of General and Specific Deterrence.” Journal of Research in Crime and Delinquency 30(2): 123–35.CrossRefGoogle Scholar
Stockman, Mark, Heile, Robert, and Rein, Anthony. 2015. “An Open-Source Honeynet System to Study System Banner Message Effects on Hackers.” In Proceedings of the 4th annual ACM Conference on Research in Information Technology, 19-2.Google Scholar
Takikawa, Hiroki, and Nagayoshi, Kikuko. 2017. “Political Polarization in Social Media: Analysis of the “Twitter Political Field” in Japan.” 2017 IEEE International Conference on Big Data (Big Data). Scholar
Testa, Alexander, Maimon, David, Sobesto, Bertrand, and Cukier, Michel. 2017. “Illegal Roaming and File Manipulation on Target Computers: Assessing the Effect of Sanction Threats on System Trespassers’ Online Behaviors.” Criminology & Public Policy 16(3): 689726.CrossRefGoogle Scholar
Wilson, Theodore, Maimon, David, Sobesto, Bertrand, and Cukier, Michel. 2015. “The Effect of a Surveillance Banner in an Attacked Computer System: Additional Evidence for the Relevance of Restrictive Deterrence in Cyberspace.” Journal of Research in Crime and Delinquency 52(6): 829–55.CrossRefGoogle Scholar
Wogalter, Michael S. 2006. “Communication-Human Information Processing (C-HIP) Model.” Handbook of Warnings: Case Studies and Analyses, 5161. Boca Raton: CRC Press.CrossRefGoogle Scholar
Ziems, Caleb, He, Bing, Soni, Sandeep, and Kumar, Srijan. 2020. “Racism Is a Virus: Anti-Asian Hate and Counterhate in Social Media during the COVID-19 Crisis.” arXiv preprint. (arXiv:2005.12423).Google Scholar
Supplementary material: File

Yildirim et al. supplementary material

Appendices A-J

Download Yildirim et al. supplementary material(File)
File 6 MB
Supplementary material: Link

Yildirim et al. Dataset


Save article to Kindle

To save this article to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the or variations. ‘’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Short of Suspension: How Suspension Warnings Can Reduce Hate Speech on Twitter
Available formats

Save article to Dropbox

To save this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about saving content to Dropbox.

Short of Suspension: How Suspension Warnings Can Reduce Hate Speech on Twitter
Available formats

Save article to Google Drive

To save this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about saving content to Google Drive.

Short of Suspension: How Suspension Warnings Can Reduce Hate Speech on Twitter
Available formats

Reply to: Submit a response

Please enter your response.

Your details

Please enter a valid email address.

Conflicting interests

Do you have any conflicting interests? *