2 Automation, terrorist content and the rule of law

At first, the deployment of AI to block terrorist content from social-media platforms seems some way removed from discussions of the rule of law, in two respects: the mode of regulation and the responsible actors. Starting with the latter, the rule of law is traditionally associated with public institutions, not private technology companies. Yet, in the current context, a steadfast public–private distinction is difficult to maintain. The importance of public–private partnership in responding to terrorists’ use of the Internet is widely recognised (Kavanagh et al., Reference Kavanagh and Conway2017). In addition to take-down requests issued by government agencies, there are a number of multi-stakeholder initiatives at both the national level (including the technology developed by the UK Home Office mentioned earlier) and the international level (including the EU Internet Forum, which brings together governments, Europol and technology companies and Tech Against Terrorism, a project that supports technology companies (particularly start-ups and small companies) that is mandated by the UN Counter Terrorism Committee Executive Directorate (UN CTED) and implemented by the ICT4Peace Foundation). More fundamentally, states have a responsibility to protect national security. As Lord Hope stated in A v. Secretary of State for the Home Department: ‘It is the first responsibility of government in a democratic society to protect and safeguard the lives of its citizens.’Footnote ⁶ At the same time, Lord Hope continued, the rights and liberties of individuals must be respected. Indeed, as the UK's CONTEST strategy emphasises, respect for human rights and the rule of law is in itself an important component of an effective counter-terrorism strategy (HM Government, 2018). It is important, therefore, that the policies and practices that technology companies adopt to counter terrorist activities also reflect this commitment to human rights and the rule of law.

In terms of the mode of regulation, the rule of law would traditionally be associated with governance by rules, such as those prescribed in statute and expounded in case-law. However, ‘From the fact that law is conceived of as a normative phenomenon, it does not follow that the ambit of jurisprudence should be limited to legal norms’ (Brownsword, Reference Brownsword2015, p. 10). As Brownsword has argued, to limit the ambit of jurisprudence to legal norms would be to exclude other kinds of normative order that are found in society (such as religious and moral codes) and which have an important influence on ‘what makes things tick in the social world [and] how legal norms fit into the full array of normative signals’ (Brownsword, Reference Brownsword2015, p. 12). Equally, it would be mistaken to limit the domain of jurisprudence to the exclusively normative:

A ‘regulatory environment’ should thus be understood ‘as an action-guiding environment in which regulators direct the conduct of regulatees with a view to achieving a particular regulatory objective’ (Brownsword, Reference Brownsword2011, p. 1327).

What makes the use of AI to prevent terrorist content from reaching social-media platforms appealing is its greater effectiveness in comparison to the moral and prudential signals found in these companies’ terms of service. At the same time, however, it is important that this ‘brute instrumentalism … is conditioned by principles that give it legitimacy’ (Brownsword, Reference Brownsword2016, p. 138). So, regardless of the different regulatory form, rule-of-law principles are of continuing relevance:

In short, ‘it is the ideal of legality together with the Rule of Law that stands between us and a disempowering techno-managed future’ (Brownsword, Reference Brownsword2016, p. 138).

3 Imperfect impossibility and tactical displacement

The study of the population of IS supporters on Twitter conducted by Berger and Morgan (Reference Berger and Morgan2015) found that, during October and November 2014, there were no fewer than 46,000 overt IS supporter accounts on Twitter – and possibly as many as 90,000. The average number of followers of these accounts was 1,004 and each account posted an average of 7.3 tweets per day over its lifetime. Since this ‘Golden Age’ (Conway et al., Reference Conway2017, p. 28), IS's presence on Twitter has been reduced significantly. Towards the end of 2014, Twitter began an aggressive campaign of suspensions. Berger and Morgan (Reference Berger and Morgan2015) found that, by February 2015, IS supporters on Twitter were having to devote far more time to rebuilding their networks (as opposed to disseminating propaganda, recruiting and other activities). A follow-up study conducted by Berger and Perez (Reference Berger and Perez2016) also found that suspension activity had a significant disruptive effect. Individual users who repeatedly created new accounts after being suspended ‘suffered devastating reductions in their follower counts’ and declines in networks persisted even when suspension pressure eased, ‘suggesting that suspensions diminish activity in ways that extend beyond the simple removal of accounts’ (p. 4). Those IS supporters who persisted in their use of the platform resorted to the use of countermeasures, such as locking their accounts so that they were no longer publicly accessible, using an innocuous image or the default egg as the avatar image and selecting a random combination of letters and numbers as the user handle or screen name. But, since ‘A conscious, supportive and influential virtual community is almost impossible to maintain in the face of the loss of access to such group or ideological symbols and the resultant breakdown in commitment’, today IS's Twitter activity ‘has largely been reduced to tactical use of throwaway accounts for distributing links to pro-IS content on other platforms, rather than as a space for public IS support and influencing activity’ (Conway et al., Reference Conway2017, p. 30) – and these throwaway accounts are also suspended before they gain much of a following (Grinnell et al., Reference Grinnell, Macdonald and Mair2017; Reference Grinnell2018).

Whilst considerable progress has been made in disrupting the use of Twitter by IS supporters, Conway et al. (Reference Conway2017) found that supporters of other jihadist groups – such as Hay'at Tahrir al-Sham (HTS), Ahrar al-Sham, the Taliban and al-Shabaab – experienced significantly less disruption. Whilst more than 30 percent of pro-IS accounts were suspended within 48 hours of their creation, the equivalent figure for these other pro-jihadist accounts was less than 1 percent.Footnote ⁷ The latter were also able to post six times as many tweets, follow four times as many accounts and gain thirteen times as many followers as the pro-IS accounts (Conway et al., Reference Conway2017).Footnote ⁸

The criminological literature on situational crime prevention highlights the fact that no pre-emptive scheme is perfect (Rosenthal, Reference Rosenthal2011). As Rich (Reference Rich2013) observes: ‘Though the goal of an impossibility structure is to make the commission of a crime practically impossible, the reality is that the ingenuity of potential criminals is inexhaustible’ (p. 823). One technique that has been used by IS supporters is out-linking, namely the use of Twitter as a ‘gateway’ (O'Callaghan et al., Reference O'Callaghan2013, p. 276), to other websites, social-media platforms and content-hosting sites. In Conway et al.’s (Reference Conway2017) study, 7,216 (12.5 percent) of the 57,574 tweets analysed contained out-links. The platform with the most out-links from Twitter was YouTube. Interestingly, however, Facebook did not appear in the top ten, whilst the less-known justpaste.it, sendvid.com and archive.org all featured in the top six.Footnote ⁹ The use of these smaller platforms appears to be an attempt to ‘exploit an overlapping ecosystem of services’, taking advantage of the fact that smaller companies ‘don't have the scale or resources to handle the challenge on their own’ (Tech Against Terrorism, Reference Tech2017). Justpaste.it, for example, is a free content-sharing service that allows content to be posted within seconds with no registration required. Owned by Mariusz Zurawek, who runs the site out of his home in Poland, the content posted on Justpaste.it began to include IS propaganda in early 2014. By March 2015, Zurawek estimated that he had removed up to 2,000 posts at the request of London Metropolitan Police (Stalinsky and Sosnow, Reference Stalinsky and Sosnow2016). Since then, he has received a large volume of take-down requests from all over the world. This poses challenges in terms of identifying what content is legal and responding to take-down requests in other languages, as well as capacity and resources (Tech Against Terrorism, Reference Tech2017).

Impossibility structures may also lead to ‘tactical displacement’ (Rich, Reference Rich2013, p. 824). Besides the use of throwaway Twitter accounts out-linking to propaganda hosted on other platforms, IS supporters have largely moved their community-building activities to other platforms, in particular Telegram. Telegram is a cloud-based instant-messaging service, providing optional end-to-end encrypted messaging. Features include: a self-destruct timer that permanently deletes messages and media after they are viewed; group chats, which users can only join when invited to do so; and channels that are public and so can be used to broadcast messages to large audiences. As IS's presence on Twitter diminished, a migration to Telegram occurred. By early 2016, Telegram was being used to share content produced by official IS channels and IS members and supporters were pushing out more than 30,000 Telegram messages each week (Prucha, Reference Prucha2016). Telegram chat-rooms and channels also possess psychologically addictive qualities, which aids efforts to build virtual communities as well as the promotion of so-called self-starter terrorism (Bloom et al., Reference Bloom, Tiflati and Horgan2017). These uses of Telegram form part of a wider movement towards the use of more covert methods (UN Security Council, 2017). As well as Telegram, other encrypted messaging services, including WhatsApp, have been used by jihadists for communication and attack-planning (Malik, Reference Malik2018). Websites have also been relocated to the Darknet: the day after the 2015 Paris attacks, for example, IS's media division Al-Hayat Media Center launched a new website on the Darknet to host its propaganda, including a video celebrating the attacks (Ragan, Reference Ragan2015). Darknet platforms are, by definition, more difficult to police than the surface and deep webs, meaning they have the potential to function as a jihadist ‘virtual safe-haven’ (Malik, Reference Malik2018, p. iv).

An important aspect of the rule of law is enforcement: rule by law is a necessary condition for the rule of law (Spigelman, Reference Spigelman2003). What the examples discussed in this section illustrate is that, if the aim of blocking terrorist content online is to be realised, regard must be had to the whole social-media ‘ecology’ (Conway et al., Reference Conway2017, p. 32). This involves broadening the focus beyond just the biggest social-media companies. It has been reported that, in 2016, the UK's Counter Terrorism Internet Referral Unit reported content across 300 different services (Tech Against Terrorism, Reference Tech2017). The platforms offered by smaller companies are also susceptible to exploitation, and the challenges and barriers to these companies self-regulating may be quite different to the social-media giants. Meanwhile, calls such as those from the UK's Home Secretary for ‘backdoors’ to be built into encrypted messaging services also raise complex issues around privacy and security (in the sense of both protection against terrorism and protection against the backdoor being exploited) (Haynes, Reference Haynes2017). Similarly, it is necessary to broaden the focus beyond IS, not just to other jihadist groups, but also to other forms of violent extremism. Extreme right-wing groups, for example, also have a significant online presence (O'Callaghan et al., Reference O'Callaghan2013). And, whilst steps have been taken to disrupt their presence on the surface and deep webs (e.g. Facebook's decision to ban Britain First from its platform (Nouri, Reference Nouri2018)), these groups also appear to be migrating to the Darknet (Deep Dot Web, Reference Deep2017).

4 Non-normativity and definitional clarity

For scholars of counter-terrorism legislation and policy, the potential for tension to exist between rule by law and rule of law will be familiar. There is a significant body of literature that examines the analogous notion that security and liberty should be balanced and, in particular, the suggestion that, if the (perceived) threat of terrorism increases, some measure of liberty should be sacrificed for the sake of enhanced security (Macdonald, Reference Macdonald2009a; Reference Macdonald2009b; Posner, Reference Posner2006; Posner and Vermeule, Reference Posner and Vermeule2007; Waldron, Reference Waldron2003; Zedner, Reference Zedner2005). Importantly, it has been pointed out that security and liberty are interdependent: whilst security may in some sense be regarded as a prerequisite for the enjoyment of liberties, in another sense, the protection of liberties is necessary to ensure security (against the power of the state) (Barak, Reference Barak2002). Moreover, as noted above, it is widely acknowledged that respect for human rights is an important component of an effective counter-terrorism strategy. So, whilst regulators must have regard to the whole social-media ecology in order to realise the objective of blocking terrorist content online, ensuring rule by law is only a necessary, and not a sufficient, condition for compliance with rule-of-law values. In this section and the one that follows, we accordingly examine two issues – the clarity with which social-media companies define terrorist content and the adequacy of the processes by which a user may appeal against an account suspension or the blocking or removal of content – that engage these wider rule-of-law values.

The challenges involved in defining ‘terrorism’, and the lack of an internationally agreed-upon definition, are well-documented (see e.g. Hardy and Williams, Reference Hardy and Williams2011; Saul, Reference Saul2006). Summed up by the slogan ‘One person's terrorist is another's freedom fighter’, perhaps the most controversial definitional issue is that of just cause. In his report on the UK's statutory definition of terrorism, the then-Independent Reviewer of Terrorism Legislation Lord Carlile recognised the initial attractiveness of a just-cause exception but concluded that it would create ‘real difficulties’ (Carlile, Reference Carlile2007, p. 44). Not only is there ‘far from total international or domestic agreement as to which regimes are/are not in breach of international humanitarian law’ (Carlile, Reference Carlile2007, p. 44), but ‘Those who opt for terror always believe their cause is just’ (Fletcher, Reference Fletcher2006, p. 906). According to Hodgson and Tadros, this results in a trilemma:

This issue arose for consideration in R. v. F.Footnote ¹⁰ In this case, the Court of Appeal upheld the conviction for terrorism offences of a Libyan man who was found in possession of information on explosives and notes on how to overthrow Colonel Gaddafi. He had fled to the UK in 2002 and been granted asylum in 2003, after members of his family and friends had allegedly been killed by Gaddafi's regime. Delivering the judgment of the Court, Sir Igor Judge P. stated: ‘the legislation does not exempt, nor make an exception, nor create a defence for, nor exculpate what some would describe as terrorism in a just cause … Terrorism is terrorism, whatever the motives of the perpetrators’ (para. [27]). Commenting on the judgment, Hodgson and Tadros argue that it amounts to a recognition that ‘the law must apply to all such cases because the court cannot adequately discriminate between legitimate and illegitimate uses of violence, acknowledging that there may be cases where this leads to serious injustice’ (Reference Hodgson and Tadros2013, p. 517).

The issue was revisited in R. v. Gul,Footnote ¹¹ in which the Supreme Court was asked to determine whether the UK's definition of terrorism contains an exception for military attacks by a non-state group against the armed forces of a state in the course of a non-international armed conflict. Whilst the definitions of some other countries do contain such an exemption, the Supreme Court held that the ‘natural, very wide’ wording of the UK's definition could not be read down so as to contain one (para. [38]). UK law is thus ‘impaled’ on the second horn of Hodgson and Tadros's (Reference Hodgson and Tadros2013, p. 517) trilemma.

Even without a just-cause exception, a pro-democracy activist may be undeterred: the paradigmatic example is Nelson Mandela. Yet one of the concerns that commentators have expressed about technological management is that it ‘might compromise the possibility of engaging in responsible moral citizenship’ (Brownsword, Reference Brownsword2015, p. 36; see also Rosenthal, Reference Rosenthal2011). Morozov explains:

The movement towards non-normative regulation could thus have significant repercussions for activists living in oppressive regimes. Gerbaudo's (Reference Gerbaudo2012) analysis of the role played by Internet communication and social media in contemporary activism argues that, against the backdrop of a crisis of public space, ‘social media have become emotional conduits for reconstructing a sense of togetherness among a spatially dispersed constituency, so as to facilitate its physical coming together in public space’ (p. 159). An example is the Facebook page ‘We Are All Khaled Said’ (KKS), which promoted the protests that led to the overthrow of the Mubarak regime in the Egyptian revolution of 2011 (Alaimo, Reference Alaimo2015). Gerbaudo (Reference Gerbaudo2012) explains that the KKS page was used to build a community of activists in a ‘Choreography of Assembly’ (p. 12). Choreographing consists of ‘the mediated “scene-setting” and “scripting” of people's physical assembling in public space’ (p. 40) and, in this particular instance, included the use of ‘feeder marches’ (p. 64) advertised on the KKS Facebook page and the advance publication of an online booklet explaining to people how they should behave during the protests.

However, the growing use of AI and increasing government pressure to block terrorist content means that today social media companies’ ‘policies and the architecture of their products will increasingly complicate collective action efforts’ (Youmans and York, Reference Youmans and York2012, pp. 325–326). A stark example is the removal from YouTube of thousands of videos documenting atrocities in Syria, following the introduction of new technology to automatically remove content that potentially breached YouTube's Community Guidelines (Browne, Reference Browne2017). These videos provided important evidence of human rights violations and some existed only on YouTube, since not all Syrian activists and media can afford an offline archive. The channel of the monitoring group Violation Documentation Center (VDC) – which housed over 32,000 videos of the conflict – was also taken down, although VDC was later able to regain access to its channel and videos (Kayyali and Althaibani, Reference Kayyali and Althaibani2017).

The Syria example demonstrates the importance of human review. Facebook, Twitter and YouTube each has processes by which users may appeal decisions to remove content. They also recognise that AI cannot always deduce ‘what supports terrorism and what does not … and algorithms are not yet as good as people when it comes to understanding this kind of content’ (Bickert and Fishman, Reference Bickert and Fishman2017). But, whilst a discretionary ‘safety valve’ (Roth, Reference Roth2016, p. 1287) may be both unavoidable and necessary, it is also important to be mindful of rule-of-law values. This is evident in discussions of the UK's statutory definition of terrorism. On the one hand, concerns have been expressed about the definition's (over-) breadth (Anderson, Reference Anderson2014); on the other hand, the UK government has highlighted the ‘complexity and fluidity of changing political events in the terrorism context and its ability to evolve and diversify at great speed’ – which, it is argued, means that a ‘flexible statutory framework’ is required in order to ‘ensure that our law enforcement and intelligence agencies can continue to disrupt and prosecute those who pose a threat to the public’ (HM Government, 2015, p. 8). Commenting on this, the Supreme Court in R. v. Gul stated:

Whilst there may be obvious differences between drafting legislation that criminalises particular conduct and terms of service that specify when content may be removed from social-media platforms, the concerns identified by the Supreme Court remain relevant. Publicly defining what will be deemed to constitute terrorist content restricts and guides the discretion of individual reviewers, reducing the risk of inconsistent – or even inappropriate – decision-making. And, even if there are already internal procedures or policiesFootnote ¹² – or other, extra-legal factors (such as institutional culture or practice) – that already limit how reviewers exercise this discretion in practice,Footnote ¹³ defining terrorism in the terms of service serves an important communicative function. It provides users with information needed to understand their rights and responsibilities when using the platform (Citron, Reference Citron2018). Here, the ‘principle of maximum certainty’ (Horder, Reference Horder2016, p. 85) is important; a definition that is unduly vague or ambiguous will not enable users to make informed decisions about their use of the platform, and also leaves open the possibility of ‘censorship creep’ (Citron, Reference Citron2018, p. 1050).

At present, YouTube's Community Guidelines state that the platform does not permit ‘terrorist organisations to use YouTube for any purpose’ and ‘strictly prohibits content related to terrorism’, but without defining the terms ‘terrorist organisation’ and ‘terrorism’. In terms of Hodgson and Tadros's trilemma, this leaves it uncertain which of the three approaches the company takes and – on the assumption that it does not seek to block content related to a just cause – leaves it unclear how it adjudicates which causes are just. By contrast, Facebook's Community Standards define a terrorist act as ‘a premeditated act of violence against persons or property carried out by a non-government actor to intimidate a civilian population, government or international organisation in order to achieve a political, religious or ideological aim’, whilst the Twitter Rules offer the following definition (of violent extremism):

These two definitions share some similarities and, as a result, raise some common issues, such as whether the reference to ‘violence’ is both too broad (it is not limited to serious violence) and too narrow (it would apparently not encompass a cyber terrorist attack that caused serious economic or environmental harm).Footnote ¹⁴ There are also several dissimilarities. Only Facebook's definition stipulates that the violence must have been premeditated, whilst only Twitter's definition raises the possibility of exceptions for groups that have reformed, are engaged in a peace process or have democratically elected representatives (though the discretion it retains in this regard is left undefined). Like the definitions of many national legislatures (Hardy and Williams, Reference Hardy and Williams2011), Facebook's definition contains an intention requirement (‘to intimidate a civilian population, government or international organisation’) and a motive requirement (‘political, religious or ideological aim’), whilst Twitter's definition instead asks whether the group identifies ‘as an extremist group’. The latter seems inadequate to meet the demands of the principle of maximum certainty, given the possibility of articulating more precise intention and motive requirements. Lastly, Twitter's definition requires that the violence that was employed or promoted targeted civilians, whereas Facebook's definition contains no such limitation.Footnote ¹⁵ The Twitter Rules thus take the first of Hodgson and Tadros's three approaches – excluding from its definition attacks on the state and its officials, with the concomitant risk of under-inclusivity – whereas, on the face of it, Facebook, like the UK government, takes the second approach, with no express exception made for just causes – leading, therefore, to either over-inclusivity and unfairness or the exercise of an unarticulated discretion.

It is also noteworthy that, for the purposes of the shared industry database of hashes, the companies have stated that the database will only contain hashes ‘of the most extreme and egregious terrorist images and videos’ (Facebook, 2016). This not only leaves terrorism undefined, which is significant given the companies’ differing approaches outlined above; it also adds another layer of ambiguity, since ‘what constitutes extreme and egregious terrorist content is unclear’ (Citron, Reference Citron2018, p. 1053).

Concocting a satisfactory definition of terrorism has been likened to the quest for the Holy Grail (Levitt, Reference Levitt1986). Whilst social-media companies cannot be expected to achieve the impossible, the rule of law requires that they explain to their users what content will be deemed terrorist and blocked as clearly as they are able. Looking to the existing definitions of national legislatures may provide some assistance, although many of these definitions have also been criticised on rule-of-law grounds (Hardy and Williams, Reference Hardy and Williams2011). Citron (Reference Citron2018) accordingly argues that, in the first instance, a multi-stakeholder group that includes both academics and human rights groups should be established to help the companies articulate more clearly what constitutes terrorist material, whilst, in an open letter to Facebook Chief Executive Officer Mark Zuckerberg, the current UN Special Rapporteur on the promotion and protection of human rights and fundamental freedoms while countering terrorism has recommended that Facebook adopt the model definition of terrorism devised by her predecessor (Ní Aoláin, Reference Aoláin F2018).

5 Prior restraint and technological due process

Machines work with data and code; they do not attribute meaning (Hildebrandt, Reference Hildebrandt2018). Inevitably, therefore, the algorithms that identify and block terrorist content cannot be expected to operate with 100 percent accuracy. Accordingly, Facebook, Twitter and YouTube each offers an appeals process. Facebook's appeal process has three steps: first, the user will receive a notification and be given the option to request additional review; a human reviewer will then conduct the review, hopefully within twenty-four hours; finally, if a mistake has been made, the user will be notified and the content restored (Bickert, Reference Bickert2018). In April 2018, Facebook announced that, for the first time, it was giving users the opportunity to appeal decisions to remove individual posts, photos or videos that had been deemed to violate its Community Standards; prior to this, users had only been able to appeal against the removal of a profile or page (Bickert, Reference Bickert2018). Twitter's most severe enforcement action is permanent account suspension. In this situation, the user may appeal through the platform interface or by filing a report, outlining the reasons why they believe they did not violate the Twitter Rules. If the suspension is deemed to be valid, the user will receive information on the policy that the account violated (Twitter, 2018b). Whilst YouTube reserves the right to terminate an account for a single case of severe abuse (such as predatory behaviour), for most violations of its Community Guidelines, it operates a three-strike policy. A user may appeal against both an account termination and the issuance of a strike. When a strike is issued, the user receives an e-mail and an alert in their channel settings. A strike remains in place for a period of three months. A strike may only be appealed once and, if the appeal is unsuccessful, the user will not be able to appeal a different strike for sixty days. If the appeal is successful, the strike will be removed and the video will normally be restored.Footnote ¹⁶ Three strikes within a three-month period result in account termination (YouTube, 2018).

The increasing resort to automated decision-making has led to concerns being expressed about ‘technological due process’ (Citron, Reference Citron2008). One specific issue in the current context is the impartiality of the human reviewers considering appeals against account suspension or the blocking/removal of content who may, for example, be inclined to defer to their company's language-understanding algorithms. An additional concern is whether sufficient information is available to users who wish to lodge an appeal. This is all the more important given that the blocking of content is a form of prior restraint and, as such, demands especial scrutiny.Footnote ¹⁷

The first type of information regards the AI that detects and blocks violating content. This resonates with wider calls for ‘algorithmic transparency’ (Chandler, Reference Chandler2017, p. 1024). Pasquale (Reference Pasquale2015) uses the analogy of a black box – that is, ‘a system whose workings are mysterious; we can observe its inputs and outputs, but we cannot tell how one becomes the other’ (p. 3) – and suggests that agencies ‘ought to be able to “look under the hood” of highly advanced technologies like … algorithms’ (p. 165) as a way to police such behaviour. This is all the more important since ‘“black box” processes, because of their mechanical appearance and apparently simple output, have a veneer of objectivity and certainty’ (Roth, Reference Roth2016, pp. 1269–1270). However, these calls for transparency face a number of difficulties. There may be commercial and proprietary objections to disclosure. There may also be concern that the information will be exploited: ‘The process for deciding which tax returns to audit, or whom to pull aside for secondary security screening at the airport, may need to be partly opaque to prevent tax cheats or terrorists from gaming the system’ (Kroll et al., Reference Kroll2017, p. 639). And, perhaps most fundamentally, ‘machine learning tends to create models that are so complex that … even the original programmers of the algorithm have little idea exactly how or why the generated model creates accurate predictions’ (Rich, Reference Rich2016, p. 886; see also Desai and Kroll, Reference Desai and Kroll2017).Footnote ¹⁸ Kroll et al. explain:

Moreover, whilst techniques do exist that offer some form of reason-giving, these do not provide reasons for individualised predictions, but instead operate on an algorithm-wide scale, describing how important an input variable was to the algorithm's accuracy during training across many individuals (Lehr and Ohm, Reference Lehr and Ohm2017).

Much of the wider literature about algorithmic transparency refers to the use of AI to generate future predictions, such as a person's creditworthiness (Citron and Pasquale, Reference Citron and Pasquale2014) or the risk of criminality (Joh, Reference Joh2016). In the current context, by contrast, AI is being used to detect violations of the platforms’ terms of service. Here, understanding how the alleged violation was detected is less important to the user wishing to lodge an appeal than understanding what violation is alleged to have occurred. This takes us back to the importance of the principle of maximum certainty; definitional clarity not only provides users with fair warning before they post content – it also provides them with the information needed to have an effective opportunity to challenge alleged violations. For example, a pro-democracy activist who wishes to challenge the removal of content may be hampered if they have no indication whether, and how, the platform distinguishes between just and unjust causes. Similarly, the Twitter Rules forbid ‘promoting terrorism’. This leaves it unclear whether the fact that content would be understood by others as promoting terrorism is sufficient to justify removing it or whether the promotion of terrorism must also have been the user's purpose (Macdonald and Lorenzo-Dus, Reference Macdonald, Lorenzo-Dus, Fuller and Finkelsteinforthcoming).Footnote ¹⁹ There is also a further danger here: in rule-of-law terms, congruence. Examinations of terrorism-related precursor offences have shown that there is a discrepancy within these offences between the offence definition and the wrong that is being targeted (Edwards, Reference Edwards2010; Tadros, Reference Tadros2007). As a result of these offences’ overly broad wording, which encompasses conduct that is not being targeted for prosecution, only some of those who fall within the offence definition will be prosecuted. Yet, at trial, the factors that led to the defendant being selected for prosecution will not be in issue; the question will instead be whether the requirements set out in the offence definition are satisfied. The effect is to deprive the trial court of the opportunity to adjudicate on the actions that the offence is targeting (Macdonald, Reference Macdonald, Jarvis and Lister2014). By analogy, if the wording of social-media platforms’ terms of service encompasses significantly more content than is in fact selected for removal, and as a result does not reflect the factors that cause the AI to block/remove content, then the effect will be to deprive human reviewers – whose task is to apply the terms of service – of the opportunity to adjudicate on the factors underlying the automated decision. Some form of reason-giving, of the type mentioned at the end of the previous paragraph, would go some way towards addressing this problem.

As well as the provision of sufficient information, an additional concern is the possibility of bias. Whilst describing an algorithm as biased might seem to perpetuate the ‘homunculus fallacy’ – ‘the belief that there is a little person inside the program who is making it work – who has good intentions or bad intentions, and who makes the program do good or bad things’ (Balkin, Reference Balkin2017, p. 1223) – it is important to recognise that ‘automated systems are not free of bias simply because they are executed by logical machines’ (Chandler, Reference Chandler2017, p. 1045). There are a number of reasons why an algorithm might impact different groups of individuals disparately, such as disadvantageous definition of the outcome variables and non-representative data collection (Lehr and Ohm, Reference Lehr and Ohm2017). For this reason, it is important to examine the actual outcomes of algorithmic decision-making (Chandler, Reference Chandler2017; Kim, Reference Kim2017). This is particularly important in the current context, given that claims of anti-Muslim prejudice are utilised by jihadist radicalisers who deploy an us vs. them discourse to Other the West (Lorenzo-Dus and Macdonald, Reference Lorenzo-Dus and Macdonald2018). Indeed, a study of IS Twitter activity found that suspension played an important role in community-building, with the majority of the accounts studied referring to Twitter's use of suspension as a specific tool to persecute Muslims (Pearson, Reference Pearson2017). Examples included the tweets ‘There has been a major Twitter purge targeting Muslims’ and ‘I got #Radicalised when me and my people started to get suspended on every social media platform’. This is not to suggest that the suspension of these accounts was not in accordance with the Twitter Rules; rather, it is to underline how important it is that the AI is equally effective at blocking and removing violating content relating to all forms of violent extremism.