2.1 Generative AI-related malicious interactions

Considering that criminal phenomena involving Generative AI systems have started to appear only in recent times, rather than verifying which specific criminal offences might apply in this context, the paper will highlight, from a criminological perspective, the categories of interactions and the areas of crime that might find, in theory, a connection with Generative AI. This approach will allow us to maintain a cross-border perspective, without needing to base the analysis on a specific criminal justice system (which will remain the next needed step in the future developments of this field of study).Footnote ¹ Without presuming to be exhaustive, when there is a human agent who participates with criminal intent, the following malicious interactions with Generative AI can be outlined:

1. (a) Support to realize content-related criminal offences, by intentionally producing or disseminating artificially generated illegal contents.

2. (b) Support to realize other crimes, by aiding or facilitating criminal activities.

Generative AI can be used or designed to create harmful contents, whose creation itself, or whose dissemination might be criminalized in most criminal justice systems. The range of contents that can be created is significantly wide. It includes terroristic contents, image-based sexual abuse, child pornography, and harmful speech, which can be relevant under criminal law, for instance, in case of some categories of discrimination-based discourse, incitement to crime, defamation (Henderson et al., Reference Henderson, Hashimoto and Lemley2023; Ousidhoum et al., Reference Ousidhoum2021; Weidinger et al., Reference Weidinger2022). Taking as a case-study a well-known chat-based LLM, GPT-4, OpenAI published a report on the safety challenges presented by the model, where they described the results of their adversarial testing processes, called “red teaming.”Footnote ² The list of harmful contents that might be generated by LLMs include: advice or encouragement for self-harm behaviors; graphic material such as erotic or violent content;Footnote ³ harassing, demeaning, hateful content; apology or incitement to terroristic activities (Sivaram et al., Reference Sivaram2023; Weidinger et al., Reference Weidinger2022).

Furthermore, fake artificially generated content is raising significant concerns in the current debate (Chesney & Citron, Reference Chesney and Citron2019), even though it can be based on a different sub-category of Generative AI, called generative adversarial networks, invented by Google researcher Ian Goodfellow (Goodfellow et al., Reference Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville and Bengio2014). In this area, a primary threat regards the so-called deep nudes, namely the use of deep fakes to perpetrate crimes of non-consensual pornography (Franks & Waldman, Reference Franks and Waldman2019). Although deep nudes are undoubtedly a criminal law concern, the broader category of deep fakes outlines a more problematic area from a criminal law standpoint, since it is questionable whether and to what extent behaviors involving disinformation campaigns and propaganda should be criminalized, since such criminal policies can compromise free speech rights.

In the area of artificial content-related offences, distinguishing between private and public harms is pivotal in assessing the criminal relevance of Generative AI content. While private harms – such as individual defamation or non-consensual dissemination of intimate content – raise significant ethical and legal concerns, which grant a sounder legitimation for criminal intervention, public harm must typically involve a higher threshold of risk to society at large. For crimes tied to terrorism or organized crime, the focus is on clear and severe threats, whereas offenses related to freedom of expression or democratic integrity, like electoral interference or incitement to violence, demand a more nuanced approach. Criminalization in these cases is generally limited to actions that demonstrate a significant probability of translating danger into tangible damage, emphasizing the need for scale, coordination and clear harm to the public (Guerini, Reference Guerini2020). As Generative AI continues to evolve, establishing robust frameworks to assess both the technological misuse and its societal ramifications will be integral to balancing innovation with legal accountability.

Generative AI systems, however, enable these crimes to be committed in ways that go beyond the mere creation of false or violent content. More harmful phenomena might be realized, for instance, through manipulative schemes that amplify the severity of the harm. LLMs might become a part of online manipulation schemes (Susser et al., Reference Susser, Roessler and Nissenbaum2019), which can be realized not only through the generation of fake contents, but also through several operations (e.g., spreading of polarized contents). Generative AI systems can be used for disinformation campaigns at scale (Chesney & Citron, Reference Chesney and Citron2019; Kreps et al., Reference Kreps, McCain and Brundage2022), and, more broadly, for covert or deceptive efforts to influence the opinions of a target audience, defined as “influence operations” (Goldstein et al., Reference Goldstein, Sastry, Musser, DiResta, Gentzel and Sedova2023), able to cause manipulative effects, such as cause alteration in the electoral and democratic process, or exploit cognitive vulnerabilities of users (DiPaola & Calo, Reference DiPaola and Calo2024; Neuwirth, Reference Neuwirth2022).

This leads to the second area of intersection between criminal law and malicious interactions with Generative AI, which involves its aiding role as a supporter or facilitator of crime. Considering the GPT-4 system card previously mentioned, two different forms of aid might surface: (a) aid in the preparatory phase of a crime, through the gathering of contents useful for planning a crime (e.g., information for developing unconventional weapons) or through the gathering of instructions for finding illegal content; (b) aid in the execution phase of a crime (e.g., LLMs can be used to find alternative, purchasable chemicals used in criminal activities). A relevant area of interest might be the area of cybercrimes. LLMs can be used to realize or improve social engineering attacks (e.g., drafting phishing emails) or hacking attacks (e.g., explaining systems or code’s vulnerabilities).

The same threats are outlined by Europol in a recent report (Europol, Reference Europol2023), which states how LLMs can be used to learn about crime, ranging from how to break into a home, to cybercrime and child sexual abuse.Footnote ⁴ A specific concern remains the fight against terroristic attacks, since the model can be used to generally gather more information that may facilitate terrorist activities, such as terrorism financing or anonymous file sharing.

The malicious interactions with Generative AI will determine a direct criminal responsibility of the human agents who committed purposely the crime, either by designing the system to realize one of the activities mentioned or by using it to the same end. Therefore, in terms of responsibility attribution, these cases are less challenging. Although some gray areas remain – especially regarding theories of criminalization (Hörnle, Reference Hörnle, Dubber and Hörnle2015) – legal systems may vary in qualifying malicious interactions either as new, potentially aggravated, ways of committing existing criminal offenses or as new criminal behaviors that require criminalization (Miró-Llinares et al., Reference Miró-Llinares, Duvac and Toader2024).

Beyond the issue of distinguishing between harm to individual victims and harm to public interests, another issue concerns the distinction between the behaviors of creation versus distribution of content. Defining the threshold for criminally relevant behavior is essential in determining when the creation or dissemination of content crosses into punishable territory. Notably, in the European Union (EU), only the creation of child pornography is criminalized in itself (even in the case of artificial or virtual content),Footnote ⁵ setting a precedent where mere content creation is rarely deemed illicit unless it pertains to certain severe offenses.Footnote ⁶ Most other content-related offenses, such as terroristic propaganda or non-consensual pornography, require dissemination to meet the threshold of criminal liability, as these materials are not inherently illicit outside of their distribution and impact on the public sphere.

In dealing with Generative AI, criminal policies might consider, on the one hand, to extend criminalization. For instance, there could be arguments in favor of the responsibility of those actors who purposefully design and make available online Generative AI platforms that enable users to realize criminal activities (e.g., platforms to create non-consensual deep nudes). In these cases, where the creation of such platforms is not criminalized itself, the crime would not be directly realized by the providers, but they might be held responsible as accomplice in the crime committed by their users.Footnote ⁷

On the other hand, to avoid overcriminalization, Courts might interpret existing criminal offenses in an evolutionary manner, so as to include within their scope new forms of conduct involving the abuse or misuse of Generative AI systems. For instance, under certain circumstances, the act of creating terrorist propaganda using Generative AI could be interpreted as a criminal offense if it aligns closely with national laws defining terrorism-related crimes.Footnote ⁸ However, this requires a case-by-case assessment grounded in the specific offences’ wording provided by each jurisdiction, but it constitutes a more balanced approach, rather than a blanket criminalization of AI-assisted content creation. Similarly, while the use of Generative AI to gather intelligence or to support criminal activities does not constitute a standalone offense, it could be relevant as an aggravating factor, amplifying existing crime’s impact and warranting more severe penalties when technology is misused in such contexts.

In addition to defining criminal thresholds, there is indeed a growing need to address the deterrence of misuse in Generative AI. Protecting the confidentiality, integrity and availability of AI is essential (Flor, Reference Flor, Magagna and Magagna2024), but equally crucial is safeguarding public trust in the technology as it integrates more deeply into society. Generative AI, with its capacity for autonomous content creation, could become a “social actor” that influences public perception and societal norms. Therefore, misuse in the form of large-scale misinformation campaigns, recruitment by terroristic organizations, or criminal campaigns aimed at democratic disruption warrants close scrutiny. The threshold here shifts to focus not only on the quantity of potential harm but also on its nature, which might contribute to the emerging of new interests, such as the public interest to protect the trustworthiness of the interactions with Generative AI.

2.2 Generative AI-related dangerous interactions

The picture becomes very different when considering dangerous interactions and the positions of the different actors involved in the supply chain of a Generative AI system that is lawfully put online or on the market. The umbrella term “operator,” keeping as a reference the definitions provided by the EU AI Act (AIA), includes different categories of actors. Namely, operators are considered as follows: “providers,” those that develop, place on the market and put into service an AI system; “deployers,” those that use an AI system under their authority in a professional activity; “distributors,” all the other remaining actors that participate in the supply chain of an AI system and make it available on the market.Footnote ⁹

From the perspective of these actors, Generative AI-related harms are unintentional, and the interactions are no longer considered malicious, but dangerous. In other words, they present the probability of the occurrence of severe harms.Footnote ¹⁰ Unintentional harms awaken the preventive function of criminal law. The legal response to unintentional harm does not end in punishing the agent that caused the harm, but it widens to providing the means to avoid the harm, on whose violation the responsibility attribution is based. However, these are unstable grounds for criminal law. The need to reassure against the insecurity created by the risks of harm might create a rise in punitiveness and an expansion of the penal state, which clashes with the legitimacy of criminal law as a modern, liberal institution (Ashworth, Reference Ashworth2013; Carvalho, Reference Carvalho2017; Donini & Pavarini, Reference Donini and Pavarini2011). Therefore, we need to stay on the fine line between avoiding broad new criminalization to prevent unintentional harms and filling gaps in the attribution of responsibility for those harms.

In their hybrid forms, theories of punishment combine retributivism and prevention. However, in the field of unintentional harm, the risk is to lean too much on prevention, where the goal shifts from avoiding the commission of crime to assuring social order. Security politics have problematically promoted in the last decades the framing of crime as a social risk and criminal law as a means of risk management (Ashworth & Zedner, Reference Ashworth and Zedner2012; Ashworth, et. al., Reference Ashworth, Zedner and Tomlin2013; Husak, Reference Husak2007). This approach, while influential, encounters significant challenges, particularly when extended into the realm of technology. The modern application of this model of punishment threats overextending its scope, as it must contend with the compounded uncertainties of complex social systems and the rapid evolution of technology as a social reality.

Sociological studies already outlined the limits of the paradigm of “control” when it comes to technology. The responsibility gaps created by technology developments are the result of an old issue, defined as the “dilemma of control,” that society faces when attempting to predict the social consequences of a technology (Collingridge, Reference Collingridge1980). The control of technology has proven to be a very difficult task because of its unanticipated social consequences. Narrowing our focus from the broadly harmful social consequences that a technology might have on society to the consequences that can be qualified as specific harms to a person or to a public interest, the dilemma of control might be addressed using a risk-based approach.

Therefore, assessing the necessity of criminal policies targeting AI-related harms depends on the existence of an objective and collective interest to follow this path (Vassalli, Reference Vassalli1942), which might be, as highlighted, the trustworthiness of AI as a human value. Such policies should require a shift beyond general or negative prevention toward positive prevention, focusing on restoration rather than mere deterrence. Here, culpability emerges not simply as a matter of blame (Kelly, Reference Kelly2018) but as a normative construct which can bring to restorative measures, suggesting a framework for criminal law that emphasizes social co-responsibility in the age of AI.

To establish whether such involvement of criminal policies might find a legitimate space, the objective nature, scope and severeness of AI-related harms must be firstly defined, especially when there are unintentional harms. Although a current analysis must contend with the phenomenon’s young age, that is, the fact that Generative AI systems have only recently been released into society, a framework of dangerous interactions can be suggested.

The paper approximately summarizes dangerous interactions into three main groups:

1. (a) Risks of harms related to users’ misuse.

2. (b) Risks of harms related to artificial emergence.

3. (c) Risks of harms related to overreliance.

The first group corresponds to the misuses listed in the previous paragraph, which become risks from the Generative AI operators’ perspective, whereas the last two groups create the more complex scenarios, where Generative AI-related harms cannot be traced back to a human agent who interacted with criminal intent. Society might face, on the one hand, harms directly caused by the Generative AI system without any user behind it (harmful AI content), and, on the other hand, harms that the Generative AI system contributed to causing with the participation of a human agent who acted without criminal intent (negligent AI content).

Harmful AI contents might be generated by emergent interactions, where “emergence” states the ability of AI to perform unpredictable behaviors (Calo, Reference Calo2015, p. 538). For instance, harmful content might be the result of the hallucinations of LLMs (Gehman, Reference Gehman, Cohn, He and Liu2020). There have already been cases where the LLMs fabricated offensive false claims against a person, falsely accusing him of a crime (Volokh, Reference Volokh2023), or cases where LLMs “degenerate into offensive language from seemingly innocuous prompts” (Weidinger et al., Reference Weidinger2022, p. 216). Furthermore, in cases of Generative AI implemented beyond the task of content-generation, new areas of risks might rise when the model is set to “accomplish goals which may not have been concretely specified or appeared in the training” (GPT-4 System Card, 2023, p. 15).

Closer to the present might be cases of overreliance of users on negligent AI content, where the harm is the result of a dangerous interaction between the Generative AI and an innocent or negligent user. Since Generative AI systems might provide part of the knowledge used in decision-making processes, if it provides manipulative information, it might encourage people to commit self-harm or it might violate their self-determination (Atillah, Reference Atillah2023, March 31; Perrigo, Reference Perrigo2023, February 17), whereas if it provides neutral, but false, misleading, or poor-quality information to a user who acts consequently based on that information, the result of the interaction might cause harm to the user herself or to others (Bambauer, Reference Bambauer2023).Footnote ¹¹ For instance, Generative AI is used to design proteins and inform patients in the healthcare sector (AlQuraishi, Reference AlQuraishi2021; Chakraborty et al., Reference Chakraborty, Hrithik, Sayani, Saroj Kumar, Ankit, Kamred Udham and Mohd Asif2022), which might cause injuries due to the health professionals’ overreliance on the AI outputs.

Each dangerous interaction might raise different modes of responsibility attribution. In scenario (a), where there is an interaction with a malicious user, the possibility to resort to doctrines of complicity (Stewart, Reference Stewart, Dubber and Hörnle2015) might become relevant, holding the operators of the Generative AI responsible for participating in the users’ crimes, whether there is an omissive contribution (i.e., failure to avoid the user’s crime) or an active contribution (e.g., development or putting into service of the Generative AI through which the user committed the crime).Footnote ¹²

In scenarios (b) and (c), where malicious users’ interactions lack, the operator might be directly responsible for the harm realized, whether there is an omissive interaction (i.e., failure to avoid the harm) or an active interaction (e.g., development or putting into service of the Generative AI which caused the harm). However, all these cases reveal several shortcomings, as the next sections explore.

3.1 The natural person behind the AI: Closed door

The traditional allocation of guilt in facing AI-related harms is problematic. Scholars analyzing criminal culpability in the AI field, although mostly focused on robotics (Consulich, Reference Consulich2022; Gless et al., Reference Gless, Silverman and Weigend2016; Piergallini, Reference Piergallini2020; Simmler & Markwalder, Reference Simmler and Markwalder2019), outline several collision points related to multiple factors, such as the distance between human behaviors and their results, the failures of causation and the challenged foreseeability of harm. The control of the users and of the operators who stand behind the technology is limited and compromised by the shared-agency dimension. That is if we follow the traditional perspective that devises responsibility attribution starting from the harm instead of from the interactions.

These collision points are not a prerogative of the field of AI-related harms. They are linked to a well-known phenomenon. The use of preventive criminal law to mitigate the risks of our contemporary and complex society is a feature of a longstanding debate related to whether and to what extent criminal law can be used to regulate the harms generated by the risk society (Stortoni & Foffani, Reference Stortoni and Foffani2004), which is now the algorithmic society. The analysis of the broader evolution of criminal law in the risk society provides a lens through which we can examine the inadequacy of the traditional punitive paradigm when applied to risks associated with Generative AI. Specifically, the conventional model of person-action-harm fails to adequately address the complexities inherent in situations where harms arise from systemic processes rather than actions of individual human agents. In other words, assigning direct personal responsibility to a singular human agent for harms resulting from systemic risks poses significant challenges, particularly concerning questions of culpability. Therefore, to regulate the harms related to the risks of the algorithmic society, it is not enough to consider the behavior of the single human agent alone. In these cases, guilt will likely close the door, limiting criminal law only to the punishment of intentional criminal behavior.

Alternatively, it may be necessary to analyze the roots and conditions of the interaction. By disentangling the behaviors of Generative AI operators from the interactions between AI systems and users, scholars (Consulich, Reference Consulich2022; Fiorinelli, Reference Fiorinelli2023; Fragrasso, Reference Fragrasso2024) suggest the possibility of employing criminal law to address novel forms of omission, such as intentional or reckless omission to comply with prescribed safety measures (i.e., failure to comply with specific obligations, such as non-delegable duties, transparency obligations or failure to provide the competent authorities with the required information). However, in addition to the concerns that arise from the use of preventive criminal law, this approach might have collateral effects, disincentivizing progress and investment, as operators might restrict their activities out of fear of facing severe punishments. It would also need to coordinate with the administrative sanctions that might already be regulated in case of violation of legal obligations, for instance by EU law.Footnote ¹⁶

Furthermore, the scope and content of the standard of due care in the field of Generative AI have not yet been established, which will depend on the set of safety international standards required by the industry (Ebers, Reference Ebers, DiMatteo, Poncibò and Cannarsa2022), since best practices will likely be insufficient. Resorting only to self-regulation does not appear to be a sufficient alternative, since delegating altogether the process of selecting the policies and measures for harm prevention to the private sector might rise concerns related to legal certainty. Moreover, certain measures (e.g., labeling unlawful content into pre-established categories) might be more efficient if based on an active collaboration with the public authorities.

3.2 Generative AI and criminal law: Whether to open the door

While the door is open for using criminal law to address malicious behavior, this paper has highlighted various setbacks to using criminal law for dangerous behavior. Given this scenario, the question then becomes whether the context of dangerous interactions might need criminal law.

Excluding criminal law completely from the set of policies related to dangerous interactions is an inadequate long-term strategy. Among the multiple theories on the purposes of criminal law (Hörnle, Reference Hörnle, Dubber and Hörnle2015), when viewed through the lens of the harm principle or the theory of the legal goods (Rechtsgüterschutz),Footnote ¹⁷ the paper outlined that it becomes essential to assess the nature of harms and risks inherent in regulating Generative AI. These include not only the harms originating from the diffusion of harmful and illicit contents artificially generated by malicious users, but they have a broader scope, which encompass the risks related to the delegation of content-generation to artificial agents in the information society. The ability to generate information in our contemporary society can have significant effects not only on individuals but also on society. This creates a base of legitimacy for criminal law, if limited only to the most serious wrongdoings and the most serious harms (in compliance with the ultima ratio principle). As the paper attempted to demonstrate, engaging in information-generating activities with artificial agents in a shared-agency context can lead to the emergence of social conflicts, manifested in the form of dangerous interactions, which include (it is worth remembering) also the risk of malicious interactions. The normative regulation and protection of a sector of social life (De Francesco, Reference De Francesco2004) highly relevant as the generation of information may ultimately necessitate criminal protection involvement in the long term. However, the condition for such involvement is that social conflicts translate into substantial wrongdoings. The question at issue then becomes whether Generative AI deployments can indeed result in substantial wrongdoing. Although this possibility exists, given that Generative AI opens the possibility for causing widespread and diffuse damage, as well as for facilitating the realization of criminal activities by simplifying their preparation or execution, it is premature to provide a definitive answer. Nevertheless, the question must be posed, and it will require finding an answer by examining the evolution of the sector and the framework of policies aimed at regulating it with an open-ended perspective, which might resort to the blurring boundaries between criminal punishment and administrative or regulatory offences (Duff et al., Reference Duff2010;Fletcher, Reference Fletcher2008).

At this stage, the paper suggests that, in the context of Generative AI and dangerous interactions, following a completely non-criminal approach, which involves civil litigation and administrative regulation, may prove insufficient in the long run. This approach would narrow the legal response to the perspective of harm mitigation, whereas criminal law aims to establish an institutional normative order aimed at protecting social values. The function of censuring and discouraging wrongdoing, which implies “should not do,” entails state intervention into individual freedoms that extends beyond providing a remedy to the victim for the invasion of protected interests (Ashworth, Reference Ashworth2013). The punitive and preventive functions, therefore, require that the distribution of responsibility in cases of serious harms relevant under criminal law follows its stricter standards, both substantively and procedurally, while granting certain protections for the defendant. In the case of AI Generative-related risks, especially in the case of emergence and overreliance, a non-criminal approach may lead to a situation where a crime occurs, and criminal law falls short because it has become challenging to find the guilty mind. However, the result of interactions remains a crime, and possibly, when behaviors become interactions within a system, a “guilty system” can be identified. According to this perspective, the attribution of responsibility must not address a gap but rather a shift (Simmler, Reference Simmler2023).

In addition, there are pragmatic considerations to ponder. Generative AI holds the potential to either facilitate or contribute to relevant harms, which may manifest with increased rapidity upon deployment of artificial agents, challenging investigation activities. Relying solely on retrospective measures to prosecute malicious uses may prove ineffective or unsustainable. The prospect of intervening during stages that precede interactions (e.g., design or online availability) or during interactions themselves (e.g., deployment) presents an opportunity to address social conflicts as they arise, preempting their escalation into broader and diffuse harms. Involvement during these phases could equip criminal law with valuable tools for regulating conflicts. For instance, civil courts lack the authority to order defendants to perform positive acts to avoid or to mitigate punishment, such as implementing reactive compliance efforts. Lastly, given the diffuse nature of risks associated with Generative AI, it prompts inquiry into the adequacy of entrusting the legal response to harms solely to private citizen initiatives, without a structured intervention of the State, especially in cases where there is no victim, and the harm caused is against collective and public interests.