1. IntroductionFootnote 1
In considering artificial intelligence (AI) in resort-to-force decision-making, we start by acknowledging that AI is not new. AI grew out of work that traces its inspiration in part from the World Wars (Russell & Norvig, Reference Russell and Norvig2009), when advances towards digital computing and wartime efforts to achieve asymmetric advantage converged (Slayton, Reference Slayton2013). Many of the most impressive current AI systems are built on an idea first published by McCulloch and Pitts (Reference McCulloch and Pitts1943). AI and its impact on (inter)national security is nonetheless a relevant, present, and rapidly evolving concern (see, e.g. Johnson, Reference Johnson2019; Wong et al., Reference Wong, Yurchak, Button, Frank, Laird, Osoba, Steeb, Harris and Bae2020). Advances in computational hardware, software, and computer science have rendered AI-enabled systems more useful than was previously possible, but their diverse technical manifestations, AI model ownership structures, and shifting regulatory environments (amongst other complexities) are contributing significantly to the challenges of managing their use (Cath, Reference Cath2018).
Current discussions about AI in resort-to-force contexts parallel those previously raised around software integration into missile defence systems (see, for instance, Slayton, Reference Slayton2013) – in particular, potential impacts on the time decision makers may have to consider evidence and decide to resort to force or not in response to a perceived threat (Zala, Reference Zala2024). One may ask: is there really a need for new approaches for considering the impact of AI on resort-to-force decision making? We believe the answer is yes – particularly given the increasing capacity of AI-enabled systems, and the ubiquity with which such systems are being created, used, and repurposed in both civilian and defence contexts. These factors shape trust dynamics for humans making use of – or being informed by – such systems (Lee & See, Reference Lee and See2004). If, as Le Coze and Antonsen (Reference Le Coze and Antonsen2023) note, “The key to understanding the system dynamics that are involved in the production of unwanted outcomes (how the system ‘works’ in different situational contexts) lies in understanding how the system components may influence and shape each other,” AI in its currently prevalent forms prompts the question: How do we prevent unwanted outcomes (as defined by stakeholder assessment of the realised actions of the system) if we cannot easily discern the components included in an AI system – much less the way such components interact to shape an outcome? The scenario we envision here is not one of a single leader declaring war based on a declarative message delivered by an AI system. Instead, we consider the subtle influence data-informed decision-support systems (DSS) may have on humans involved in such dynamics. These acts of influence are harder to trace, but we must consider them carefully, particularly when the deployment of nuclear arsenals – and the subsequent aftermath triggered by that act – are a plausible consequence.
We identify three dimensions of complexity that current developments in AI are likely to add to (or amplify) in resort-to-force decision making in the coming years. We then explore potential adverse impacts such dimensions may add and propose recommendations we believe will help mitigate some of these impacts. Our focus – both in the exploration of adverse impacts, and in our recommendations for mitigating them – draws on our systems engineering perspective to explore harm identification and prevention, which we associate with the term safety. Given the vast literature linked to this term, we discuss our definition in the following.
In the next section, we provide a brief overview of recent advances in AI and relevant factors contributing to their complexity.
1.1. The complexity of current AI systems
Given conflicting definitions of AI in the literature (Russell & Norvig, Reference Russell and Norvig2009), we narrow our discussion to machine-learning and transformer models, which are prevalent in applications of AI. We provide a brief overview of such models here.
Machine-learning and transformer models are built on artificial neural networks, an approach first used by McCulloch and Pitts (Reference McCulloch and Pitts1943) to describe cognitive activity through logic. Artificial neural networks conceptually mimic the function of a biological neural network, but in practice are a mathematical model that can be adjusted via a learning process to produce predictive responses (known as “inferences”) to a given input or set of inputs. They are probabilistic in nature, which means their outputs are shaped by statistical predictions rather than deterministic logic.
The type of feedback used in the learning process is a helpful categorisation to make when discussing such models and, for this, there are three key categories of learning: supervised, unsupervised, and reinforcement. For supervised learning, humans label the training data, and the model is automatically tuned to reproduce the human-provided labels. For unsupervised learning, no label is offered; instead, the model identifies patterns (e.g. clusters) in the data. Human input may be required to make sense of the detected patterns. For reinforcement learning, a reward and loss approach is used to process data through what is known as a loss function. The loss function is a quantitative expression of the goal the model maker wants their creation to fulfil. If the loss function is designed well, large loss values suggest the model is not performing as desired, prompting significant changes in future responses, while smaller loss values serve as a signal that the model’s performance is on track. Human input can also be used as feedback. In many cases, a mix of learning processes may be used to “train” a model. Once trained, a model may undergo further refinement using such methods to perform well for a particular context (a process known as “fine-tuning”).
More recently, “transformer” models have been developed and used (Vaswani et al., Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017) to create “foundation” models. Such models are very large, trained on significant amounts of data, and are designed to be useful for a wide range of tasks. They are trained through “self-supervised” learning, which we will describe using a conceptual example. In our example, the training data set consists of English language sentences. Self-supervised learning makes the word order in each sentence an important part of the dataset (e.g. “I am free” becomes “{I,1}, {am,2}, {free,3}.”). Word order then becomes a pattern the model can analyse and optimise for (using a loss function once again). This approach works well if one has access to a large and sufficiently diverse quantity of training data, which is often scraped from large openly accessible datasets (e.g. the internet) (Metz, Kang, Frenkel, Thompson & Grant, Reference Metz, Kang, Frenkel, Thompson and Grant2024).
These foundation models – which are much better at making accurate predictions on unseen data than their predecessors – can then be fine-tuned with relatively small amounts of data to work well for new applications. The development of transformer models – which is driving generative AI – began in natural language processing and has since been extended to other domains (Child, Gray, Radford & Sutskever, Reference Child, Gray, Radford and Sutskever2019).
These models are shaped by human decision making and action, including but not limited to: the collection, selection, and use of training data; the design of the training process; the design and refinement of the inference process; the way the model outputs are communicated; and the way the model gets used, updated, or refined. The impacts such choices can have on AI models are complex, and – when investigated – are often found to reflect, sustain, and even amplify existing social inequities, as explorations of data labelling practices have shown in recent years (Paullada, Raji, Bender, Denton & Hanna, Reference Paullada, Raji, Bender, Denton and Hanna2021). This does not suggest a lack of care (though that may be the case); rather than cast judgment here, we simply present this as one of the inherent challenges of such systems. At various points in any AI model pipeline, humans may be forced to classify data categories or model outputs into discrete categories even when their choices are highly contestable and tied to a given social, cultural, or political context (D’Ignazio & Klein, Reference D’Ignazio and Klein2020).
Ultimately, AI models are systems – collections of elements that are interconnected in a coherently organised way to achieve a goal (Meadows & Wright, Reference Meadows and Wright2011). Where there are multiple elements interacting with and adapting to one another, a complex system emerges (Arthur, Reference Arthur1999; Rind, Reference Rind1999). AI-enabled systems – consisting of a mix of AI models and other components and often designed with adaptive or learning components – tend to be complex. Here, the term complex refers to a system of many parts, interacting in multiple ways and culminating in emergent properties. Emergent properties are aptly described by Whitchurch and Constantine (Reference Whitchurch, Constantine, Boss, Doherty and LaRossa1993):
Wholeness is characteristic of systems because there are properties or behaviours of the system that do not derive from the component parts themselves when considered in isolation. Rather, these emerge from their specific arrangement in a particular system and from the transactions among parts made possible only by that arrangement. These are called emergents or emergent properties because they emerge only at the systemic level.
In other words, emergence refers to properties or behaviours that individual elements of the system do not demonstrate on their own. The interconnections and interactions with other elements of the system generate emergence. What is important to note is that complexity in itself is an emergent property (Ladyman, Lambert & Wiesner, Reference Ladyman, Lambert and Wiesner2013).
Predicting emergent behaviours for such systems can be difficult – if not impossible. In the military context, real-world verification and validation are a challenge. Realistic “tests” of an AI-enabled system’s likely behaviour in real-world contexts may not be possible until decisions are made, and the consequences are realised.
Of particular concern is the impact a DSS may have on deterrence and escalation dynamics. As Wong et al. (Reference Wong, Yurchak, Button, Frank, Laird, Osoba, Steeb, Harris and Bae2020) found, the introduction of AI and autonomous systems in military contexts may lead to changes in such dynamics. In their wargaming study focussed on deterrence, AI-enabled systems were found to be poor at understanding human signals of deterrence, resulting in greater likelihoods of conflict escalation. There are also studies that suggest AI human-machine interactions can negatively shape military decision making because of factors such as miscalibrated trust (in which human expectations of a system’s capabilities do not match up with reality, leading to over- or under-use), and inappropriate biases within AI systems (Gilli, Reference Gilli2019).
Resort-to-force decision making sits at the precipice between deterrence and conflict escalation. The introduction of AI-enabled tools in resort-to-force decision making, e.g. AI-enabled DSS, may destabilise this tipping point of warfare.
In this work, we assume that an unnecessary resort to force is an undesirable outcome that must be anticipated and mitigated, and associate the mitigation of such outcomes with the term “safety” – as clarified in the next section.
1.2. A focus on safety
Safety is a word that has many interpretations. Given this, we start our discussion of the term with the safety engineering literature because of its importance in shaping how safety is managed in complex sectors such as defence (see, e.g. Dwyer, Reference Dwyer1992). Traditionally, safety engineering has pertained to the minimisation of harm across the entire lifecycle of a system (Roland & Moriarty, Reference Roland and Moriarty1990). Prevention of harm is achieved through a knowledge of what potential harms are and the use of appropriate measures to mitigate their occurrence, before a system is allowed to operate.
The challenge of AI-enabled systems is complexity, an emergent property that makes the prediction of harms less straightforward. There are also many potential harms to consider (see, for example, AI harm taxonomies in Shelby et al., Reference Shelby, Rismani, Henne, Moon, Rostamzadeh, Nicholas, Yilla-Akbari, Gallegos, Smart, Garcia and Virk2023), some of which can only be understood and assessed in the broader social context in which a model is being used. A small subset of these harms would normally be considered in traditional safety engineering practice, though the expanding use of AI in safety-critical applications is prompting change (Habli & McDermid, Reference Habli and McDermid2024). In the “AI safety” field – to the extent that it can be defined – a similarly narrow approach exists in some communities (Salhab, Ameyed, Jaafar & Mcheick, Reference Salhab, Ameyed, Jaafar and Mcheick2024).
Given the hypothetical nature of this exploration, we prefer an inclusive approach. We therefore adopt the following definition: safety is “the freedom from harm or other undesirable outcomes” (Leveson, Reference Leveson2023, p. 43). The key questions our chosen definition of safety should raise are as follows: who or what decides if something is harmful or undesirable, or if harms and negative outcomes have been avoided? And how do harms, undesirable outcomes, risks, or risk mitigations evolve with time and circumstance? These questions may generate diverse answers. Safety is therefore contextual – something to consider as we discuss the connection between different aspects of complexity of AI-enabled systems in this context, and their impacts on safety. In the context we are discussing in this work, many definitions of safety may be relevant, and indeed, a leader involved in a resort-to-force decision may need to make careful decisions about what aspects of safety to support (and how to distribute such support), as well as what harms to impose to achieve a strategic goal. Our chosen definition of safety is meant to serve as a means of analysing the potential impacts AI-enabled technologies may have in different contexts, and to different individuals or communities (each of which may have their own perceptions of what constitutes “harm” or “undesirable outcome”). Such an analysis could then be used to interrogate and adapt existing national security decision-making processes to a world in which AI-enabled systems are both triggering rapid change and constantly evolving in their own right.
With this in mind, let us return to our discussion of AI use in the resort to force by state actors.
1.3. AI and the resort to force
When considering the possible role of AI in the resort to force, it helps to consider the range of capabilities, roles, and properties relevant AI-enabled systems may have. Such systems may be diverse, may be designed to augment human decision making and/or replace/augment human action (e.g. surveillance or intelligence gathering) in whole or in part, and may also be distributed virtually and physically in space through sensor networks, cloud computing, physical or wireless networking infrastructure, and so on. Such systems may be integrated into or provide data for one or more DSS – software-enabled systems that generate outputs designed to inform human decision making (Larson & Hayes, Reference Larson and Hayes2005). An AI-enabled DSS is designed to collect and analyse large amounts of data over short periods of time to generate outputs to inform decision making (Cartolovni, Tomicic & Mosler, Reference Cartolovni, Tomicic and Mosler2022).
Since AI-enabled DSS are not (to our knowledge) currently used in resort-to-force decision-making (except indirectly through intelligence collection and analysis), we use a hypothetical example to explore the potential implications such a system may pose. Our example draws on historical resort-to-force scenarios – the Cuban Missile Crisis and the Petrov incident, both described in Zala’s work (Zala, Reference Zala2024). Key elements used to shape resort-to-force decision making in these events include the capacity to detect small signs of conflict and signal loss (Cuban Missile Crisis), and a system that warns that an adversary may be initiating a first-strike action (Petrov incident). We also draw inspiration from “Lavender” – an AI-enabled tool used by the Israel Defence Forces (IDF) since 2023 to identify targets in Gaza with minimal human oversight. This utilised a database generated using publicly acquired data that had been annotated under civil contexts (Raska, Reference Raska2024). Finally, we draw on Zala’s discussion of potential uses of AI in resort-to-force contexts (Zala, Reference Zala2024).
In our imagined resort-to-force DSS, data from geographically distributed and connected sensor networks designed to identify signs of armed conflict escalation (e.g. missile launches, weapons fire, increased activity along strategic transit lines, etc.) are processed to detect deviations from peacetime status using AI models that have been trained with supervised learning. A second set of unsupervised learning models identifies “outlier” data in unstructured streams of intelligence data, media reports, policy briefs, communication transcripts, etc. Both model sets inform a threat analysis, which is presented to the decision maker to support their decision on how to proceed. The same threat analysis is used as an input to a third set of models, which has access to many datasets, including geographic information, military resource information, and key adversary movements (the latter of which, like “Lavender,” has been trained on data collected and tagged for civilian contexts, presumably in times of peace). These models suggest effective strategies for the resort to force because the decision maker is not required to use the DSS, and is assumed to have other (human) advisors providing additional or parallel guidance.
We will now use this hypothetical system to explore how complexity emerges in AI-enabled systems in resort-to-force contexts. To support our exploration, we will define the DSS, the decision maker, and the broader organisation the decision maker acts within to be a socio-technical system – an intentional hybrid, as Amir and Kant (Reference Amir and Kant2018) describe, of human, organisational, and technological elements that are purposefully intertwined to support the achievement of a specific goal. For this discussion, it is worth noting that a machine-learning model pipeline (with all the humans involved in deciding how a model comes to be) can be treated as a socio-technical system that operates to create a machine-learning model. The machine-learning model this socio-technical system creates can later be used purposely as an element within a new socio-technical system designed to achieve an entirely different goal.
In this discussion, our hypothetical system is meant to support effective resort-to-force decision making and subsequent action. Some AI elements of this system may be used for more than one purpose (be “entangled,” to use a term proposed by Acton (Reference Acton2018), in other socio-technical systems used to support different goals), and – depending on the boundaries one draws and the analysis one wishes to perform – can be treated as technical or socio-technical systems in their own right.
In the following, we explore the complexities AI-enabled systems serve to amplify in our hypothetical system, and explore how such complexities pose both challenges and opportunities to the system’s ability to fulfil its goal.
2. Complexity, safety, and the resort to force
We argue that the complexity of AI-enabled systems emerges in three distinct ways. The first is interactive and nonlinear complexity, which concerns the intertwined components that make up an AI system. The second is software complexity, which deals with the increasing level of intelligence of AI systems. The third is dynamic complexity, which concerns the speed and scale of AI development and implementation. These complexities and their associated safety challenges are discussed in the following section.
2.1. Interactive and nonlinear complexity
The first form of complexity emerges from the intertwined components, both internal and external, of an AI-enabled system. Here, our discussion focuses on the technical aspects of AI-enabled systems, which operate via an amalgam of internal and external components. Internal components are those within the system itself, which include code, sensors, and algorithmic functions (Gacek, Abd-Allah, Clark & Boehm, Reference Gacek, Abd-Allah, Clark and Boehm1995). External components include dependencies on things outside of the system itself, such as external networks and inputs or prompts from users (Rajhans et al., Reference Rajhans, Cheng, Schmerl, Garlan, Krogh, Agbi and Ajinkya2009).
The interaction of these elements means an error, malfunction, or unexpected outcome in one element may have a domino effect on other element(s), because they are all interdependently linked. As the number of components increases, the number of interactions between components also increases. This is what is referred to as interactive complexity (Tiwari & Kumar, Reference Tiwari and Kumar2014).
Because of the interdependent nature of an AI system, the cause and effect of these errors, malfunctions or unexpected outputs may not be related in direct or linear ways. This is referred to as nonlinear complexity (Leveson, Reference Leveson2011). Nonlinear complexity makes problem solving in the event of an error, malfunction, or unexpected output challenging and time-consuming to remediate. Interactive and nonlinear complexity go hand in hand.
The safety challenges associated with interactive and nonlinear complexity span both technical and socio-technical considerations. In the next section, we widen our boundaries to include the human (decision making) elements of the system, so we can focus on the safety challenges misinformed decision making may pose in the context of interactive and nonlinear complexity.
2.2.1 Misinformed decision making
There is a dissonance between AI-enabled systems and military grey zones, which stems from the former operating on statistical calculations shaped by data, and the latter operating in the nuanced intricacies of political power (Snyder, Reference Snyder1960). AI is a software-enabled capability which – even in the most sophisticated models that exist today – is inherently limited by the past choices of its (human) creators and/or the data on which it is trained. By comparison, military grey zones, the operational space between peace and war (Dobias & Christensen, Reference Dobias and Christensen2022), involve “…coercive actions to change the status quo below a threshold that, in most cases, would prompt a conventional military response, often by blurring the line between military and non-military actions and the attribution for events” (Morris et al., Reference Morris, Mazarr, Hornung, Pezard, Binnendijk and Kepe2019, p. 8). These grey zone operations often sit at the precipice of decisions on deterrence or resort to force.
AI-enabled DSS used to support decision making on the resort to force would need to capture and analyse nuanced and politically entangled data. Data in its raw form consists of numbers, words, images, etc., all of which require labelling and contextualisation to become useful information. Labelling, which underpins AI models that use “supervised learning,” is still conducted largely by humans (Verma, Nagar & Mahapatra, Reference Verma, Nagar, Mahapatra, Satpathy, Choudhury and Satpathy2021).
For most civil mainstream AI applications developed by major technology companies, data labelling is predominantly outsourced to workers in the global south: Kenya, India, and the Philippines (Okolo, Reference Okolo and Lindgren2023). This work is labour-intensive, and it requires knowledge and skill to annotate large quantities of data to high standards of accuracy. From the outset, many may hold the assumption that AI-enabled military technologies would be developed differently from their civil counterparts, with tasks like data labelling being conducted with more care and discretion. However, as noted earlier, the IDF use of “Lavender” proved otherwise.
In our hypothetical system, the AI model tracking key adversary movements draws on civilian data gathered in peacetime, and predictions of the likely success of any given resort-to-force plan rely at least in part on this model. In a moment like this, an adversary’s actions may indeed reflect a likely move towards conflict – a scenario the model is unlikely to have been trained on. Therefore, the AI system consists of internal and external components of opposing contexts. The training data internal to the system were derived from publicly available sources, annotated under civil contexts, and gathered in peacetime, while the external user inputs to the system are rooted in contexts of likely conflict and political instability.
The interactive complexity of these juxtaposing contexts is evident at both the technical and socio-technical levels. At the technical level, inaccurate or contextually inappropriate data annotations can lead to incorrect inferences – i.e. if our hypothetical system used peacetime data to predict an adversary’s likely location. Unless data labellers were annotating data with a verifiable indication of escalating political or military tensions, inaccuracies in the system outputs are likely. It should also be noted that curating a data set that captures every aspect of an operating environment is close to impossible (Saidulu & Sasikala, Reference Saidulu and Sasikala2017).
At a socio-technical level, inaccurate outputs presented to a human agent involved in informing or making decisions can have flow-on effects. In the context of the resort to force, the extent of potential harm which may ensue from misinformed decision making spans political, economic, and military domains (Davis, Reference Davis2024). For example, if there were information on adversaries of interest potentially located in an area and the DSS used this information to project the adversary’s likely next move, an incorrect prediction of both their location and likely movements could lead to the unnecessary and ineffective use of force. This lesson – untestable until the moment of such a decision – may lead to overconfidence that a resort-to-force decision may lead to success, or alternatively, suggest an inappropriate plan of action in the event the decision is made. This may contribute to a higher probability of initiation or escalation of conflict, along the lines of what Pauly and McDermott (Reference Pauly and McDermott2023) describe as a risk that “comes from within the system, not outside it.” The decision maker retains choice in this context, but the choice in question may be inappropriately influenced by misleading AI-generated recommendations.
Additionally, because of the nonlinear complexity of AI-enabled systems, identifying points of error within the system when poor recommendations are made may not be easy. From a liability perspective, the argument of “the machine is at fault” cannot hold because software does not act on its own accord. Rather, it is integrated among a broader system which includes hardware, processes and procedures, humans, and environments within which it performs its functions (Hardy, Reference Hardy2012). Chain of responsibility models for AI-enabled capabilities exist that can be used to assess the complete lifecycle of an AI system, from design through to deployment, to determine where along this chain the points of failure, and thus the attribution of liability, lie (Walker-Munro & Assaad, Reference Walker-Munro and Assaad2023).
In the case of the Petrov incident, the initial point of failure may have been the early warning data; however, had a different decision been made at the time, the attribution of liability would ultimately have sat with the person responsible for making the final decision, without which force would not have been initiated.
2.2. Software complexity
The second form of complexity comes from the software underpinning the system. The types of AI discussed here all have a capacity for learning. They are also probabilistic in nature – which can pose significant problems when input data is not representative of what the model was trained to handle. These characteristics mean the outputs of AI-enabled systems are not always known or entirely predictable, and the computational process for how outputs are achieved may be difficult to ascertain (Pedreschi et al., Reference Pedreschi, Giannotti, Guidotti, Monreale, Ruggieri and Turini2019).
Anticipating harms and implementing mitigations against them to create a controlled environment is a foundational approach to safety (Leveson, Reference Leveson2023). The complexity of AI-enabled systems challenges the predictability that safety engineering manufactures by design.
It is important to note that while AI-enabled systems are unpredictable, their unpredictability is bounded by the role or purpose of the machine, known as the machine functions, and what the machine is capable of doing, known as the machine capabilities (Assaad, Reference Assaad2022). Therefore, while an AI-enabled system will demonstrate unpredictability because of its capacity to learn over time and its probabilistic nature, this unpredictability will not exceed the bounds of the machine’s purpose and capabilities.
Despite the bounded nature of this unpredictability, safety challenges still emerge from software complexity. Of relevance to resort-to-force contexts is the loss of knowledge that comes with complex software.
2.2.1 Loss of knowledge
The more intelligent a system is, the less knowledge there is about how outputs are produced. These are referred to as “black box” systems. The outputs of these systems are not readily interpretable because the computational process for achieving outputs is too complex (Nugent & Cunningham, Reference Nugent and Cunningham2005).
The purported purpose of a DSS is to assist humans in making better and more informed decisions without making the decision for them (Larson & Hayes, Reference Larson and Hayes2005). Military operations are conducted in dynamically changing environments, which necessitate the analysis of large volumes of data (Horyn, Bielewicz & Joks, Reference Horyn, Bielewicz, Joks, Visvizi and Bodziany2021). An AI-enabled DSS can aid in filtering through large quantities of data in very short time frames.
It is the role of a human to discern how to interpret and use outputs from a DSS. Interpreting information becomes difficult when the logic for how that information was produced is not clear or transparent. From a technical safety perspective, misinterpretation of information can result in incorrect or inaccurate understandings of a situation, environment, or operation, leading to divergent or conflicting decision making.
The Cuban Missile Crisis is a salient example of the importance of correctly interpreting information. Had the naval officer Vasily Arkhipov interpreted information differently, this would have led to the deployment of a nuclear-armed torpedo, which would have initiated a nuclear war between the United States and the Soviet Union. When analysing information provided by a system, be it AI-enabled or otherwise, it is imperative to question what that information is based on and to consider how much confidence there is in that information given other circumstantial considerations. These are all critical pieces of knowledge a human decision maker would need to assess before deciding on the resort to force. For our hypothetical DSS, the distributed sensor networks and information streams feeding the system are likely to be complex and may feed multiple systems with divergent purposes; possible sensor, network, or data stream errors, failures, or vulnerabilities may be difficult or impossible to identify; and the many ways such issues are likely to shape the AI model output (particularly in times of war) may be untestable. All of this poses a considerable challenge to the decision maker and their team in a necessarily tense resort-to-force decision-making scenario.
From a socio-technical perspective, a loss of knowledge in how outputs are achieved can lead to inappropriate levels of trust for the human operator (Parasuraman & Riley, Reference Parasuraman and Riley1997). Trust is a multifaceted concept, which is diversely defined across the literature. Within the context of this article, trust is defined as confidence in the reliability of a system when used in the intended operation of use (Assaad & Boshuijzen-van Burken, Reference Assaad and Boshuijzen-van Burken2023). It is important to note that this confidence is an attitude a human (or human system – organisation, state, etc.) holds in the face of some possibility that they will experience harm. In essence, there must be an element of risk involved for the human(s) choosing to use the system in question. Trust enables humans to use such systems despite such risks.
Trust can change with time and circumstance, the question always being whether those changes are fitting or not given the risks at hand. In the absence of knowledge on how system outputs are produced, the amount of confidence a human can have in a system often leans too far in either direction: complete confidence under the assumption the system will not and cannot fail (over trust), or no confidence at all because of the lack of knowledge in how the system operates (under trust).
In situations of over trust, humans are less likely to pick up on errors or to interrogate outputs because they assume the system is always correct (Aroyo et al., Reference Aroyo, Bruyne, Dheu, Fosch-Villaronga, Gudkov, Hoch, Jones, Lutz, Sætra, Solberg and Tamò-Larrieux2021). In instances of under trust, humans may ignore or misuse a system, thereby underutilising its capabilities and purpose (Ullrich, Butz & Diefenbach, Reference Ullrich, Butz and Diefenbach2021). These opposing challenges have led to discussions of how to calibrate trust – in other words, how to ensure a match between perceived and actual trustworthiness of an AI-enabled system (Wischnewski, Krämer & Müller, Reference Wischnewski, Krämer and Müller2023). As Wischnewski et al. (Reference Wischnewski, Krämer and Müller2023) discuss, there are various ways of calibrating trust – but all rely on some means of properly characterising the AI system and its capabilities so the humans engaging with the AI systems can be trained appropriately. This can only occur when the scenarios in question can be properly characterised and trained for, which is unlikely to be the case for a DSS system used in resort-to-force decision making.
Returning to the Cuban Missile Crisis, in an instance of over trust, the human decision maker may have trusted his advisor’s interpretation of the significance of the American depth charges exploding around his submarine without question. The response to this assumption would have been nuclear warfare.
The stressful nature of decision making in military contexts exacerbates the implications of miscalibrated trust. The impacts of stress on decision making have been studied across a multitude of safety-critical industries, such as aviation, the military, and emergency services (Flin, Salas, Straub & Martin, Reference Flin, Salas, Straub and Martin1997; Gamble et al., Reference Gamble, Vettel, Patton, Eddy, Davis, Garcia, Spangler, Thayer and Brooks2018). What distinguishes AI-enabled systems from more traditional safety-critical systems is their complexity.
For advanced and complex AI-enabled systems, an inability to interrogate those systems can fuel greater levels of under trust. The impossibility of fully testing such systems in resort-to-force contexts is likely to exacerbate this challenge.
Additionally, the decisions made by a human in the context of the resort to force would need to be justifiable on multiple levels – legally, operationally, morally, etc. Without an understanding of how information was generated, it becomes far more challenging to justify and defend decision making.
In terms of the unpredictability of AI-enabled DSS, safety challenges emerge from how humans respond to that unpredictability. In the event of an unexpected or unpredicted output, it may be difficult for a human to discern if that output is an error, anomaly, or outlier case. Unpredictability can create confusion – and when coupled with a loss of knowledge on how an output has been produced, the result is greater discrepancies in trust calibration, which can lead to unique safety challenges.
2.3. Dynamic complexity
The third form of complexity emerges in the form of speed and scale. AI-enabled systems are evolving fast enough that appropriate safety, regulatory, or legal controls cannot keep pace. Furthermore, AI applications have scaled across a plethora of applications, spanning almost every industry and discipline, from healthcare to defence.
Because AI is a data-driven capability, the complexities of scale and speed also manifest in the data used to train and operate AI. The scale of data being used and the speed at which it is analysed far exceeds the capacity of any human. This is partly why the use of AI has been adopted in DSS. Dynamically changing environments, such as those common within military contexts, involve copious amounts of information that need to be considered to inform decision making. AI-enabled systems provide a mechanism for analysing large amounts of data in a matter of minutes. For resort-to-force decision making, having more data to make these deliberations may seem like an optimal option; however, concerns around ubiquity and timescales, discussed in the next section, prove otherwise.
2.3.1 Ubiquity and timescales
The ubiquity of digitally enabled systems means they are not bound by geography. One capability can be distributed across various locations and accessed remotely by others. From a technical perspective, a lack of physical proximity to a system and the use of digital systems for data collection can result in both fault detection challenges, as well as latency (i.e. delay) between receiving and transmitting information.
The challenge of erroneous data due to component failure is somewhat obvious. The potential impact of latency and timing on such a system may be more subtle. In our hypothetical system, the distributed sensor networks may collect and send data with different (and variable) frequencies and latencies, while the unstructured intelligence/media data the final DSS recommendation draws upon may be influenced by human communication networks or processing labour with unknown or highly variable transmission latencies. The data informing the DSS recommendations is therefore not representative of one moment, but may instead couple together data from disparate and potentially unquantified pasts. This effect may lead to a recommendation that is potentially inaccurate or even irrelevant to the present moment.
Operational performance is an attribute of safety (Wang, Cao & Yang, Reference Wang, Cao and Yang2023). When a system does not perform in an efficient or effective manner, it compromises safety because it impacts how the system operates in a given environment and how humans, other systems, or entities respond to that change. In the case of AI-enabled DSS, inefficient operational performance can lead to reduced opportunities for strategic decision making and greater likelihoods of risk and failure in the decisions that are made. For resort-to-force decision making, the repercussions of poor decision making can be highly consequential.
The boundless geographical span of AI implementation also means systems may be trained against data that reflects other contexts and environments. As was discussed previously, the existing dominant technology companies outsource data labelling to countries such as Kenya, India, and the Philippines. Even if these data labellers were told to consider a particular place or context, it is unlikely they would be capable of thoroughly capturing the depth and nuance of an environment or context they are unfamiliar with.
The data sets these organisations develop can be used in military contexts, as was demonstrated with Lavender in April 2024, even if this was not the original intention of use. Multi-purpose use cases of data sets are common practice (Prasad & Park, Reference Prasad and Park1994). When adopting a pre-curated data set, it is common practice to put time and effort into further annotating and categorising that data to ensure it is fit for its intended use (Ostertag, Hendler, Diaz & Braun, Reference Ostertag, Hendler, Diaz and Braun1992). However, poor or inconsistent data set design and documentation practices can pose a challenge to those seeking to carry out this work (Reid & Williams, Reference Reid and Williams2023a, Reference Reid, Williams, Muresan, Chen and Casey2023b).
An additional safety consideration is the varying timescales of datasets. The data AI models are trained on reflect a particular span in time. The instability of changing political climates can result in the obsolescence of data collated in the past.
The ubiquity and varying timescales of AI present challenges in any environment; however, in the changing political landscape of military domains, these challenges are heightened.
3. The human condition
As Erskine (Reference Erskine2024a; Reference Erskine2024b) has argued, many of the current narratives around AI embody anthropomorphic language, attributing human characteristics to what is in essence, a statistical construct. Veiling AI in human traits and characteristics is a strategic way of absolving humans from liability.
Because of this and the many safety challenges we have raised earlier, it is the position of this article that decisions on the resort to force are, and should always be, made by humans. We do not suggest that AI-enabled systems are incapable of fulfilling this role from a technical perspective; rather, we are highlighting that human beings are, and always will be, the central decision makers. If an AI-enabled system were to make an actionable decision on the resort to force, it would only be because a human decided to allow an AI-enabled system to fulfil that role. Handing over the decision making to a machine does not absolve a human of liability. There exists a hierarchy of control and a chain of decision making that would have led to the point of an AI-enabled system making decisions on the resort to force. This chain of responsibility cannot be ignored under the false assertion of AI operating with a mind of its own.
Military operations and changing political landscapes are complicated, nuanced, and amorphous environments that require strategic and considered decision making. The notion of delegating decisions on the resort to force to a machine does not align with the careful consideration this kind of decision demands. If an AI-enabled tool were to be delegated this level of decision making, concerns should not be directed at the system; rather, they should be directed at the person or persons responsible for implementing that system in that capacity.
It is through this lens that the following recommendations are presented for states or individuals utilising AI-enabled DSS to support human decision making on the resort to force:
• Recommendation 1: There is an intersection between the safety of AI-enabled DSS and resort-to-force decisions. Human decision making is impacted when safety is compromised. When implementing AI-enabled DSS, these safety considerations should be formally captured through a safety review process conducted by appropriately trained experts as described by Davis (Reference Davis2024). This process can either end with the decision not to proceed or identify mitigations that support safe use. In the latter case, the review process should also identify points (or triggers) in the system life cycle that would necessitate a re-assessment of risks and required mitigations, and ensure appropriate organisational support for ongoing re-assessment.
• Recommendation 2: The notion of safety of AI encompasses both technical and socio-technical considerations. When assessing the potential safety challenges of AI-enabled DSS, it should be considered holistically to include broader considerations such as security, trust and liability.
• Recommendation 3: The roles and capabilities of AI-enabled tools are commonly misunderstood or embellished, particularly when determining what human roles and responsibilities are in relation to these tools. When implementing AI-enabled DSS, the roles and responsibilities of both the system and the humans operating alongside the system should be clearly identified, documented, and understood by those responsible for making use of such systems. Appropriate training and education should also be provided for those operating alongside these systems.
• Recommendation 4: Given the magnitude of the risks in question, states should consider ways of implementing policy that supports safety for people and organisations responsible for systems making use of AI in such high-risk applications. This may include setting up an independent regulator responsible for ensuring the safety reviews recommended earlier are conducted appropriately and that any risk mitigation strategies are properly applied. Approaches used to control other high-hazard technologies (e.g. nuclear) may provide useful inspiration.
4. Conclusion
We have argued that the complexity AI introduces into resort-to-force decision making creates unique safety challenges. We consider safety through a socio-technical lens, embodying considerations that extend beyond physical harm. In the case of AI-enabled DSS used in support of resort-to-force decision making, the interaction between safety and security concerns is highlighted. Three complexities and their associated safety concerns are discussed. Interactive and nonlinear complexity can result in misinformed decision making. Increased software complexity leads to a loss of fundamental knowledge in how systems operate, and dynamic complexity creates challenges with ubiquity and varying timescales. Decisions on the resort to force are made by humans, with AI-enabled DSS likely to be implemented in support of those human roles. Because resort-to-force decision making is fundamentally a human practice, the safety challenges complex AI systems introduce manifest at the human level. It is the human decision makers who must grapple with misinformed decision making, a loss of knowledge, and the ubiquity and varying timescales of AI. It is the human decision makers who must determine what risks they are and are not willing to accept. The line between deterrence and conflict escalation is maintained by decisions on the resort to force. This complex and nuanced space is made increasingly more complex with the introduction of AI-enabled DSS. While AI does present strategic opportunities for resort-to-force decision making, the unique safety challenges these systems introduce cannot be ignored.
Funding statement
No funding was received in relation to this article.
Competing interests
The authors declare no competing interests.
Zena Assaad is a Senior Lecturer in the School of Engineering at the Australian National University. She has held fellowships with the Australian Army Research Centre and Trusted Autonomous Systems. Her research explores the safety of human-machine teaming and the regulation and assurance of autonomous and AI systems. Dr Assaad is a Member of the expert advisory group for the Global Commission on Responsible AI in the Military Domain.
Elizabeth Williams is a Nuclear Systems Discipline Lead in the School of Engineering at the Australian National University (ANU). She is a nuclear physicist by training, with a PhD in nuclear physics from Yale. She joined ANU in 2012 and has held an Australian Research Council Discovery Early Career Researcher Award Fellowship in nuclear reactions. Her current research focuses on the responsible integration of AI-enabled systems in safety-critical contexts.
Open access
