First, the rise of agentic AI

Agentic AI refers to systems designed to act as autonomous agents that can plan, make decisions and carry out sequences of actions to reach a goal. Unlike traditional AI that responds to a prompt or performs a narrow task, agentic systems can decide what to do next, use tools and adapt their behaviour based on results. From a humanitarian perspective, through the so-called RAG, these agents can ground their outputs within key guidelines, such as the mhGAP Humanitarian Intervention Guide (WHO and UNHCR, 2015), and the Sphere Handbook, which are the global minimum standards for humanitarian response (Sphere Association, 2018). For example, such systems could support community health workers by grounding guidance in mhGAP protocols during supervision or triage. While these systems can use internal self-evaluation to flag uncertainty for human review, their autonomous nature requires robust accountability frameworks, mechanisms that are currently absent from humanitarian AI governance. In mental healthcare, the consequences of weak accountability are particularly serious: if an AI misreads a local expression of grief as symptoms of psychosis and a clinician accepts this label without challenge, the result can be wrongful escalation, inappropriate medication or unnecessary hospitalisation. This tendency to accept AI-generated information without critical reflection, known as automation bias, has been shown to lead clinicians to follow incorrect AI recommendations at substantially higher rates (26%) than they would otherwise (Goddard et al., Reference Goddard, Roudsari and Wyatt2012). These effects are amplified under high workload, time pressure and limited experience, conditions characteristic of humanitarian frontline work (Goddard et al., Reference Goddard, Roudsari and Wyatt2012).

Second, the emergence of an agentic workforce

AI agents are increasingly operating as team members alongside human workers, and many technology companies are actively pushing for this. In MHPSS, the appeal is strongest where treatment gaps are wide and documentation burdens are high. AI agents can be used to streamline case notes, produce structured supervision summaries, support translation and adaptation of materials, and suggest guideline-aligned options. However, for humanitarian MHPSS response, it can import assumptions about productivity into settings where relationship and rapport building are often a key part of the intervention. What counts as an efficiency gain in corporate workflows may be a clinical loss in MHPSS if it reduces active listening, presence or trust building. This is particularly risky for trauma-affected populations and communities who have justified concerns about surveillance or misuse of sensitive information. The central governance question is therefore not whether AI agents work, but which tasks can and should be delegated without weakening care.

Third, open-source AI models

Open-weight models, those whose underlying parameters are publicly released, allowing anyone to download, run and modify them (i.e., DeepSeek R1, Qwen 3, Llama and Mistral), are performing equally well or better than leading proprietary models, which are accessed only through the developer’s servers (i.e., GPT-4o) (Sandmann et al., Reference Sandmann, Hegselmann, Fujarski, Bickmann, Wild, Eils and Varghese2025; Stachura et al., Reference Stachura, Konieczna and Nowak2025). In practice, the availability of open-source models means they can be run locally, fine-tuned on new data and used in secure environments. For humanitarian MHPSS, this matters because, in theory, it makes high-quality models usable under tighter control: organisations can fine-tune on region-specific protocols, run systems offline in low connectivity settings and adopt community-led governance without sending sensitive data to external corporate APIs, for example. This is potentially relevant for MHPSS serving displaced populations where data sensitivity is essential, and offline use could be transformative, though, to our knowledge, empirical evaluations of fine-tuned open-source AI models in humanitarian MHPSS settings are not yet available.

However, open-weight also relocates responsibility. With proprietary models, the developer retains control over safety testing, content moderation and clinical guardrails. With open-weight models, if a humanitarian organisation downloads and adapts a model, that duty sits primarily with the deployer (i.e., an NGO) rather than the vendor (i.e., DeepSeek R1). In a sector already marked by limited AI capacity and governance, there is a real risk of open-weight models being rolled out without sufficient clinical containment, increasing the chance of unsafe guidance or discriminatory outputs, with direct implications for triage, risk assessment and frontline decision-making, among others.

Fourth, the collapse of inference costs

The rapid decline in AI inference costs may affect the feasibility of digital tools in humanitarian health. Inference is the cost of running a trained model in real time, such as answering questions or translating text. Recent analyses suggest that the price to achieve a given level of AI performance has been declining roughly tenfold per year (Gundlach et al., Reference Gundlach, Lynch, Mertens and Thompson2025). Whether this trajectory continues, plateaus or reverses is uncertain and depends on hardware, regulation and energy costs. If even a fraction of these gains continues, applications that were prohibitively expensive 2 years ago, like real-time translation or quality monitoring, will become more economically feasible. Lower costs could, in turn, make multilingual psychoeducation, supervision support and translation of clinical encounters, for example, more feasible in resource-constrained settings. Though feasibility will depend on language coverage, safety testing and integration costs that are not captured in the per-token price.

However, these gains are not evenly shared across languages. Because models are optimised for English-language performance, non-English languages systematically require more tokens to express the same meaning. For example, a quick analysis of the token tariff on different EU languages showed an ~87% higher token use for Dutch, 151% for Polish and 94% for Portuguese (van der Boor, Reference van der Boor2025). Since AI providers typically bill organisations per token processed, this technical inefficiency translates directly into a financial tax, creating a built-in cost and performance penalty for non-English interactions. The performance gap compounds the cost gap: in a recent psychiatric evaluation, an open-source LLM produced significantly more hallucinations and significantly lower diagnostic accuracy when processing clinical notes in Korean compared to their English translations (Kim et al., Reference Kim, Hwang, Chang, Chang, Roh and Park2025). In humanitarian MHPSS, where many languages differ further from English, the same budget buys fewer and less safe interactions for the people most in need. Some efforts, such as language-specific tokenizers and targeted training data, reduce this gap, but they tend to follow market demand, leaving lower-resource languages less supported and more costly. Addressing this gap will likely require coordinated investment by multilateral agencies, humanitarian donors and research partnerships to develop tokenizers and training datasets for underrepresented languages.

These technological developments do not determine how AI should be used in humanitarian mental health care, but they shape which applications are technically and economically feasible. Their implications depend on how systems are governed, which tasks are delegated to AI and how risks related to safety, bias, and accountability are managed within fragile service environments and rapid response. They also raise questions of equity, as many crisis-affected populations rely on languages that remain poorly represented in current models.

We use the multi-layered intervention pyramid from the 2007 Inter-Agency Standing Committee Guidelines on MHPSS (IASC, 2007) as an organising framework to show how AI capabilities might apply to MHPSS response in humanitarian contexts. The IASC Guidelines were developed through inter-agency consensus and are the most widely endorsed framework for organising MHPSS in humanitarian response, structuring services into four progressively specialised layers. The following section examines how the emerging AI capabilities might translate into concrete applications across the levels of the IASC MHPSS pyramid.

Misreading cultural expressions of distress

Mental health conditions often manifest through culturally shaped idioms of distress, somatic complaints, spiritual explanations and language and symptoms that vary across populations (Cork et al., Reference Cork, Kaiser and White2019, Kirmayer et al., Reference Kirmayer, Gomez-Carrillo and Veissière2017). AI systems trained predominantly on Western clinical taxonomies may misinterpret these expressions, leading to inappropriate or absent referrals (Kim et al., Reference Kim, Hwang, Chang, Chang, Roh and Park2025). This is not inherent to AI; human providers can also struggle with cultural interpretation, but AI systems lack the relational context that allows clinicians to probe, clarify and adapt. These risks are compounded in non-English languages, where model performance is further degraded, as discussed above.

Over- or under-pathologising crisis reactions

A related risk is the misclassification of normal coping reactions to abnormal circumstances as clinical disorders, or conversely, failing to identify genuine suffering expressed through somatic language, metaphor or behavioural withdrawal. AI screening tools calibrated to clinical populations may not distinguish adaptive distress from pathology in crisis contexts. Generative systems may also produce confident, persuasive explanations for incorrect classifications and their outputs can vary across near-identical prompts, making errors harder to anticipate, reproduce and audit in clinical workflows. Avoiding this requires both domain expertise and training in how to use AI appropriate for such workflows.

Amplifying bias

Several biases that exist in general AI use are likely to be further increased in crisis contexts, which has an impact on MHPSS response. First, as previously described, automation bias is well-documented and may increase under time pressure, uncertainty and among non-specialist workers (Goddard et al., Reference Goddard, Roudsari and Wyatt2012; Lyell and Coiera, Reference Lyell and Coiera2017). Second, language models show a tendency to align responses with users’ expressed beliefs over truthful ones, a behaviour known as sycophancy (Sharma et al., Reference Sharma, Tong, Korbak, Duvenaud, Askell, Bowman, Cheng, Durmus, Hatfield-Dodds, Johnston, Kravec, Maxwell, McCandlish, Ndousse, Rausch, Schiefer, Yan, Zhang and Perez2024). In mental health interactions, this carries particular risk: a model that echoes a user’s negative thoughts rather than gently challenging them could deepen distress rather than ease it. These risks are not unique to humanitarian MHPSS, but they are likely to be magnified in populations with elevated rates of trauma, depression and anxiety, and in workforces with potentially limited expertise and capacity to second-guess confident-sounding AI outputs.

Lack of containment and risk management

General-purpose AI systems today are not trained to apply specific therapeutic techniques, such as grounding, and will struggle to pick up cues to respond to dissociation or provide the interpersonal safety required during moments of acute distress. Without capacity for containment, there is a risk of (re)traumatisation, overwhelm or abandonment. Humanitarian workers are trained to assess self-harm risk, recognise safeguarding concerns and initiate protective referrals. Current AI systems cannot reliably perform these tasks. This unpredictability is particularly problematic in safeguarding situations, where consistent responses and traceable decision logic are needed.

Data governance and consent

This is particularly acute for refugee and displaced populations. If data governance is weak, information about mental health status, trauma disclosures and/or psychosocial distress could be accessed by third parties, exposing individuals to physical and/or social risks. When LLMs are connected to case management systems or automated workflows that can perform actions, additional measures, such as limiting access to authenticated users, should be taken to avoid potential data exfiltration through LLM-specific risks like prompt injection attacks. Prompt injection attacks are where manipulated inputs are used to override system instructions or trick the model into revealing sensitive information or performing unauthorised actions.

These risks are compounded by the difficulty of getting genuine informed consent in humanitarian settings. Crisis-affected populations often depend on services with limited alternatives, and refusing data collection could be seen as risking access to assistance (Kreutzer et al., Reference Kreutzer, Orbinski, Appel, An, Marston, Boone and Vinck2025). AI adds further challenges: lay people may not understand AI processing, future uses of data by commercial models cannot be fully specified, and data used to train or fine-tune models currently cannot be withdrawn (Kreutzer et al., Reference Kreutzer, Orbinski, Appel, An, Marston, Boone and Vinck2025). In this context, meaningful consent may be difficult to guarantee, and mental health information may carry particular risks for individuals, as disclosures can lead to stigma and family, legal, social or protection-related harms.

Disruption of trust and community-based coping

If AI reduces human connection in MHPSS, it risks undermining trust in communities with histories of trauma, state violence and institutional mistrust, potentially increasing withdrawal and reducing help-seeking. Trust also operates on the provider side: if tools are poorly integrated, unclear or misaligned with workflows, clinicians may either avoid them altogether or adopt informal workarounds, potentially creating new safety and privacy risks. AI tools may also focus primarily on individualised screening and referral, neglecting collective forms of resilience, peer support and traditional healing networks that are central to recovery in many cultural contexts.

The benchmarking gap

In other domains, validated benchmarks are being developed to evaluate AI performance (e.g., SWE-Bench for software engineering [Rashid et al., Reference Rashid, Bock, Zhuang, Buchholz, Esler, Valentin, Franceschi, Wistuba, Sivaprasad, Kim, Deoras, Zappella and Callot2025] and MedQA for clinical knowledge recall [Jin et al., Reference Jin, Pan, Oufattole, Weng, Fang and Szolovits2021]). Most of these are not conversational, and those that are, like Healthbench by OpenAI (Arora et al., Reference Arora, Wei, Hicks, Bowman, Quiñonero-Candela, Tsimpourlas, Sharman, Shah, Vallone, Beutel, Heidecke and Singhal2025), are 85% in English and do not focus on mental health. No equivalent frameworks exist for AI in humanitarian MHPSS. Without rigorous evaluation, limitations may only be discovered through error and potential harm. Developing context-specific benchmarks, that is, co-designed with local mental health professionals, affected communities and humanitarian practitioners, should be a research priority.

Who builds these systems?

Foundation models are developed by technology companies (e.g., OpenAI, Anthropic, Google DeepMind and DeepSeek) with commercial priorities. This raises questions of technological dependency, vendor lock-in and reliance on tools poorly suited for applications in humanitarian MHPSS, as they were designed for other purposes.

Synthetic data: Opportunity and risk

AI-generated synthetic data could increase sparse training datasets for humanitarian contexts. However, as data generation becomes easier, creating methods for ensuring data quality becomes more important. New approaches to track data provenance, verify authenticity and ensure quality are needed to ensure high-quality data can be used to train and evaluate AI applications in context.

While the risks are presented as a list, they do not all operate at the same level of the IASC pyramid. Table 2 summarises each risk and indicates the levels at which we consider it to be most acute.

Table 2.

Summary of risks in the use of AI for humanitarian MHPSS, mapped to IASC pyramid levels

Table 2. Long description

The table consists of three columns: Risk, Summary, and I A S C levels most affected.

* Misreading cultural expressions of distress: A I trained on Western taxonomies misinterprets local culture and idioms of distress. Affects all levels, especially 2, 3, 4.

* Over- or under-pathologising crisis reactions: Misclassifying adaptive distress as pathology, or missing genuine suffering. Affects levels 3, 4.

* Amplifying bias: Automation bias, sycophancy and confirmation bias magnified in distressed populations. Affects all levels, especially 2, 3.

* Lack of containment and risk management: A I cannot reliably manage acute distress, dissociation or safeguarding. Affects levels 3, 4.

* Data governance and consent: Sensitive disclosures, weak consent, lack of transparent data flows and ownership. Affects all levels.

* Disruption of trust and community-based coping: A I reducing human connection, collective resilience and imposing an individualistic approach to support. Affects levels 1, 2, 3.

* The benchmarking gap: No validated evaluation frameworks for humanitarian M H P S S. Affects all levels.

* Who builds these systems: Commercial priorities, vendor lock-in and technological dependency. Affects all levels.

* Synthetic data: opportunity and risk: Opportunities for scarce data offset by quality and provenance risks. Affects all levels.