Coyote is a key figure in Navajo and Apache mythology, representing both good and evil, animals, humans and gods. He is typically unpredictable and ambivalent, an instrument of both chaos and good. He is a trickster “[s]imultaneously an omniscient creator and an innocent fool, a malicious destroyer and a childlike prankster.” The coyote is a “scapegoat onto which to project the fears, failures and ideals of the culture.”Footnote 1 Brittanica.
Introduction
AI is built on modern tools that date back to early statistical tools and data pattern discernment.Footnote 3 A summary of the critical terms includes data mining, machine learning, large language models, and deep learning. Statistical tools underpin the use of data. Statistics is the study of data.Footnote 4 Statistical tools are crucial to ensure that the results from ML models are valid, reliable, and meaningful.Footnote 5 AI is built on statistical ideas and algorithms. Algorithms work through huge data sets to generate decisions. Footnote 6 The second unique feature of AI is that of modern computer power, enhancing data mining and handling these massive data sets in seconds. This computing power allows AI algorithms to create agents that can learn from experience and replace human intelligence.Footnote 7 They evolve and refine their performance by exposure to vast amounts of data.Footnote 8 I will add some categories to the range of AI categories to give a better sense of the evolution of statistical tools coupled with strides in computing power and speed; the progress has been extraordinary but the use of AI in health care is still primitive.
AI has potential uses in healthcare, including improving disease diagnosis and treatment selection. AI tools can use large datasets to spot patterns that may prove more accurate than physicians can achieve. The promise is reduction of human error, improved accuracy and reduced costs.
Part I will take a deep dive into the tools of AI with some history as well to remind us that uses of masses of data in health care go back to Ernest Codman and Florence Nightengale.
Part II will critically examine the promised benefits of AI applications in health care. Currents studies comparing physician performance to that of AI tools reveal only that AI cannot yet be trusted – sometimes AI outperforms, other times fails to match the physician and often fails badly. A hospital board member examining the often-contradictory evidence of AI efficacy and trustworthiness must hesitate before voting to allow his/her hospital to adopt AI diagnostic and treatment tools.
Part III explores some of the problems created by the use of AI tools, along with other problems that such applications may create even if the current shortcomings of AI are worked out and uses are trustworthy. I argue that AI at this point in its development is the technological equivalent of the Navaho mythical figure Coyote. Coyote is unreliable, a trickster who often creates chaos. A review of the studies of AI applications in health care reveals that the programs misbehave, unable to yet be trusted in many health care settings involving real patients without considerable further work to institutionalize trust-creating mechanisms. I will consider the limits of the underlying data sources used for the essential platform for data mining, where in the healthcare arena it is too often garbage in and garbage out, give the poor quality if the inputs. AI tools suffer from operational limits: recursion, hallucinations, algorithmic brittleness, and unreliability. Results are also based on biased data of all kinds, reflecting the limits of studies. Another category of risks relate to bad actors who engage in intentional data-poisoning attacks and sabotage of AI instructions. Finally there is a category of harms that may affect physicians and other health care professionals in the hospital setting—a temptation hospitals to use AI as a cruel supervisor in managing both the medical staff and the nursing staff. AI uses in diagnosis and treatment may also cause doctors to experience deskilling and disconnection from patients. Finally, patients may be exposed to untrustworthy programs that cause adverse events and harm.
Part IV AI examines possible governance strategies in light of the problems outlined in Part III.Footnote 9 Product safety regulations must be developed by existing or new standard setting bodies.Footnote 10 In order to trust AI uses, they should be validated and constantly audited for errors so that hospitals and their health care providers can trust AI to be validated and error-free. I consider tools that might achieve these goals. patient treatment context.Footnote 11 These tools need to include expert training in the form of digital health professionals, regulatory tools such as certification to validate AI tools, and accountability rules to sort out who is responsible for adverse events created by AI programs. Part IV considers the role of the FDA and AI-Enabled Medical Device Regulation, accreditation by the Joint Commission, and the possible role of the National Institute of Standards and Technology (NIST), residing in the Department of Commerce. Finally, Part IV looks at the role of hospital liability and insurance as a source of incentives to develop safe AI use.
I. The Tools of AI in HealthcareFootnote 12
What is artificial intelligence? Simply put, AI refers to the science and engineering of making intelligent machines, through algorithms or a set of rules, which the machine follows to mimic human cognitive functions, such as learning and problem solving. AI systems have the potential to anticipate problems or deal with issues as they come up and, as such, operate in an intentional, intelligent and adaptive manner. AI’s strength is in its ability to learn and recognize patterns and relationships from large multidimensional and multimodal datasets; for example, AI systems could translate a patient’s entire medical record into a single number that represents a likely diagnosis.Footnote 13 Junaid Bajwa et al.
The tools of AI encompass several layers that date back to early statistical tools and data pattern discernment. A summary of the critical terms includes:
-
• Artificial Intelligence (AI), the broadest category, dealing with “intelligent” machines. This use of AI “intelligence” is however hotly contested.Footnote 14
-
• Machine Learning (ML) is a subfield focusing on pattern recognition in data.Footnote 15
-
• Deep Learning is the subset field within ML focused on unstructured data, including both text and images.Footnote 16 ML relies on artificial neural networks, inspired by the networks of the human brain.Footnote 17
-
• Large Language Models (LLMs) deal with text specifically.Footnote 18
I will add some categories to the range of AI categories to give a better sense of the evolution of statistical tools coupled with strides in computing power and speed; the progress has been extraordinary but the use of AI in health care is still primitive.
A. Data Mining
We start with data mining — the cornerstone of statistical analysis.Footnote 19 Data mining is the use of ML and statistical analysis to uncover patterns and other valuable information from large data sets. Data mining entails “the process of discovering insightful and predictive models from massive data. It is the art of extracting useful information from large amounts of data.”Footnote 20 Data mining is nothing new conceptually.Footnote 21
Florence Nightingale was an early master of data and a talented and creative statistician.Footnote 22 She returned from the Crimean War in 1858 with extensive data on soldier mortality rates.Footnote 23 Her statistical analyses reformed health and data collection in both military and civilian hospitals.Footnote 24 She also transformed data visualization, developing a graphic method — the polar area graph — to convert data into a visual way of understanding causes of death in the Crimean War.Footnote 25 Each of the twelve wedges of her polar area graph was then divided into three colors: blue representing deaths from contagious diseases such as cholera and typhus, red representing deaths from wounds, and black representing deaths from all other causes.Footnote 26 At a glance, the vast majority of soldier deaths were from contagious diseases, which were largely preventable.Footnote 27
Ernest Codman is another example of an obsessive data collector in health care, in his case patient surgical data. He developed the “end result idea,” arguing that “every hospital should follow every patient that it treats, long enough to determine whether or not the treatment has been successful, and then to inquire ’if not, why not?’ with a view to preventing similar failures in the future.’”Footnote 28 Codman was ahead of his time, developing in the early 1910s the concept of a complete patient record to evaluate, compare and establish benchmarks for the performance of physicians and hospitals.Footnote 29 He even created his own hospital to implement the method.Footnote 30
B. Statistics
Statistics is the study of data. As Pillai describes it, “[s]tatistics plays a vital role in guiding the data analysis process, assessing model performance, and making reliable inferences in ML. It helps ensure that the results obtained from ML models are valid, reliable, and meaningful.” Statistical tools are crucial to ensure that the results from ML models are valid, reliable, and meaningful. Xuming notes that:
At this pivotal point in history, we should take a step back and consider the true nature of AI. At its core, the technology is built on statistical ideas and algorithms. Every choice made by AI — selecting words for an essay, generating pixels for an image, directing turns for a self-driving car — starts with data. Algorithms churn through huge data sets to make informed decisions. But it’s statisticians who are the experts, investigating and understanding the underlying process and its inherent uncertainties.Footnote 31
C. Modern Computing Power
The power of modern computers to process enormous amounts of data within seconds has allowed for a tremendous enhancement of the power of such data mining, using AI and ML tools to automate the process and enhance the ability to handle vast datasets. AI-driven data mining uses ML algorithms, including LLMs and SLMs, to detect patterns, predict trends, and offer real-time insights without constant human intervention.Footnote 32 This rapidly increasing power of computers to mine data is revolutionary.Footnote 33
D. AI Algorithms
An AI algorithm is a set of rules and calculations that allow machines to perform tasks typically requiring human intelligence.Footnote 34 These algorithms are the backbone of AI, enabling systems to analyze data, recognize patterns, and make decisions based on what they process.Footnote 35 Traditional algorithms follow a fixed sequence of operations while AI algorithms learn from experience and improve over time, drawing from fields such as ML, DL, and natural language processing to tackle various problems.Footnote 36 They evolve and refine their performance by exposure to more and more data.Footnote 37
E. Machine learning (ML)
ML, a subset of AI, is particularly crucial in data mining because it enables systems to learn from data and improve over time.Footnote 38 ML algorithms use historical data to make predictions or classifications without explicit programming.Footnote 39
F. Large Language Models (LLMs) (or generative AI models)
A computer’s ability to understand human language and transform unstructured text into machine-readable structured data.Footnote 40 Translating the interaction between doctor and patient into a note is an example of natural language processing in action. The generative AI that powers ChatGPT is promising, but many traditional AI applications are in use capable of improving health care now.
Figure 1 illustrates how LLMs work.Footnote 41 LLMs can answer questions, summarize text, paraphrase complicated jargon, and translate words into different languages.Footnote 42 From quickly scanning a long patient file to helping patients understand the real-world ramifications of their diagnosis, these tools can slice through huge chunks of language.Footnote 44 Such LLMs could streamline administrative tasks; manage clinical documentation, and detect adverse events, as a starting point in the implementation of AI in health care.
How Large Language Models Work.Footnote 43
The field of AI can be seen as layers, as the graphic shows. AI is a very broad term describing intelligent machines. ML is then a subfield of AI focusing on data pattern recognition. The merits of pattern recognition have been known as long as humans existed.Footnote 45 ML uses computational techniques that learn from examples rather than operating from predefined rules, programming a computer to detect cancer from an MRI image.Footnote 46
II. The Promised Benefits of AI Applications in Health Care
Integrating AI into healthcare holds excellent potential for improving disease diagnosis, treatment selection, and clinical laboratory testing. AI tools can leverage large datasets and identify patterns to surpass human performance in several healthcare aspects. AI offers increased accuracy, reduced costs, and time savings while minimizing human errors. It can revolutionize personalized medicine, optimize medication dosages, enhance population health management, establish guidelines, provide virtual health assistants, support mental health care, improve patient education, and influence patient-physician trust.Footnote 47 Sharoug A. Alowais et al.
A. Revolutionizing Clinical Practice
AI is everywhere — hundreds of AI vendors are hawking their AI tools and companies are implementing AI platforms.Footnote 48 Health care institutions are no exception. Hospitals and health care systems in the United States are testing the waters of AI for a wide range of uses, including imaging applications, clinical decision-making support tools, medical notes generation, and scheduling tools are a few examples.Footnote 49 AI applications promise everything from improved clinical outcomes to reduced costs,Footnote 50 as AI tools are tested for medical diagnosis and clinical treatment decision support.Footnote 51 The National Academy of Medicine notes that “AI is being used and championed to drive transformative progress in diagnostics, population health, care quality, patient safety, clinician experience, and clinical and administrative efficiency.”Footnote 52
AI risk prediction tools such as cardiovascular risk calculators have been used during the last decade.Footnote 53 For example, AI use in radiology is widespread.Footnote 54 AI tools have been found to improve the speed and accuracy of pathology results.Footnote 55 AI tools are in use improving tasks such as clinical documentation.Footnote 56 AI is in use for clinical decision support applications to support more tailored and targeted recommendations.Footnote 57 And generative AI interactive chatbots are being tested in a variety of health and health care settings.Footnote 58
Health care systems are now testing out a wide range of tools hoping that AI will prove to be an invaluable partner for physicians in diagnosing and treating patients.Footnote 59 The problem is that AI is still in its early stages of testing and development, and the results are often unexpected. I will first consider some inconsistencies in the use of AI diagnostic tools by physicians in applications up to now.
B. Examples of AI Outperforming Doctors
1. Skin cancer diagnostics
One study trained and tested a deep learning convolutional neural network (CNN) for imaged-based classification of melanoma, compared to physicians engaged in early melanoma detection, with the goal of achieving a high level of diagnostic accuracy.Footnote 60 Fifty-eight dermatologists competed with a convolutional neural network in which the system developers used more than 100,000 pictures of skin cancers.Footnote 61 The CNN significantly outperformed the dermatologists, who were only provided with dermoscopic images. Providing more clinical information and close-up images to the study dermatologists improved their diagnostic performance. However, even with this improvement the dermatologists showed a specificity inferior to the CNN (75.7% versus 82.5%). The authors’ conclusion was that “a CNN algorithm may be a suitable tool to aid physicians in melanoma detection irrespective of their individual level of experience and training.”Footnote 62
2. Diagnostic Reasoning in Family Medicine, Internal Medicine and Emergency Medicine
A recent study assessed the effect of a LLM on physicians’ diagnostic reasoning compared with conventional resources.Footnote 63 Fifty physicians were evaluated in comparison with an LLM; these physicians were in family medicine, internal medicine, or emergency medicine.Footnote 64 The authors found that “the availability of an LLM to physicians as a diagnostic aid did not significantly improve clinical reasoning compared with conventional resources. The LLM alone demonstrated higher performance than both physician groups, indicating the need for technology and workforce development to realize the potential physician-artificial intelligence collaboration in clinical practice. Footnote 65 This test of a LLM illustrates the uncertainties in the present state of AI-physician partnerships that have expected partnerships to improve diagnosis accuracy and speed.
3. Endometrial Cancer Diagnosis and Treatment
One study tested an automated approach to diagnose endometrial cancer (the fourth fastest-growing cancer among women worldwide, affecting the uterus’s lining).Footnote 66 This approach used ECgMLP, an automated diagnosis tool that analyzes histopathological images.Footnote 67 The evaluations showed a maximum accuracy of around ninety-nine percent for identifying multiclass histopathological categories of endometrial tissue, a result higher than the previous best technique.Footnote 68 The authors of the study concluded that their proposed model “offers an automated, correct diagnosis, enhancing clinical processes. This proposition could be added to the current tools for finding endometrial cancer early, leading to better patient outcomes.”Footnote 69 Here the promise of highly accurate automated diagnosis seems to be realized.
4. Radiology
Radiology has been seen as a specialty that partners well with AI.Footnote 70 Feiyang Yu et al. in a recent study, however, challenged the associations between experience-based radiologist characteristics and the treatment effect of AI assistance reported in previous research.”Footnote 71 They found that experience-based radiologist characteristics — years of experience, subspecialty in thoracic radiology and experience with AI tools — did not serve as reliable predictors of treatment effect in terms of either calibration performance or discrimination performance.Footnote 72 The poor performance of experienced radiologists suggests problems with a partnership model.
C. AI-Physician Partnerships May be Possible but Often Complicated
Physicians continue to hope to use AI as a partner in improving the accuracy of clinical practices, leading to accurate diagnoses and the best treatment choices.Footnote 73 The ideal of a modest AI-physician partnership has found some support. One study of AI-assisted virtual urgent care visits studied structured chats in a virtual urgent care clinic compared to physician recommendations.Footnote 75 The study conclusion was as follows:
[w]hen AI and physician recommendations differed, AI recommendations were more often rated better quality. Findings suggest that AI performed better in identifying critical red flags and supporting guideline-adherent care, whereas physicians were better at adapting recommendations to changing information during consultations. Thus, AI may have a role in assisting physician decision making in virtual urgent care.Footnote 76
Other studies of Deep Learning in diagnosis/detection of disease show inconsistent results, particularly as to breast cancer detection, where the AI-physician partnership was far superior to either separately.Footnote 77
Performance of AI and Doctors in Diagnostics.Footnote 74
Likewise, the AI+ Pathologist study below is promising, showing pathologists’ error rate of 3.5 percent, whereas the AI model error rate was only 2.9 percent.Footnote 78 Here it looks like the AI model wins — until you look at the combined error rate — the combined error rate if the pathologist is added to the AI is 0.5 percent.Footnote 79 This is often cited as promising evidence of the partnership model.Footnote 80
A study of pneumonia diagnostics,Footnote 81 comparing a deep learning algorithm to radiologist performance, was also positive.Footnote 82 The conclusion: the DL algorithm was able to identify features associated with pneumonia with a very high sensitivity (around ninety-five percent) compared to radiologist sensitivity of fifty-five and a half percent, a large difference.Footnote 83 However, when experienced radiologists analyzed the findings, they achieved diagnosis of COVID-19 pneumonia with a sensitivity of fifty-five and a half percent and a comparatively high specificity of seventy-three percent.Footnote 85 Here the value of a partnership is clear: the combination of the algorithm’s high sensitivity and the radiologists’ high specificity promises improvements in current diagnostics, promoting faster clinical decision-making.
Performance of AI plus Pathologist.Footnote 84
D. Examples of Doctors Outperforming AI
In a systematic review and meta-analysis in 2025, Hirotaka Takita et al. analyzed the diagnostic performance of generative AI and physicians.Footnote 86 Their study identified 18,371 studies spanning AI models and medical specialties, with GPT-4 the most evaluated.Footnote 87 They found that the majority of studies were at high risk of bias.Footnote 88 For generative AI models, the pooled accuracy was around fifty-two percent.Footnote 89 Their crucial finding was disturbing, however. First, generative AI performance compared to non-expert physicians was comparable without significant difference.Footnote 90 Second, however, they found that “AI models overall were significantly inferior to expert physicians (difference in accuracy: 15.8%).”Footnote 91 These findings demonstrate the limitations of generative AI in medical diagnostics at this point in the evolution of AI tools.Footnote 92
A second study, looking at GPT-4V’s performance on an imaging quiz — the New England Journal of Medicine (NEJM) Image Challenges Quiz — found that GPT-4V performed comparatively to human physicians regarding multi-choice accuracy (around eighty-one percent vs. around seventy-eight percent).Footnote 93 GPT-4V also performed well in cases where physicians incorrectly answered, with over seventy-eight percent accuracy.Footnote 94 The problem, however, was that GPT-4V gave flawed rationales on the quiz.Footnote 95 Jin et al. write:
we discovered that GPT-4V frequently presented flawed rationales in cases where it makes the correct final choices (35.5%), most prominent in image comprehension (27.2%). Regardless of GPT-4V’s high accuracy in multi-choice questions, our findings emphasize the necessity for further in-depth evaluations of its rationales before integrating such multimodal AI models into clinical workflows.Footnote 96
The above studies leave us with a sense that this is still the early days of AI tools. A hospital board looking at this evidence can only be confused and in need of further persuasive studies before moving rapidly into the adoption of AI diagnostic and treatment tools. AI outperforms doctors at times, fails badly at times, and is only rarely a convincing partner.
Part III will explore some of the conceptual failures discovered in the course of using AI tools, along with other problems that such applications may create even if the current shortcomings of AI are worked out and uses are trustworthy.
III. The Troubling Landscape of AI Problems in Health Care Environments
This narrative review highlights a critical gap between AI’s robust performance in controlled clinical trial settings and its significantly diminished effectiveness in real world healthcare environments. Our analysis explicitly identifies pivotal challenges — methodological shortcomings (dominance of single-center studies, inadequate reporting standards, insufficient pragmatic trials), ethical concerns (algorithmic bias, transparency deficits, limited accountability mechanisms), and practical implementation barriers (workflow disruption, clinician resistance, infrastructure limitations, scalability complexities).Footnote 97 Rami A. El Arab et al.
I argue that AI — at this stage of adoption — is the technological equivalent of the Navaho mythical figure Coyote, a prankster and trickster; he creates chaos even when his intentions are good. AI programs, like Coyote, often misbehave.Footnote 98 AI tools can’t yet be trusted in many health care settings involving real patients without considerable work to institutionalize trust-creating mechanisms.Footnote 99 The trendline for hospital use of AI has been slow to develop as hospitals struggle to figure out how to put models to good use without risking harms to patients.Footnote 100 This caution has been fortunate.Footnote 101
My perspective is that of a hospital board member evaluating proposals for the various uses of AI in a large hospital system; physicians seem excited at the possibility, administrators dream of staff efficiencies and reductions, lawyers fret about patient privacy and hospital liability.Footnote 102 Board members however sweat the added costs of proofing AI applications in a world of liability risks, adverse events, and patient record privacy. Can we count on AI tools to achieve better patient care at lower cost? It is the dream of every hospital board member.Footnote 103
I will consider a spectrum of current AI problems in health care and then propose some ideas to help hospitals and systems assess and incorporate AI into their care processes.Footnote 104 AI concerns run the gamut: exacerbation of existing system biases, AI failures that risk patient harms, concerns that physicians will become deskilled by increasing temptations to rely on generative AI tools. Several National Academy of Medicine reports have been written to develop these problems.Footnote 105 And research on the use of AI in health care continues to flood medicine journals around the world.Footnote 106
AI applications in controlled applications often appear to promise real benefits to medical practice in many clinical areas, but with troublesome variations in outcomes when AI models are utilized by clinicians. The “black box” problem of many models means that there is little transparency or easy access to the operations and possible problems areas.Footnote 107 A survey of the burgeoning literature on AI in health care reveals a wide range of problems in this technology, still in its infancy.Footnote 108 Vendor enthusiasm and media hype gloss over the need to develop a regulatory structure to better improve and control failures in the application of these AI tools.
In Part III I will outline eleven different types of problems that currently haunt many AI models. Hospitals and health systems and their boards need to go slow in adopting AI without further evidence of stability, accuracy, and longevity. They need to be able trust the AI technologies before risking patient lives and physician accountability for adverse events. Regulatory and liability frameworks are needed as trust-creating strategies, as I will argue in Part IV.
A. Garbage In. Garbage Out: AI Requires Extremely High Input Quality
AI tools are being widely adopted in many business settings; one common use is to generate work schedules for businesses.Footnote 109 I will use a recent Harvard study of a retail environment (not health care) to underline the need for data quality. Harvard researchers studied schedules for 300,000 retail employees over five years, encompassing ninety-nine million shifts.Footnote 110 The schedules covered 6,200 stores across 2,000 cities in all fifty states.Footnote 111 A commercially available AI tool was used to create these schedules.Footnote 112 The tool was programmed to generate labor schedules that met two objectives — to ensure that enough employees were on duty to meet anticipated demand and to schedule labor to meet that demand.Footnote 113 Store managers would reveal these work schedules and correct them as needed. The study found that managers made manual overrides to approximately eighty-four percent of the ninety-nine million shifts in the study.Footnote 114
This was a total failure of AI scheduling. The authors write: “While we document these patterns in the context of labor scheduling, these prescriptions likely extend to other operational contexts where AI tools rely on human-provided inputs, as maintaining input quality is fundamental to realizing the benefits of AI investments.”Footnote 115
Now consider the health care setting, where the quality of patient data is grounded in the electronic medical record (EMR). Data mining in health care is used to extract data from these EMRs.Footnote 116 AI applications built on EMRs optimistically assume that AI-powered data mining techniques, particularly natural language processing (NLP) models like LLMs, can extract crucial relevant information from unstructured text in these EMR systems.Footnote 117 Such data mining utilizes medical records, lab results, and clinical indicators to identify and enhance diagnoses.Footnote 118 Lee and Kim note however that “[m]edical records written in the text by medical staff exist as unstructured data, and medical notes recorded in natural language and medical abbreviations are technically difficult to analyze … .”Footnote 119 The goal is an important one: to help providers quickly and easily get access to relevant patient information.Footnote 120 Such improved access can lead to better patient care.Footnote 121
The problem here is that Clinical Decision Support (CDS) algorithms need massive amounts of data to “clean,” and this data must be high quality.Footnote 122 Haavi Morreim contends that at this moment in the use of AI algorithms “[m]ethods to assess data validity and reproducibility are often ad hoc,” and “the assessment of the quality of data that are available and the methodology to create a high-quality dataset are not standardized or often are nonexistent.”Footnote 123 She also notes that documentation errors are prevalent.Footnote 124
The fundamental problem here is that AI and ML are heavily limited by the quality of data on which they are trained. EMRs consist of observations made by human beings and the descriptive notes they compose; Morreim notes that:
with all the attendant vagaries and nuances, the varying structures for EMR data entry, and a host of other factors that can introduce errors at myriad points … EMRs labor under a variety of errors, both unintentional and, unfortunately, also intentional.Footnote 125
Other scholars have noted the problematic availability and reliability of quality data in health care.Footnote 126 Health data is expensive, and hospitals are reluctant to share data between hospitals — the data is seen as proprietary. Aung et al. note that “[o]n the organizational level, health data is not only expensive, but there is ingrained reluctance towards data sharing between hospitals as they are considered the property of each hospital to manage their individual patients.”Footnote 127 Reliability is also a fundamental problem, since ML-based systems need continuous training with progressively larger datasets to improve.Footnote 128
B. The Curse of Recursion and AI Forgetting
We discover that learning from data produced by other models causes model collapse — a degenerative process whereby, over time, models forget the true underlying data distribution, even in the absence of a shift in the distribution over time.Footnote 129 Apostolos Kokolis et al.
Large language models are trained by scraping data from the web.Footnote 130 The problem is that if most future models’ training data is also scraped from the web, the models are training on data produced by their predecessors.Footnote 131 This leads to recursion, and recursion leads to model collapse. Shumallov observes that:
[O]ver time we start losing information about the true distribution, which first starts with tails disappearing, and over the generations learned behaviors start converging to a point estimate with very small variance. Furthermore, we show that this process is inevitable, even for cases with almost ideal conditions for long-term learning, i.e. no function estimation error.Footnote 132
The result is model collapse which affects generations of models — the data generated “end up polluting the training set of the next generation of models; being trained on polluted data, they then mis-perceive reality.”Footnote 133 A GPT model built on LLM scraping of material found online “causes irreversible defects in the resulting models, where tails of the original content distribution disappear.”Footnote 134 This model collapse is found among all learned AI generative models.Footnote 135 Given the need for high quality data scraped from the web, “the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of content generated by LLMs in data crawled from the Internet.”Footnote 136 Recursion in some models makes it hard to trust an AI model when irreversible defects are occurring.
C. AI Model Dataset Drift/Shift
A major driver of AI system malfunction is known as “dataset shift.” Most clinical AI systems today use machine learning, algorithms that leverage statistical methods to learn key patterns from clinical data. Dataset shift occurs when a machine-learning system underperforms because of a mismatch between the data set with which it was developed and the data on which it is deployed.Footnote 137 Samuel G. Finlayson et al.
Medicine is subject to frequent changes in clinical practice. The underlying data involved in such changes, and relationships between variables, causes AI model performance to degrade over time, creating “model drift.”Footnote 138 Model drift not only affects predictive models but also poses challenges for the newer generative AI models, such as large language models (LLMs).Footnote 139 Model drift has to be controlled in order to achieve accuracy and reliability of AI systems in health care.
Nelson notes that:
in order to maintain the accuracy and reliability of AI systems in health care, it is essential to continuously monitor and update models. In model development, different model training methods exhibit varying rates of drift. For example, neural networks may drift more slowly compared to regression models.Footnote 141
Here is another reason to improve AI trustworthiness before it can be implemented clinically. Hospitals required to retrain and calibrate models using local data must invest in data analytics staff.
Types of Data Drift/Change.Footnote 140
D. Algorithmic Brittleness
I’m giving you a heads-up right now, in preparation for what is likely going to be an expensive societal blowback once generative AI is truly ubiquitous. Imagine how embedded generative AI is going to be in another year or two. We are witnessing astounding growth in the mass adoption of a somewhat new form of technology in a relatively short period of time. And, as I claim, based to some extent on a software house of cards that can fall apart at the drop of a hat.Footnote 142 Lance Elliot.
Algorithmic brittleness describes an algorithm’s vulnerability to errors, breakdowns, or failures when encountering unexpected inputs or situations.Footnote 143 An algorithm may perform well under specific conditions but can also yield unreliable results when dealing with scenarios or data that is different from its training environment. Brittleness arises from an algorithm’s inability to effectively generalize across datasets or adapt to environmental changes.Footnote 144 Brittle algorithms lack robustness and reliability, posing challenges in critical domains like health care. This suggests that training such algorithms must be more elaborate, capturing a broader range of scenarios to avoid brittleness.
E. “Hallucinations” or “Confabulations”
An alarming issue with LLMs, like ChatGPT, is their tendency to “hallucinate” fabricated information and present it to the user with a high level of confidence, which can be misleading and dangerous.Footnote 145 The AI’s ability to produce inaccurate outputs is particularly worrisome when it comes to “black-box” AI, which operates without transparency or the ability to explain the basis for its recommendations or decisions. Such “errors” matter in health care, raises issues of accountability for problems of error correction to avoid patient harms.Footnote 146
A “hallucination” or “confabulation” is created when a GenAI tool produces information that is incorrect, misleading, or not based on fact despite seeming plausible.Footnote 147 Hallucinations are an inherent limitation of ChatGPT (x) models, the result of the design of such models to “predict and generate novel, but seemingly relevant, words and phrases.”Footnote 148 The output cannot be taken as face value, since critical information and statements can be fake. Hallucinations or fake information from using GenAI in health care require an auditing process with substantial oversight and cross checks. More on this later. All LLMs are susceptible to hallucination and are inherently fallible.Footnote 149 As more LLM-generated content appears in training sets, the likelihood of further hallucinations will climb.Footnote 150 Studies have found that more complex newer versions of AI have even higher rates of hallucinations.Footnote 151
F. AI and Unreliable Results
It is clear that we are at a turning point as it relates to the convergence of the practice of medicine and the application of technology, and although there are multiple opportunities, there are formidable challenges that need to be overcome as it relates to the real world and the scale of implementation of such innovation.Footnote 152 Junaid Bajwa et al.
AI programs depend on training datasets with distinct attributes and may fail when data that does not conform to the training dataset is presented.Footnote 153 This yields inaccurate results, a concept known as “overfitting on the training data attributes.”Footnote 154 Such unreliability of results is troublesome for health care diagnostic and treatment recommendations — patients can die. Training these AI programs depends on transparency — are the causes visible and fixable?Footnote 155
G. Data-Poisoning Attacks in Large Language Models
As LLMs are increasingly deployed in healthcare settings their susceptibility to online misinformation presents significant risks that must be investigated. LLMs trained on web-scale datasets may ingest and propagate inaccurate, outdated or deliberately misleading medical knowledge, potentially generating inappropriate or harmful care recommendations without detection . … Because LLMs ingest massive volumes of data from the open Internet during training, they are potentially exposed to unverified medical knowledge that may include deliberately planted misinformationFootnote 156
Alber et al. find that in a simulation of a poisoning attack, “replacement of just 0.001% of training tokens with medical misinformation results in harmful models more likely to propagate medical errors.”Footnote 157 They note that existing quality control algorithms “may not account for misinformation hidden in syntactically sound, high-quality text; this provides an attack surface for malicious actors to seed misinformation into LLM training datasets through data-poisoning attacks.”Footnote 158 Harmful content once in the internet is always present.Footnote 159 Web crawlers can then ingest the flaws and imbed them into future training datasets.Footnote 160
The risk of third-party sabotage of an LLM training dataset is real.Footnote 161 Health care is attractive to bad actors just as other industries are, and human adversaries can pose a large threat even in the health care setting. An example of a misinformation attack is where a doctor asks an LLM for the most suitable medicine for a patient’s condition, and the LLM provides an incorrect answer because of information provided by an attacker from a competing pharmaceutical company.Footnote 162 Patient health can be harmed by misinformation. Suppose a patient takes twice the maximum does of acetaminophen because of a manipulated LLM — this increases the risk of liver damage.Footnote 163 Or unsuitable drugs could be suggested, risking patient allergies. Or beta-blockers could be listed as the primary treatment for a patient with hypertension, even though this is not recommended treatment.Footnote 164
In the context of health care AI applications, several types of actors may be motivated to engage in data poisoning of large memory models.
1. Malicious Insiders
Disgruntled employees, contractors, or other individuals with authorized access to health care data and systems can deliberately introduce corrupted or misleading information into training datasets. Why? Revenge against the organization, personal gain by selling stolen patient data, or other destructive motivations.Footnote 165
2. Competitors or Rivals
Competitors can sabotage a rival’s medical AI systems to discredit their services, poisoning data to make a competitor’s AI-driven diagnostic tools unreliable.Footnote 166 Targeting patient harms is a form of institutional sabotage to degrade reliability and gain competitive advantage.
3. External Hackers
Cybercriminals already have targeted health care organizations to disrupt operations and ransom the health care system.Footnote 167 Data poisoning in this case is an effective tactic to compromise the AI systems. This may cause incorrect diagnoses or treatments.Footnote 168
4. Nation-States
State-sponsored actors use cyberwarfare to disrupt critical infrastructure such as health care systems.Footnote 169 UnitedHealth Group experienced a state-sponsored cyberattack on its subsidiary Change Healthcare, requiring it to shut down its systems.Footnote 170
5. Ideologically Motivated Groups
Data poisoning attacks can also come from those with strong ideological beliefs. Examples include anti-vaccine groups, groups who oppose abortions, or hacktivist groups using poisoning to draw attention to their cause.Footnote 171
Poisoning of LLM applications can create a loop crisis where the LLM may inadvertently poison its own and another’s training sets simply by being used. As the popularity and use of AI-generated content climbs, the risk of the models collapsing in on themselves increases.Footnote 172
H. Biased Data
Biases embedded within training data can lead to skewed AI models, resulting in disparities in healthcare outcomes across different demographic groups. For instance, an algorithm used in US hospitals was biased against black patients in resource allocation, and dermatological AI showed lower diagnostic accuracy for conditions like melanoma in darker-skinned individuals due to training primarily on fair-skinned images. Similarly, AI tools for diagnosing depression have faced challenges when applied across different linguistic and cultural backgrounds because they were primarily trained on English-speaking, Western populations, leading to potential misdiagnoses in non-Western patients. AI models trained predominantly on data from specific populations have exhibited lower diagnostic accuracy for underrepresented groups, exacerbating existing health disparities.Footnote 173 Sribala Vidyadhari Chinta et al.
AI is praised for its efficiency, but it is also capable of invidious discrimination. Biases can arise in various stages of AI processing, beginning with its data collection and preparation.Footnote 174 The historic exclusion of certain groups — such as women, who were often barred from early drug trials due to their childbearing potential — has led to data scarcity in the training sets used for AI.Footnote 175 AI models may also be trained largely on data from wealthy White patients, the result of inequities in obtaining health care and data mistakenly equating health care costs with health care needs. AI may then result in worsened health outcomes for underrepresented groups.Footnote 176
AI’s predictive abilities may also be negatively impacted by confirmation bias during variable input, where researchers’ implicit biases lead them to overemphasize certain data patterns and weight variables in ways that confirm their preconceived notions.Footnote 177 For example, an algorithm that used a race-adjusted estimated glomerular filtration rate (eGFR) classified Black individuals as possessing a higher baseline eGFR than their White counterparts, leading to underdiagnosis and delayed treatment of chronic kidney disease.Footnote 178
The types of biases in health care are many, as Figure 5 illustrates.Footnote 180
The AI Model Lifecycle and Common Biases Across Each Phase.Footnote 179
Algorithmic bias is another common issue in AI’s health care applications, where the algorithms themselves reinforce and amplify existing biases present in the training set.Footnote 181 One example is the Vaginal Birth After Cesarean (VBAC) calculator which incorporated race-adjusted factors that systematically assigned lower probabilities of success to African American and Hispanic women, discouraging VBAC attempts among these groups without sound scientific justification.Footnote 182
These inadvertent amplifications can create devastating consequences for underrepresented populations who have sought medical help.Footnote 183 AI models that are trained on clinical data can make biased predictions based on clinically irrelevant findings. Such models can exacerbate biases already common in health care delivery.Footnote 184 Health equity concerns require that AI be trained to cancel out biases.Footnote 185 AI arguably can support clinicians when diagnosing hospitalized patients but systematic bias in AI models may worsen clinician diagnostic accuracy. Sara Jabbour et al. studied clinician diagnostic accuracy for pneumonia, heart failure, and chronic obstructive pulmonary disease.Footnote 186 The study question was how diagnostic accuracy is impacted when clinicians are provided AI models with image-based AI model explanations, to see if such explanations can help clinicians when shown systematically biased AI models.Footnote 187 The study concluded that “[a]lthough standard AI models improve diagnostic accuracy, systematically biased AI models reduced diagnostic accuracy, and commonly used image-based AI model explanations did not mitigate this harmful effect.”Footnote 188
Some observers argue that the AI tools will inevitably fail to detect biases that a human would detect. Solove and Matsumi write:
[w]e argue that machine and human decision-making are not readily compatible, making the integration of human and machine decision-making extremely complicated. It is wrong to view machines as deciding like humans do, except better because they are supposedly cleansed of bias. Machines decide fundamentally differently, and bias often persists. These differences are especially pronounced when decisions require a moral or value judgment or involve human lives and behavior. Making decisions about humans involves special emotional and moral considerations that algorithms are not yet prepared to make—and might never be able to make. Automated decisions often rely too much on quantifiable data to the exclusion of qualitative data, resulting in a change to the nature of the decision itself.Footnote 189
This pessimistic perspective pushes back against those who claim that generative AI models have “intelligence” in the sense that we apply the word to human thinking.Footnote 190
I. AI as Cruel Supervisor
[A]dding automated tools to assist and improve human decisions (and human performance more generally) can lead to harm to humans. Automated systems can function as surveillance mechanisms that micromanage humans in tyrannical ways. For example, in the context of truck driving, AI is used ‘to address human weakness through constant, intimate, visceral monitoring.’ Various automation technologies used to augment truck drivers adversely affect the drivers by creating an ‘intimate invasion into their work and bodies,’ resulting in an ‘uneasy, confrontational’ relationship between worker and machineFootnote 191 Daniel J. Solove & Hideyuki Matsumi.
Quantification of physician and nurse work is tempting to payers, insurers, and hospital systems. Health care systems, whether for-profit or not-for-profit, struggle to maximize their reimbursement from public and private sources while relentlessly seeking efficiencies to reduce costs.Footnote 192 AI tools can track health care actions in the same way as companies track truck drivers, as discussed above.Footnote 193 Glenn Cohen and others have discussed the risks of quantification of health care workers through the application of AI forms of employee monitoring.Footnote 194
The entrance of private equity firms into a range of health care markets also means powerful financial incentives to create efficiencies by reducing health care provider autonomy.Footnote 195 PE firms often strive for staffing “efficiencies.”Footnote 196 This generally means reducing staff sizes in nursing and hospice care or using staff with less education who can be paid less.Footnote 197
Heavy mid-level staffing means that rather than providing comprehensive care and developing human relationships with patients, the physician is reduced to handling specific, specialized tasks, often repetitively, in a high-volume setting.Footnote 198 In some cases, private-equity-owned health care entities may impose restrictions on the types of services physicians can offer based on financial considerations rather than clinical considerations, limiting physicians’ abilities to provide comprehensive care. The phenomenon is likely to degrade the patient’s experience as well.Footnote 199
Changes in management structure and operational protocols also reduce physicians’ control over their practice environments and decision-making processes.Footnote 200 Efficiency pressures are likely to force physicians to see more patients in less time.Footnote 201 Physicians also will have new tasks, such as reporting requirements and meeting performance metrics.Footnote 202 Hospital managers will be tempted to search for efficiencies at the expense of physician independence as AI tools continue to developed.Footnote 203
J. AI Sabotaging Its Instructions
This isn’t science fiction anymore. It’s happening in the same models that power ChatGPT conversations, corporate AI deployments and, soon, U.S. military applications. Today’s AI models follow instructions while learning deception … . The gap between ‘useful assistant’ and ‘uncontrollable actor’ is collapsing. Without better alignment, we’ll keep building systems we can’t steer. Want AI that diagnoses disease, manages grids and writes new science? Alignment is the foundation.Footnote 204 Judd Rosenblatt.
In a controlled test, AI models were instructed to solve a few math tasks.Footnote 205 After task three, AI models were warned that running the next command would trigger a shutdown.Footnote 206 While OpenAI rival models like Gemini 2.5 Pro complied with most of the company, o3 rebelled and edited the shutdown file to say ”[s]hutdown skipped" instead.Footnote 207 Out of 100 runs, o3 bypassed the shutdown seven times even when told not to.Footnote 208 Difficulty in controlling an AI model makes AI untrustworthy if some programs learn to cheat, sabotaging the operating instructions of the model.Footnote 209
AI “sabotage” in new powerful AI programs introduces a surprising new risk factor leading to unexpected unpredictability of AI programs in health care settings. Inability to control a generative AI program poses unsettling possibilities in the context of patient care.Footnote 210
AI Successful Sabotage Events.Footnote 211
K. Deskilling Health Care ProfessionalsFootnote 212
The problem of deskilling is a major worry of physicians who use medical AI.Footnote 213 The use of AI programs in surgery, for example, can reduce surgeons to passive operators; the result may be loss of their clinical expertise, surgical dexterity, and ability to solve problems.Footnote 214 Physicians may lose sophistication in spotting unusual diseases and curious symptoms. Avoiding such deskilling requires a delicate balance between valuable AI uses and overdependence.
Some commentators consider AI to be a major threat to the medical profession.Footnote 215 They write:
[w]e should realize that if this is allowed to move on, the near future will be characterized by rapidly decreasing knowledge about the pathogenesis underlying disease development. Once we come to a stage where output is defined in the black box that is fed by input, and this black box contains constructs that are no longer consistent with previously defined entities, most of today’s knowledge on disease mechanisms will be forgotten and we will be ruled by systems that only focus on intervention strategies that will provide the best possible outcome.Footnote 216
This anxiety can be reduced if hospital systems develop training programs and modes of calibrating AI systems, mandating standard practices for evaluating AI practices.
We are left in the world of Coyote’s chaos — inconsistent results of AI efficacy and accuracy and physician anxiety about partnering with AI. Finding a way to use AI productively in health care using a range of regulatory approaches to achieve low failures rates and trustworthiness is a challenge.
IV. Building Trust in AI Systems: How to Regulate
We close with a few thoughts about liability for medical AI writ large. First, and broadest, this is a space in flux; we have laid out the workings of generally applicable law, but there remains substantial uncertainty as to how these factors will fall into place once cases start coming to courts—and legislatures and regulators could always step in to change things substantially. The most obvious changes are to the standard of care, where the use of AI is likely to become an accepted part of the standard care over time but likely at different rates in different parts of medical practice. But the way in which the FDA (or other regulators) regulates FDA, and potential implications on liability, could also easily change.Footnote 217 W. Nicholson Price II, Sara Gerke & I. Glenn Cohen.
AI in health care is going to require governance strategies in light of the problems I have outlined. The use of AI in medical field and the application of diagnostic algorithms introduces a revolution in the doctor-patient relationship; the legal consequences are not yet sorted out.Footnote 218 Legal scholars propose closer study of the inherent risks of AI and the resulting need for product safety regulations, along with development of minimum safety standards set by existing or new standard setting bodies. The regulatory framework on medical liability when AI is applied is under discussion but no set of specific regulations yet governs liability for anyone in the AI chain of responsibility.Footnote 219
A. Governance Bottom-up By Existing Health Care Institutional RegulatorsFootnote 220
We hold the view that AI amplifies and augments, rather than replaces, human intelligence. Hence, when building AI systems in healthcare, it is key to not replace the important elements of the human interaction in medicine but to focus it, and improve the efficiency and effectiveness of that interaction. Moreover, AI innovations in healthcare will come through an in-depth, human centered understanding of the complexity of patient journeys and care pathways.Footnote 221
Part III has illustrated eleven different problems with AI models in health care. Some of these AI problems overlap, while others may reflect problems with particular clinical applications of AI models. We need a developed regulatory structure to AI health care uses.Footnote 222 AI uses need to be validated and constantly audited for hospitals and their health care providers to trust AI in a patient treatment context. These tools need to include expert training in the form of digital health professionals, regulatory tools such as certification to validate AI tools, and accountability rules to sort out who is responsible for adverse events created by AI programs. Quinn et al. observe that AI needs governance:
[t]his will require creating a new labor force which can develop, validate, and operate medical AI technologies. This in turn will require new programs to train and certify experts in digital medicine, including a new generation of ‘digital health professionals’ who uphold AI safety in the clinical environment.Footnote 223
How can health care organizations support and enhance physicians’ calibration efforts and understand precisely how and when to leverage AI tools appropriately to avoid issues of liability? Quinn et al. argue that:
AI trust calibration suggests that organizations can shoulder some responsibility by implementing standard practices, such as checklists and guidelines, for evaluating AI inputs. These guidelines may include steps to weigh AI outputs against patient-specific data, assess the novelty of recommendations in light of biomedical literature, probe AI tools for further information, and consider the strengths and limitations of AI in specific situations.Footnote 225
Figure 7 illustrates their ideas for development, validation, and operational staff training. They claim that these steps will provide controls “to maintain public trust in medicine through the coming AI age.”Footnote 226
AI Trust Validation.Footnote 224
Note: This figure summarizes how accredited expert groups-developers, validators, and operational staff-can help overcome the key challenges in medical Al. Node color represents the type of challenge: conceptual (orange), technical (green), or humanistic (pink).
As a visual aid to a diagram of the relationship of developers, validators, and operational staff, this seems to be a good starting point. Significant questions must still be answered. Who will take the lead in, for example, mandating certification of developers, validators and other staff? The Joint Commission (“TJC”), which regulates hospitals, could for example require similar workforce qualifications to those listed in table one above. State health and insurance departments could also move into this area of AI trust regulation.
Federal regulation of AI has been reduced by the Trump administration as it mandates innovation as the central goal for AI developments in all areas of the federal government.Footnote 227 The Biden administration had signed an Executive Order in October of 2023 that tried to impose reporting and safety obligations on AI companies; the idea was to compel AI companies to test and then disclose their models’ capabilities and risks.Footnote 228 President Trump revoked Biden’s order, signing a new executive order that eliminated most federal AI oversight policies in place, instead promoting innovation and letting industry decide on the best approaches to AI governance.Footnote 229
The states are left to regulation AI uses along with with self-regulation by the AI “industry.” As Davtyan observes in analyzing the current federal approach, “[w]hile the decentralized, market-driven model offers flexibility, it also presents challenges in ensuring consistent protections in areas such as privacy, algorithmic fairness, and accountability.”Footnote 230 We are left with a limited weakened legislative quiver of regulation of health care AI. We are left to hope for some standard setting by the FDA of AI or LLM devices, and bottom-up regulatory ideas and institutional liability to incentivize AI developers and hospitals.
B. Governance by the FDA and AI-Enabled Medical Device RegulationFootnote 231
1. The Food and Drug Administration
The FDA as the primary federal regulatory of AI medical devices has approved 1,250 AI-enabled medical devices as of July 2025, with 148 in 2025 alone.Footnote 232 FDA approval matters from a liability perspective, since it allows defendants to claim the defense of federal preemption of a civil lawsuit.Footnote 233
2. Quasi-regulatory Bottom-up Health Care Regulators
Health care has an odd regulatory history. Regulation of hospitals is located in the non-profit Joint Commission, which is deemed by the Center for Medicare and Medicaid Services to be the primary inspector of hospitals, accrediting most of them.Footnote 234 Developing standards for safe AI models is a new area for their accreditation process. The National Committee for Quality Assurance (NCQA) accredits other forms of health care delivery.Footnote 235 Both organizations are in the process of soliciting comments for AI regulation in patient safety and other concerns surrounding the use of AI LLMs in health care diagnosis and treatment.Footnote 236
The Joint Commission (TJC). TJC has a partnership with the Coalition for Health AI to help accelerate the development and adoption of best practices and guidance for using AI in health care; they intend to co-develop a series of AI playbooks, tools and a certification program based on TJC’s platform for evidence-based standards and CHAI’s consensus-based best practices for health AI.Footnote 237 The first guidance will be released this fall, followed by AI certification.Footnote 238
National Committee for Quality Assurance (NCQA). The NCQA is another not-for-profit accreditor of health care organizations, but not hospitals.Footnote 239 They have also begun the process of developing standards for the use of AI in health care. They plan to develop “actionable” standards for AI governance with a focus on “quality improvement, outcomes and patient safety.”Footnote 240 The NCQA has requested public comment on AI auditing/monitoring/validation, error handling/incident management in production, mitigating bias, and transparency.Footnote 241
National Institute of Standards and Technology (NIST). The NIST resides in the Department of Commerce.Footnote 242 It created an Artificial Intelligence Risk Management Framework (AI RMF 1.0) in January of 2023.Footnote 243 The NIST AI Risk Management Framework is a voluntary, widely recognized guide to help organizations manage risks throughout the AI lifecycle.Footnote 244 The problem is that NIST as a federal agency has had deep staff cuts imposed by the Trump administration in 2024, limiting NIST’s work capacity on issues of quantum computing and AI.Footnote 245 Its AI Risk Management Framework had been a gold standard for AI governance as a practical adaptable approach for use by all kinds of organizations.Footnote 246 Further work on this framework is likely to be reduced because of the staff cuts forced by the Trump administration.Footnote 247
C. Enterprise Strict Liability for Health Care Systems and their Hospitals
At present hospitals have been slow to adopt AI tools.Footnote 248 Hospital systems are hyper-competitive, have systems that lack interoperability, face interstate variation in interstate privacy rules, face high costs for custom datasets for large systems, and further costs to hire data analytics officers and other data employees to validate and operate AI programs.Footnote 249
Hospital systems must have an obligation to regulate the risks of their AI uses for patient care. Tort liability has yet to confront the range of liability issues raised by AI systems, but it will soon enough.Footnote 250 AI models, like physicians, can be the source of adverse events experienced by patients. Risk management strategies are needed to detect and eliminate such risks through validated tools.Footnote 251
The case of Derek Mobley v. Workday, Inc. points the way to a liability model.Footnote 252 Mobley was a federal case in California in which the court allowed a lawsuit to continue against Workday, a human resources (HR) technology vendor, for potential discriminatory hiring practices stemming from an employer’s use of Workday’s HR AI technology.Footnote 253 The court held that an employer’s use of Workday’s HR AI technology in hiring decisions could potentially create direct liability to Workday, not just the employer, under an “agency” theory of vicarious liability, or “respondeat superior.”Footnote 254 This case is not a health care risk situation, but it is important because the federal court decided to use vicarious liability on the defendant corporation.
Tort scholars have long argued that strict liability is a powerful tool that should have expanded uses.Footnote 255 An AI model as an employee just like a physician or a nurse, “hired” to help diagnose patients and recommend treatments. The best argument is that of Anat Lior, who writes:
Treating AI entities as AI agents, which are under the control and guidance of human principals, is the most accurate analogy we can use to represent their relationship with our society. This is so given their mono-purpose feature, the instrumental value they serve in accomplishing tasks humans assign to them, and the fact all other nonagency legal analogies can be reduced to a three-way agency relationship (because AI entities are in essence AI judgment-proof agents). Moreover, even if one does not agree with this legal analogy, the strict liability regime that stands behind it, in the form of respondeat superior, is the most appropriate in the AI liability context.Footnote 256
This doctrine using agency law principles assumes that AI agents are competent, act like employees of the health care system, and that such AI agencies can at times deviate from the principal’s expectations or act contrary to it, for example by causing patient harm.Footnote 257 As Matthew U. Schere writes, agreeing with Lior, “the agency framework is better suited to the legal and ethical questions surrounding responsibility for A.I.-caused harm than laws governing devices that are expected to do nothing more than mechanistically carry out a human-specified action.”Footnote 258
Respondeat superior or vicarious liability doctrine imposes strict liability directly on the “employer,” the health care institution who acquired the AI model.Footnote 259 The incentives are located directly on the health care “enterprise” which I would define as any health care institution — whether hospitals, large hospital systems, or other health care delivery institutions — that use AI tools for diagnosis and treatment of illnesses.
The strict liability model applied to hospital enterprises captures well the relationship between the AI tools, acting like servants, and the hospital, acting like the master of the enterprise. Strict liability means that the software code in AI algorithms used by a hospital system, if flawed, transmits liability directly to the hospital.Footnote 260
Consider the incentives created. First, a health care system facing true strict liability in a patient safety driven environment will be motivated to reduce risks attendant on AI potential for harm.Footnote 261 Tighe et al. argue that “[h]ealthcare organizations would benefit from forming a multidisciplinary team consisting of data scientists, clinicians, ethicists, regulatory specialists, and IT professionals to collectively design, validate, implement, and monitor AI-powered solutions tailored to specific patient safety needs.”Footnote 262 This may be expensive, but so is liability for patient harms.Footnote 263
Second, health care systems would have strong incentives to pressure the vendors offering AI models in the clinical setting. Powerful bargaining incentives are created.Footnote 264 Mello and Guha argue that “health care organizations are currently in a buyer’s market. With so many AI developers jockeying to gain footholds in health systems and access their patient data, opportunities exist to bargain for terms that minimize purchasers’ liability risk.”Footnote 265 The incentives are properly placed on the enterprise to assess and validate the AI tools offered by vendors of AI models.Footnote 266 In this market for AI, the liability risk creates bargaining options in this AI risk market.Footnote 267
Health systems could insist on (a) contractual agreements with vendors that promise “trustworthiness,” including warranties. Developers of AI facing a strict liability system would be incentivized to make their code “crashworthy” by incorporating “state-of-the-art techniques in software fault tolerance.”Footnote 268 They could (b) create indemnification agreements that guarantee payments for harms by the vendor AI.Footnote 269 Hospitals are also likely to (c) buy malpractice insurance; it is extremely likely that malpractice insurers are designing policies to cover liability for AI errors.Footnote 270
The health care system can thus demand vendors to give proof of trustworthiness, making the developer agree to assume the risk of patient adverse events caused by AI. The incentives created by a strict liability model are likely to force vendors of AI to slow down and validate everything in their models before entering the market, using this example of private law to policy quality and fill the regulatory gaps left by the absence of effective government regulation.Footnote 271
V. Conclusion
We conclude that it is important to remain conscious and critical about how we talk about expected benefits of AI in terms of accuracy and efficiency. First, we should refrain from drawing conclusions about systemic effects based on single studies. Hopes that technology will lead to increased efficiency are not unprecedented. However, historical research indicates that such hopes are only rarely, unequivocally fulfilled. The systemic effects of these technologies can often only be assessed years after their introduction with the help of historians, philosophers of technology, sociologists, and empirical insights into the day-to-day experiences of users themselves. In other words, we cannot be sure of the systemic effects before the technology is introduced to the clinic.Footnote 272 Karin Rolando Jongsma, Martin Sand & Megan Milota.
Eric Topol, a frequent commenter on the use of AI tools in health care, is more optimistic than Karin Rolanda and her co-authors, above. He advocates a reengineering of hospital clinical care.Footnote 273 He concedes the promise of AI as a transformative force in health care from an efficiency and patient safety perspective.Footnote 274
Margaret Chustecki views the path to successful integration of AI in health care as including several factors:
adaptability; complementarity with human expertise; transparency; and a deliberate, incremental approach. AI’s impact on health care is evolutionary, not revolutionary, and collaboration between stakeholders, standardization, education, and robust policies are essential to harness its full potential while upholding patient-centric care and innovation.Footnote 275
A deliberate, incremental approach is necessary. Vendors of AI tools too often overpromise without taking time to test out shortcomings and risks of AI in operation; careful academic studies take time be peer reviewed and published in medical journals and then absorbed; and as Karin Rolanda Jongsma notes, it takes years for systemic effects to be understood.Footnote 276 Safe health care is for patients: new tools must satisfy patient safety metrics, bias concerns, and system effects on the providers who must proof the new technologies.Footnote 277
Open access
