1. Introduction
A simple search on ssrn.com on the topic large language models (LLMs) returns more than 400 results only for 2024. Many papers focus on the technical aspects of LLMs, to highlight glitches and potentials of these artificial intelligence (AI) models that have had such a strong recent development: these contributions put the focus on innovation aspects (Diab, Reference Diab2024; Moreau et al., Reference Moreau, Prandelli and Schreier2023). Other papers analyze the risks, challenges or benefits that the use of such models may have in different areas of application or on our world at large: these papers put emphasis on the impact of these models on contemporary society (Ferrara, Reference Ferrara2024; Novelli et al., Reference Novelli, Casolari, Hacker, Spedicato and Floridi2024; Pagallo, Reference Pagallo2022; Wachter et al., Reference Wachter, Mittelstadt and Russell2024). Still other papers focus on the ability of LLMs to reshape traditional activities, jobs or professional fields: these contributions highlight the transformative power of these technologies (Fagan, forthcoming, Reference Fagan2025; Nelson, Reference Nelson2024; Surden, Reference Surden2023). Finally, some papers focus on the ability of such models to change our perception and understanding of reality, creating new pieces of knowledge: these contributions examine the epistemological impact of LLMs (Delacroix, Reference Delacroix2024; Krook, Reference Krook2024).
This wide range of topics indirectly shows that LLMs potentially have a strong impact on every aspect of reality, with innovative and transformative effects raising risks, challenges and opportunities that require to be increasingly addressed not only from a broad normative angle (ethical, legal, political, economic, social, etc.) but also from a deep epistemological perspective, as they change the way we grasp and know our own reality. This raises a complex and thorny governance problem, since law has traditionally been used to regulate behaviors and legally relevant consequences arising from behaviors (whether the product of human actions or, increasingly, also of actions enacted by artificial agents) but not models (and particularly AI models) that affect the perception and understanding of reality or the production of knowledge.
Consider, for example, the difficulties faced by the European or American lawmakers in legally defining and regulating such new AI models. The European regulation on artificial intelligence (AI Act, hereinafter)Footnote 1 defines AI system in terms of
a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments (Art. 3(1) AI Act).
However, the considerable developments in the field of AI have prompted the European lawmakers to include in Art. 3(63) AI Act a further definition of the so-called “general-purpose AI models,” defined as follows:
an AI model, including where such an AI model is trained with a large amount of data using self-supervision at scale, that displays significant generality and is capable of competently performing a wide range of distinct tasks regardless of the way the model is placed on the market and that can be integrated into a variety of downstream systems or applications, except AI models that are used for research, development or prototyping activities before they are placed on the market.
US have taken a distinct approach in the Executive Order No. 14110 on the safe, secure, and trustworthy development and use of AI, signed by the US President Joe Biden on 30 October 2023,Footnote 2 proposing a definition of the term AI,Footnote 3 another of the AI model, Footnote 4 and a third one about the AI system. Footnote 5 Furthermore, the §3(p) of the Executive Order also includes a specific mention of generative AI, defined as follows:
the class of AI models that emulate the structure and characteristics of input data in order to generate derived synthetic content. This can include images, videos, audio, text, and other digital content.
The approaches taken by the US and the EU are different. In the former case, generative AI (Gen AI) is explicitly mentioned, whereby LLMs are the best known and widely used example. In the latter case, the EU legislator ultimately decided not to make explicit reference to Gen AI but instead to include the general-purpose AI models, an example of “Large generative AI models,” as specified in recital 99 of the AI Act.Footnote 6 A specific definition of LLMs has not been attempted, although the consequences arising from their ability to automatically generate “content in response to prompts written in natural-language conversational interfaces” (UNESCO, 2023, 8), investigated in this study, fall within the scope of the AI Act. This issue is often addressed through a normative approach, which aims to identify the advantages and disadvantages of applying a specific technology.
However, the normative challenges of LLMs require us to adopt an epistemological approach, which calls for examining the main differences between the representation of reality of human beings and that of LLMs. This epistemological dimension underlies the way LLMs operate in practice and poses challenges to regulation: it is therefore crucial to preliminarily address the problem from an epistemological point of view, as observed and extensively expounded by (Wachter et al., Reference Wachter, Mittelstadt and Russell2024) in a systematic study on the subject.
Therefore, the paper focuses on the epistemological approach behind LLMs (section 2), wondering what representation of reality LLMs produces and how we interact with this representation. Five epistemological challenges of LLMs are identified in terms of qualification (section 3.1); reliability (section 3.2); pluralism and novelty (section 3.3); technological dependence (section 3.4) and relation to truth and accuracy (section 3.5). In light of these epistemological challenges, the social implications of LLMs are framed, proposing some final remarks (section 4).
2. Epistemological approach
LLMs produce outputs in forms of answers to prompts, which can be understood as pieces of knowledge (Kelemen & Hvorecký, Reference Kelemen and Hvorecký2008; Lenat & Feigenbaum, Reference Lenat and Feigenbaum1991; Lenat & Marcus, Reference Lenat and Marcus2023), offering a specific representation of reality, because of “the human tendency to attribute meaning and intent to natural language” (Wachter et al., Reference Wachter, Mittelstadt and Russell2024, 3). From a qualitative point of view, the LLM’s representation of reality may accidentally reach high standard and, as a result, discerning between synthetic and non-synthetic outputs becomes challenging for humans. Accordingly, apart from normative considerations, it becomes essential to scrutinize the representation of reality offered by LLMs and how human beings become subjects of interaction, since people increasingly rely on AI-generated outputs, which are produced by AI models, trained on ever larger dataset and endowed with ever-increasing computational power sparking significant consequences on human life (Durante, Reference Durante2021, 13).
From an epistemological perspective, LLMs and humans are indeed different. While human beings are extraordinary “semantic engines” (their understanding and representation of reality is rooted in semantic skills, i.e., the ability to give meaning and sense to the world), a LLM is an extraordinary “syntactic engine” (Floridi, Reference Floridi2023a, 44), which predicts the most likely string of words that comes next in a piece of text in response to a prompt, operating on the basis of the ability to process huge amounts of data and parameters through structures of great complexity, such as neural networks, without possessing knowledge or awareness of what is being processed. Unlike the scientific method, where knowledge entails verifiable predictions based on models and explanations, AI-generated outputs lack such theoretical grounding. Thus, questioning the underlying functioning of LLMs is pivotal due to its profound impact on human actions and the criteria for justifying them.
In doing so, the analysis is limited to considering the comparison with a specific epistemology, namely the legal epistemology. Without going into the age-old and thorny question of the definition of law and its role, we can say that, from an epistemological standpoint, law acts as a normative system, which interprets and applies legal statements within a shared legal framework, verifying the reliability of this interpretation and application through forms of reasoning and specific tools (principles, categories, evidences, review systems, etc.) that the law itself provides and regulates, which have a fundamental common feature. Every piece of knowledge produced by the interpretation and application of the law is offered to the scrutiny, control and discussion of a counterparty and a third party (and more widely to the community of legal scholars over time).Footnote 7 Legal science is progressive and incremental and requires the contribution of all legal experts. This method, and the resulting representation of reality that it conveys, may differ from or be in contrast with LLM’s representation of reality, for three main reasons.
First, the representation produced by LLMs may happen to be accurate (where there is a validated standard or metric to measure accuracy) but it is not based on the ability to attribute meaning and intersubjectively verify its epistemic reliability. In analytical terms, LLMs (and more widely Gen AI) do not have the ability to operate a semantic ascent (Floridi, Reference Floridi2023a; Quine, Reference Quine1953) or descent in relation to artificially produced sentences; namely, they are not able to move from the linguistic to the metalinguistic level in order to qualify the content of their statements or argue in favor of its accuracy (as remarked with regards to general-purpose LLMs, “despite their generality, responses rarely include linguistic signals or measure of confidence” [Wachter et al., Reference Wachter, Mittelstadt and Russell2024, 2, also referring to Lin et al., Reference Lin, Hilton and Evans2022; Mielke et al., Reference Mielke, Szlam, Dinan and Boureau2022]).
Second, the law claims to define the extent to which an explanation is legally relevant both in terms of causal regression (i.e., determination of the chain of events of a legally relevant fact) and in terms of prediction (of the legally relevant consequences of the fact in question). Against this epistemological background, the law has always independently decided where it intends to stop the legally relevant retrospective or predictive explanation of an event. LLMs cannot account for a causal retrospective or predictive explanation of any knowledge it produces. How could the law, from this perspective, regulate an event if it cannot appraise and determine where (the knowledge of) its origin and consequences begin and end? (Durante, Reference Durante2021, 9–10).
Third, each piece of knowledge created by LLMs does not require the existence of a specialized community of interpreters to be concretely applied, while law is a social product and cannot give rise to progressive and incremental knowledge except through the contribution, discussion, interpretation and scrutiny by a community of experts (Zeno-Zencovich, Reference Zeno-Zencovich, Resta and Zeno-Zencovich2023, 445). While legal epistemology is focused on a shared process of representation and validation, LLMs propose solely the product of output, leading to concerns regarding both reproducibility and control. In particular, the aspect of control is central to the extent that “whoever controls the questions controls the answers and whoever controls the answers control reality” (Floridi, Reference Floridi2023b, 5).
Consider, for instance, the ability of ChatGPT to reiterate a given answer, potentially offering a very different response, sometimes totally contradicting the former. Variable outputs stress that LLM’s representation of reality needs mechanisms ensuring accountability. Furthermore, LLM’s inability to attribute meaning or validate the reliability of its outputs raises crucial questions about its role in producing pieces of knowledge on which decision-making is grounded. Nevertheless, this should not be a limitation for the study of potential opportunities arising from the use of LLMs in legal settings (as discussed in Fagan, Reference Fagan2025; Nelson, Reference Nelson2024; Surden, Reference Surden2023).Footnote 8
Investigating the epistemological dimensions of LLMs involves broader philosophical inquiries regarding knowledge acquisition, representation and validation. Understanding how to balance LLM’s production of outputs and representation of reality with human interpretation becomes crucial. What safeguards can be implemented to mitigate the issues emerging with LLMs from an epistemological standpoint? These issues require a multifaceted approach that recognizes the epistemic abilities and limits of LLMs. By critically examining some assumptions and implications of the production of outputs and the representation of reality of LLMs, the study paves the way for a more informed and coherent approach to its development and deployment. Therefore, a set of epistemological challenges have been identified and will be discussed below.
3. Epistemological challenges
AI has been described in terms of “epistemic technology,” where the term epistemic “is meant to refer to the broader category of practices that relate in a direct manner to the acquisition, retention, use and creation of knowledge” (Alvarado, Reference Alvarado2023, 12). The epistemic nature of AI derives both from the type of content it manipulates and the types of operations it performs on that content, but also for “its own internal capacities” (Alvarado, Reference Alvarado2023, 15). However, being an epistemic technology does not mean that the content or representation of reality provided is reliable. In other words, “accepting AI as an epistemic technology does not mean that we must also accept that it is an epistemically trustworthy epistemic technology” (Alvarado, Reference Alvarado2023, 14). For these reasons, “the need to involve challenging epistemic and methodological scrutiny is even more immediately pressing” (Alvarado, Reference Alvarado2023, 22). In doing so, five major epistemological challenges need to be analyzed: (1) qualification; (2) reliability; (3) pluralism and novelty; (4) technological dependence and (5) relation to truth and accuracy.
3.1 Epistemic qualification
The qualification challenge consists in the fact that LLMs defy the human ability to qualify the epistemological status of outputs (i.e., the answers and pieces of knowledge) conveyed. This occurs primarily in two ways.
First, LLMs tend to elide the distinction between what is created (i.e., which is the outcome of a proper creative process) and what is merely reproduced and mirrored (i.e., which is the result of the ability to mimic a creative activity). If one considers only the result and not the process, human beings are often unable to distinguish, for instance, the synthetic from the non-synthetic result. In the legal context, it remains crucial not only to be able to evaluate a piece of knowledge but also to be able to control the process by which a piece of knowledge is produced (Tasioulas, Reference Tasioulas2023, 9).Footnote 9
Second, LLM challenges the human ability to qualify between true and false. Considering the statistical dimension of LLM’s outcomes, pieces of false knowledge (the so-called hallucination, Ji et al., Reference Ji, Lee, Frieske, Yu, Su, Xu and Fung2023; Rebuffe et al., Reference Rebuffe, Roberti, Soulier, Scoutheeten, Cancelliere and Gallinari2022) are frequently generated however plausible: “Readers and consumers of texts will have to get used to not knowing whether the source is artificial or human. Probably they will not notice, or even mind” (Floridi & Chiriatti, Reference Floridi and Chiriatti2020, 691). For example, LLMs provide answers, which they are unable to reflexively qualify as to their degree of truthfulness or reliability. Where questioned in this regard, human beings can reflexively evaluate a statement of their own, qualifying it as certain, probable, plausible, etc. (using expressions, which precede the statement itself, such as “I am certain that,” “I believe that,” “it is highly probable that,” etc.). Any answer given by an LLM should, in this sense, always be preceded by an expression such as: “It is only probable that …,” in order to reveal that any AI-generated statement cannot constitutively raise any claim to truth (UNESCO, 2023).Footnote 10
It is a crucial matter, since, as argued by Stefano Rodotà, long before the emergence of LLMs, “the meaning of truth in democratic societies […] presents itself as the result of an open process of knowledge” (Rodotà, Reference Rodotà2012, 225 [our translation]), remarking that “In a society omnivorous of information, and continually productive of representations, the ‘truth’ of the latter takes on a special significance” (Rodotà, Reference Rodotà2012, 221 [our translation]). This is even more relevant considering the so-called “feedback loop:” “[…] Large Language Models learn from content that is fed to them, but because content will be increasingly wholly or partially AI-generated, it will be partly learning from itself, thus creating a feedback loop” (van der Sloot, Reference van der Sloot2024, 63). This feedback loop leads to the crystallization of LLMs and Gen AI’s representation of reality, with serious risks where it produces distorted or unreliable knowledge.
3.2 Epistemic reliability
The challenge of epistemic reliability represents the problem of human ability to rely on the answers and representation of reality conveyed by LLMs, which reiterates patterns identified starting from its training dataset, without having access to real-world understanding and knowledge. As has been pointed out more generally with regards to AI-generated output, this
can lead teachers and students to place a level of trust in the output that it does warrant. […] Indeed, Gen AI is not informed by observations of the real world or other key aspects of the scientific method, nor it is aligned with human or social values. For these reasons, it cannot generate genuinely novel content about the real world, objects and their relations, people and social relations, human-object relations, or human-tech relations. Whether the apparently novel content generated by Gen AI models can be recognized as scientific knowledge is contested (UNESCO, 2023, 16).
In addition, it is worth emphasizing that with regard to the debate on scientific and objective knowledge, the results of LLMs may not even be consistent with the main characteristics and principles of the open science policies and the FAIR (i.e., Findable, Accesible, Interoperable and ReusableI)data requirements (Paseri, Reference Paseri2023, Reference Paseri2022). This can raise three main consequences.
First, this produces an impact not only on epistemic trust but, precisely because of the inaccuracy, opacity and inexplicability of its model, also on social acceptance. In this perspective, it is indeed crucial to note that: “Assessing the likelihood or the accuracy of such outputs will depend on how well we think the model analyzed the training data. Hence, epistemic reliability of the analytical processes will be relevant […] to the trust we can allocate to AI generated text, search, etc.” (Alvarado, Reference Alvarado2023, 22). Hence, it follows that this epistemic limitation “is also a key cause of trust issues around Gen AI (Nazaretsky et al., Reference Nazaretsky, Cukurova and Alexandron2022a). If users don’t understand how a Gen AI system arrived at a specific output, they are less likely to be willing to adopt or use it (Nazaretsky et al., Reference Nazaretsky, Cukurova and Alexandron2022b)” (UNESCO, 2023, 15).
Second, this has serious consequences for the allocation of responsibility. In an ever-increasing situation of deep intermingling between human and artificial collaboration in content generation, “New mechanisms for the allocation of responsibility for the production of semantic artefacts will probably be needed” (Floridi & Chiriatti, Reference Floridi and Chiriatti2020, 692). Consider, for instance, in this regard the growing debate in the field of scientific research, which is not only about “reproducibility and robustness in their results and conclusions” (European Commission, 2024, 6) but also about the very authorship of the content produced by LLMs.
Third, this has a considerable environmental impact on the infosphere (Floridi, Reference Floridi2014, 205–216). As in the case of an upstream polluted source, flawed AI-generated content can not only spread polluted content downstream, which is inextricably mixed with trustworthy and genuine content, but in the long run it also addicts end users, lowering their defenses.Footnote 11
3.3 Epistemic pluralism and novelty
Every authentic cognitive inquiry rests on the ability to question. There is nothing more difficult in the search for knowledge than to construct the relevant and pertinent questions, from which any real investigation can start. However, for an investigation to be fruitful and satisfy our need for research, it is necessary that the answer be obtained within the boundaries set by the question, but it is equally necessary that, within those boundaries, the answer brings something new, which is not already included in the prompt of the question; otherwise, the inquiry remains tautological. Moreover, any authentic answer, which gives us access to the knowledge of something new, raises new questions, that is, the possibility of questioning reality from different standpoints, ensuring pluralism of research and knowledge: “Each piece of semantic information is a response to a question, which, as a whole, raises further questions about itself, which require the correct flow of information to receive adequate response, through an appropriate network of relationships with some information source” (Floridi, Reference Floridi2013, 274).
What puzzles us about LLMs is that the ability to produce pieces of knowledge by predicting probabilities of language patterns found in data is based on a static training dataset that describes the world as it is, “assuming what we know about the observed world” (Pearl & Mackenzie, Reference Pearl and Mackenzie2018, 9; Vallor, Reference Vallor2024). This is likely to reaffirm the primacy of the existent over openness to novelty and pluralism in a more or less surreptitious manner. In this way, the knowledge and representation of reality produced by LLMs and Gen AI at large is not instrumental in imagining new and different worlds but rather in reaffirming worlds already entrenched in traditional views and conventions. As properly remarked, “Gen AI, by definition, reproduces dominant worldviews in its outputs and undermines minority and plural opinions” (UNESCO, 2023, 26). This brings about two main consequences, according to UNESCO’s view of generative AI, which are relevant to epistemology and democracy, respectively. First, “Accordingly, if human civilizations are to flourish, it is essential that we recognize that Gen AI can never be an authoritative source of knowledge on whatever topic it engages with” (UNESCO, 2023, 26). Second,
Data-poor populations […] have minimal or limited digital presence online. Their voices are consequently not being heard and their concerns are not represented in the data being used to train GPTs, and so rarely appear in the outputs. For these reasons, given the pre-training methodology based on data from internet web pages and social media conversations, GPT models can further marginalize already disadvantage people (UNESCO, 2023, 17).Footnote 12
This last remark raises – or rather reiterates – a crucial concern for democracy. No matter how democracy is defined, it is generally agreed that people and populations should be able to modify their starting conditions. Like other technologies, LLMs can increase or emphasize starting conditions, favoring those who are already advantaged and disfavoring those who are already at a disadvantage. The risk is that LLMs become the battlefield where the challenges of innovation may be won by a few (also in the field of legal professionFootnote 13), and where conformity, division and social exclusion may also develop and spread, jeopardizing equity and pluralism in democracy.
3.4 Technological dependence
Technological dependence resulting from the increased or systematic use of LLMs, which are entrusted with tasks of producing knowledge, or even just aiding or supporting human activities, raises at least three different types of issues from an epistemological point of view, which add to the already stressed reliability problem.
First, technological dependence stresses the importance of measuring and comparing the outputs produced by human activity or AI, raising the need of standards for assessing the outputs of LLMs. On the one hand, recourse to such systems should not be discouraged in areas where it has proved better or safer than human activities; on the other hand, it would be desirable and fair to expect a higher standard from LLMs (Durante & Floridi, Reference Durante and Floridi2022, 93–112) where they are proved to be able to achieve it (European Commission, 2019, 25). In more general terms, comparing human- or AI-generated results will raise tensions that may result in third-party protected expectations (e.g., should the lawyers inform the client that their legal analysis has been generated or supported by an AI system?).
Second, it is to reaffirm what has been qualified as the “principle of functional equivalence” as closely related to the “principle of equal protection under the law” (Durante & Floridi, Reference Durante and Floridi2022, 112). The first principle asserts that operation or assistance by an AI system (in this case, LLMs) should be treated no differently from human operation or assistance in case of harm to a third party, for equal protection under the law to be ensured to all those who are the receivers of the effects or the performance of these systems. Similar cases must be dealt with in similar ways unless there is a reason to make a distinction. In this regard, a remark by the Expert Group on Product Liability in the field of AI and Emerging Technologies is relevant from an epistemological standpoint, although stated in a partially different context: “[It is] challenging to identify the benchmark against which the operation of non-human helpers will be assessed in order to mirror the misconduct element of human auxiliaries” (European Commission, 2019, 25).
Third, technological dependence raises an additional epistemological challenge posed by the need for humans to increasingly rely on technology itself to distinguish synthetic from non-synthetic outputs and to detect the use of LLMs in producing pieces of knowledge. Consider, for example, the tools developed in academia to understand whether a text is generated by an artificial or human agent, which raises relevant issues in terms of liability, plagiarism and copyright. Accordingly, it is important to mention the publication of guidelines for the responsible use of LLMs and Gen AI in research, according to which researchers “are accountable for the integrity of the content generated by or with the support of AI tools” as well as it is expected that they “do not use fabricated material created by generative AI in the scientific process, for example falsifying, altering, manipulating original research data” (European Commission, 2024, 6). This becomes even more challenging when dealing with so-called hybrid data, i.e., “the offspring of historical and synthetic data” (Floridi, Reference Floridi2023a, 36).
All these cases highlight the crucial importance of standards as they constitute key benchmarks for evaluating the reliability and accuracy of AI-generated outputs. Furthermore, the identification of these standards implies a complex assessment (from a legal, ethical and social standpoint), in order to guarantee adequate levels of transparency and responsibility when using LLMs and Gen AI systems, taking also into account the rapid pace of technological advancement and the associated need to periodically update measurement criteria.
3.5 A legal duty to tell the truth?
A similar set of epistemological concerns has motivated some scholars to advance the hypothesis that there may be a legal duty on LLMs providers to create models that “tell the truth” (Wachter et al., Reference Wachter, Mittelstadt and Russell2024). This study represents, at the state of the art, the most systematic examination of what has been defined as the risk of “epistemic harm” (Wachter et al., Reference Wachter, Mittelstadt and Russell2024, 5), meaning the set of dangerous and harmful consequences that can result from relying on epistemologically flawed AI-generated content.
This study, building on the assumption that a risk of epistemic harm depends not only on the inherent limitations of LLMSs (which “are not designed to tell the truth in any overriding sense” [Wachter et al., Reference Wachter, Mittelstadt and Russell2024, 2]) but also on the extrinsic epistemic limits of users (who “are both encouraged and innately susceptible to believing LLMs are telling the truth” [Wachter et al., Reference Wachter, Mittelstadt and Russell2024, 3]), highlights a key point: namely, that such risk is not specific or limited to a given area but systematic in nature, since it gives rise to “careless speech” which is bound to cause “unique long-term harms to science, education, and society which resist easy quantification, measurement, and mitigation” (Wachter et al., Reference Wachter, Mittelstadt and Russell2024, 5).
The study proposes a detailed analysis of truth-related obligations in EU human rights law, AI Act, Digital Services Act, Product Liability Directive, and AI Liability Directive. These legal frameworks contain limited, sector-specific truth duties, from which, according to the authors, it is hard to derive a general duty to tell the truth:
Where these duties exist they tend to be limited to specific sectors, professions, or state institutions, and rarely apply to private sector. The harms of careless speech are stubbornly difficult to regulate because they are intangible, long-term and cumulative. […] Current frameworks are designed to regulate specific types of platforms or people (e.g., professionals) but not to regulate a hybrid of the two. Future regulatory instruments need to explicitly target this middle ground between platforms and people (Wachter et al., Reference Wachter, Mittelstadt and Russell2024, 47–48).
However, the authors trust that: “Despite this general regulatory landscape, the limited duties found in science and academia, education, and libraries and archives offer an interesting avenue to explore as LLMs serve a similar function” (Wachter et al., Reference Wachter, Mittelstadt and Russell2024, 48). On this basis, they believe that a general duty to tell the truth can be assumed from more limited, sector-specific duties. In fact, they contend that the “scope of this duty must be broad; a narrow duty would not capture the intangible, longitudinal harms of careless speech, and would not reflect the general-purpose language capacities of LLMs” (Wachter et al., Reference Wachter, Mittelstadt and Russell2024, 48).
As already mentioned, (Wachter et al., Reference Wachter, Mittelstadt and Russell2024) is a systematic and in-depth study that would deserve extensive analysis and detailed review, particularly with regard to the fields of science, academia, education, archives and librariesFootnote 14 (which serve as a point of reference for founding a general duty to tell the truth). Here, however, the aim is limited to analyzing a few problematic facets of a general duty to tell the truth, related to what is relevant to this study, regarding the epistemological investigation of LLMs.
It may be doubted that LLMs can be technologically designed in such terms as to tell the truth (or raise a claim to truth), given the inherent statistical dimension with which they provide answers to a given prompt. This does not prevent their accuracyFootnote 15 (or “ground truth”) from being improved over time within the inherent epistemic limits on which they are based.Footnote 16 Moreover, it is open to question whether accuracy, reliability or truthfulness by design do not risk increasing, rather than decreasing, the tendency to over-rely on AI-generated contents. However, it is not the technological feasibility of truth-telling LLMs that we intend to discuss here; rather, the focus is on the desirability of a legal duty to tell the truth, from a threefold standpoint: (i) the incentive for accurate information; (ii) the risk of paternalism and (iii) the generalization of context-based truth duties.
First, careful speech is essential for democracy, but it is costly: it requires time, effort, resources and often expertise. There is a basic, deeply rooted asymmetry in the production of information: it is easy to produce false or inaccurate information, while it is hard to produce true and accurate information. Careful speech is the result of choice, investment and effort, which cannot simply be made mandatory by law. Commitment to true or accurate information is part of a process for which there must always be public or private incentives other than fear of breaking a legal norm.
Second, protecting users from relying on flawed AI-generated contents by providing a duty to tell the truth that takes on a general scope with respect to LLMs sounds somewhat paternalistic. Pointing out the epistemic limits, biases and flaws that pollute the ground truth of LLMs is crucial, but it does not automatically lead to embed into them, by design, a supposed predisposition to truth that would shelter users from the risk of being exposed to falsehood and careless speech. After all, is such risk not naturally part of the basic dialectics of democracy?
Third, the law has always resisted the temptation to prosecute and punish the utterance of falsehood as such. In fact, the law does only regulate the specific circumstances under which a legally relevant and punishable act is committed by the assertion of a falsehood: e.g., giving false testimony in court; attributing an untrue fact to someone by defamation; conducting negotiations in bad faith by misrepresenting an item of sale, etc. In all these cases, it is not the misleading or false statement as such that is legally relevant, but the fact that it represents a form of conduct under the given circumstances that the law sees fit to pursue and punish. Misrepresentation of a fact is usually not legally relevant except in a specific context, as in the case of perjury in court or bad faith negotiation. The law sets the rules for defining the context (e.g., a trial), and then when and how misrepresenting a fact is configured, for instance, as a perjury in that context. Law does not regulate a falsehood independently of a given context in which that falsehood counts (i.e., is legally relevant) as a tort or crime. A specific context is the only level of abstraction (Floridi, Reference Floridi2008) at which a falsehood can be deemed legally relevant.
There is actually a good reason for this, which is based on a fundamental principle of legal and democratic civilization: in a state of law and in a democratic society nobody can or should have sole possession of the truth (historically, States have a monopoly on the use of force, but not a monopoly on truth). If we were to charge a public authority with the responsibility of pursuing and punishing all false statements (or careless speeches), we would implicitly be investing this authority with the power to distinguishing truth from falsehood and thus, ultimately, the power to declare what is true or false in a given society (Durante, Reference Durante2021, 89).
It is fundamental to identify specific circumstances in which telling falsehoods (or speaking carelessly) is legally relevant, but this is not the same as assuming a general duty to tell the truth with reference to human or artificial agents. In other words, it is certainly important to clarify the specific circumstances in which LLMs are likely to produce legally relevant consequences, such that they may entail forms of legal liability on the part of their providers; but it is questionable – and perhaps not even desirable for the reasons given above – that a general duty to tell the truth can be derived from these specific circumstances.
The reasoning does not lead one to deny the risks implicit in a systematic careless speech, which does not merely generate specific and limited inaccuracies or misrepresentations of reality as the content of discourse but is capable of long-term alteration of the very conditions of discourse. Rather it seems fair to argue that democracy is more protected by the discussion on and unconstrained pursuit of truth open to reasonable disagreement than by the imposition of a general duty to tell the truth.
4. Conclusions
LLMs raise several challenges that can be examined according to a normative and epistemological approach. The former, increasingly adopted by European institutions, identifies the pros and cons of technological advancement. With regard to LLMs, the main pros concern technological innovation, economic development and the achievement of social goals and values. The disadvantages mainly concern cases of risks and harms produced by means of LLMs. The epistemological approach investigates how LLMs produce data, information and pieces of knowledge in ways that differ from – and often are irreducible to – humans. To fully grasp and face the impact of LLMs, our paper contends that the epistemological approach should be prioritized, since the identification of the pros and cons of LLMs depends on how this model of AI works from an epistemological standpoint and our ability to interact with it. To this end, our analysis compared LLM’s and legal epistemology and highlighted five major problematic issues: (1) qualification; (2) reliability; (3) pluralism and novelty; (4) technological dependence and (5) relation to truth and accuracy.
Furthermore, the analysis of such epistemological challenges contributes to understand some potentially long-term consequences of LLMs on relevant democratic aspects of our contemporary society. These consequences may be understood in terms of an impact on (i) human autonomy; (ii) semantic capital and (iii) level playing field.
Human autonomy is impacted by LLMs in several ways: from being unable to verify the accuracy of outputs to being subject to misrepresentations; from reliance on untrustworthy results to the risk of manipulation of opinions formed on biased or hallucinatory content, etc. (Novelli & Sandri, Reference Novelli and Sandri2024). In all these cases, the limitation of human autonomy generates a lack of social trust and a loss of individual self-determination.
The production of data, information and pieces of knowledge by LLMs has a strong impact on the production of semantic capital, which is defined as “any content that can enhance someone’s power to give meaning to and make sense of (semanticise) something” (Floridi, Reference Floridi2018, 483). This power plays a crucial role in every democracy and society. The increasing number of outputs and tasks delegated to LLMs can circumscribe or alter human power to give meaning to and make sense of reality.
It should be also noted that the production of data, information and pieces of knowledge made possible by the technological development of LLMs requires resources, investment, time and energy, with a great impact on our society (van der Sloot, Reference van der Sloot2024, 66–67) and, above all, the environment (Berthelot et al., Reference Berthelot, Caron, Jay and Lefèvre2024; Mannuru et al., Reference Mannuru, Shahriar, Teel, Wang, Lund, Tijani and Vaidya2023). As we have tried to highlight, this can benefit the few, already advantaged, and weaken the many, already disadvantaged, by altering the level playing field.
Ultimately, the risk from the epistemological perspective is that the focus on outputs tends to obscure the importance of the path through which the process of knowledge unfolds.
Competing interests
The authors declare none.
Open access
