2. Situating the methodology

ATLAS is a product of the AI as Infrastructure (AIINFRA) project, which is implementing methodological baselines and tools to support the evaluation of RAG systems in a way that opens them up for empirical as well as epistemological analysis by HASS researchers. This is pressing work being undertaken in a fast-moving technological and methodological environment that is not always well aligned to HASS and GLAM priorities. Existing mainstream approaches to the evaluation of LLMs largely focus on leaderboard tables, by which the performance of different LLMs is tracked against benchmark tests. These are designed by industry and STEM disciplines to their standards, assess foundation models rather than RAG architectures (see below) and do not account for the semantics of historical source documents or HASS and GLAM sector values or scholarly requirements (Lee et al. Reference Lee, Yasunaga, Meng, Mai, Park, Gupta and Zhang2023; Li et al. Reference Li, Cheng, Zhao, Nie, Wen, Bouamor, Pino and Bali2023; Lin et al. Reference Lin, Hilton and Evans2022). As the International Panel on the Information Environment (IPIE 2025) noted in early 2025, issues with labour laws, violations of human rights and the rights of Indigenous modes of knowledge and scholarship are never noted in such frameworks. Environmental impact is often considered but political and gender bias, and racism, are not adequately confronted (IPIE 2025, 11).

This set of problems speaks to a wider issue. A benchmark specifically designed to test historical knowledge has concluded that “while the overall performance of the LLMs on expert historical knowledge is better than random guessing, it falls short of comprehensive expert-level knowledge” (Hauser et al. Reference Hauser, Kondor, Reddish, Benam, Cioni, Villa, Bennett, Hoyer, Francois, Turchin and del Rio-Chanona2024). Those tests were conducted by comparing the output of LLMs against the Sheshat Global History Databank, a broad-ranging human-curated collection of historical data. While these results are important, not only for demonstrating the limitations of current LLMs for HASS research but for establishing a baseline against which future versions can be tested, they are not reflective of the ways in which LLMs are likely to be deployed in GLAM or HASS settings. Like the commercial leaderboards they are derived from, they were not designed to test the performance of LLM RAG systems or to evaluate how LLMs and LLM RAG systems might be evaluated in terms of broad sociotechnical metrics. This more holistic mode of evaluation is important because it reflects modes of deployment likely to occur across the HASS and GLAM sectors, as AI becomes more tightly integrated into RI. DH scholars need to gain a sense of the baseline accuracy of LLMs, but there are many other issues to consider as well, ranging from basic issues of scientific best practice (transparency, explainability, reproducibility) to environmental impact, cultural safety and deployment cost.

STEM and computational linguistics research communities are aware of the importance of testing and have been developing tools and methods for many years, noting that “[w]ith the rapid development and widespread use of LLMs, the importance of evaluating them in practical applications and research has become crucial” (Chang et al. Reference Chang, Wang, Wang, Wu, Yang, Zhu, Chen, Yi, Wang, Wang, Ye, Zhang, Chang, Yu, Yang and Xie2024, 39:31). The implications of this statement are more profound than might be supposed. The commercialisation of LLMs has depended on the interaction between evaluation, testing and subsequent technical development, initially through empirical (often automated) benchmarking but increasingly through human-in-the-loop evaluation that allows for “greater human feedback during the evaluation process” (Chang et al. Reference Chang, Wang, Wang, Wu, Yang, Zhu, Chen, Yi, Wang, Wang, Ye, Zhang, Chang, Yu, Yang and Xie2024, 39:31). The ability of DH scholars to design and develop LLM research tools is heavily dependent upon their capacity to engage in wide-ranging software test engineering.

Yet engagement with software test engineering alone is insufficient. The opacity of AI technologies demands not just technical competence but critical reflection on the systems being built and evaluated. Testing has always been integral to software development, but the opacity of AI technology heightens the stakes. As is well known from landmark publications such as Emily M. Bender et al.’s (Reference Bender, Gebru, McMillan-Major and Shmitchell2021) “stochastic parrots” article “… an LM [language model] is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning …” (Bender et al. Reference Bender, Gebru, McMillan-Major and Shmitchell2021, 617). This results in the well-known tendency towards “hallucination” of facts and citations (Xu et al. Reference Xu, Jain and Kankanhalli2024), and the deep opacity of AI systems that lend them the appearance of classic “black box” technologies (impenetrable to meaningful interpretation) (Pasquale Reference Pasquale2015). Opacity is increased again through the commercialisation of LLMs into products and apps that actively seek to hide complexity from end users, in the interests of protecting intellectual property (IP) rights and improving user experience. As David Berry has pointed out, it is important not to over-compensate for these issues and assume meaningful knowledge of AI products is impossible, but commercial products make robust evaluation of LLMs difficult (Berry Reference Berry2023). Evaluation and testing can never provide full transparency, but they can ameliorate the opacity of LLM-based products to a significant degree and enable informed decisions regarding LLM use.

This is of huge importance in epistemological as well as political and sociotechnical terms. If DH scholars can harness the power of language models, gaining leverage over their tendency to reflect biases from their training data in their output, they will be better positioned to integrate them into research workflows, contribute to their ethical design, development and regulation and support the broader GLAM community with its engagement with LLM technologies. Our contribution to this space is to implement a prototype test harness in the spirit of “critical technical practice” presented by Philip Agre in 1998, introducing technical methods that might have some lasting value in the development of responsible AI for HASS and GLAM (Agre Reference Agre, Bowker, Gasser, Star and Turner1997). Our prototyping method is necessarily reliant on established engineering methods but is informed by a commitment to the values and principles of the HASS and GLAM communities, incorporating the spirit of responsible and slow AI (Huggett Reference Huggett2024; McLennan et al. Reference McLennan, Fiske, Celi, Müller, Harder, Ritt, Haddadin and Buyx2020). If we, as DH scholars, can create models, workflows and prototypes that support thinking through the realities of working with and against AI technologies, we stand a much better chance of either developing production-grade versions of those technologies or (at least) understanding them well enough to build mutually beneficial relationships with commercial vendors.

3. Architecture

Having outlined the case for critical technical engagement with LLM technologies, this section describes the design choices that have shaped ATLAS’s development and details its architecture.Footnote ¹ ATLAS is a prototype developed using the Python frameworks FastAPI and LangChain, the frontend framework Vue and the vector store Chroma DB (Figure 2). The key-value store Redis is used to enable feedback telemetry and manage the message queue in production. AWS Cognito is used for user authentication. Telemetry is enabled using the Open Telemetry schema for delivery to the Phoenix Arize cloud platform. Heavy use has been made of AI coding tools, including GitHub Copilot, Cursor, Windsurf and Claude Code. The sole developer (Smithies) is an amateur programmer, augmented by insights gained from two decades working with software engineering teams in the commercial, government and academic sectors. This process of coding with AI in a research context is outside the scope of this paper but is an important topic for further investigation, with significant implications for HASS research.

The ATLAS harness enables reproducible and transparent experiments with relatively large corpora and multiple models. The system uses a mixture of deterministic and probabilistic technologies and can be configured to use any number of LLMs, including commercial (OpenAI, Anthropic, Google) and more open models (Llama, Mistral, DeepSeek). The user interacts with a standard question and answer (Q&A) chatbot interface (Figure 3). Although the choice of technologies stemmed from a preference for Python-based tools, it was strongly influenced by the state of technology when development started in mid-2024. At that point in time, there were few low and no-code solutions, and desktop and commercial products had limited functionality and worse transparency. The LangChain framework was the leading option but was under rapid development and criticised by experienced programmers for being unnecessarily abstract (Safjan Reference Safjan2023). Rival frameworks were even less mature. At the time of writing, version 0.2.1 of the ATLAS prototype is about to be released and the tool has been used in multiple production-level contexts.

The hope is that ATLAS can continue to develop alongside rapid changes in the technical environment, but it is envisaged as an ephemeral prototype. The sustainability of using complex development tools, designed for enterprise use cases, is questionable, and it is unlikely that the project has the broad appeal needed to develop a technical community. The primary benefit of the chosen technology stack is its ability to provide a view “behind the veil” of LLM RAG systems, exposing the components that influence the system output (Figure 1) and modelling possible approaches to LLM RAG testing, rather than as a long-term solution suitable for widespread production use.

Figure 1.

ATLAS large language model (LLM) retrieval-augmented generation (RAG) architecture.

Figure 1 Long description

The diagram illustrates a workflow involving data processing, a research tool and a user interface. On the left, 'Source Processing (Chunking)' is depicted with a box containing documents labeled 'Harvest 1901 ALL NZ, UK Sources' and 'Word Embedding Model, Metadata / Entities'. This connects to a 'Vector Store' represented by a cylinder labeled 'Harvest 1901 ALL NZ, UK Vector Store'. An arrow leads from the vector store to a 'Foundational Model', symbolized by a starburst shape, with text indicating 'ChatGPT, Llama, Anthropic etc. with different characteristics'. This foundational model connects to a 'Research Tool' box in the center, containing 'Retriever Code' and 'System Prompt'. Dotted lines extend from the research tool to a 'Q and A User Interface' on the right, which is linked to a stick figure labeled 'Historian'. Below the diagram, text reads 'LLM, embedding model definition, API layers, metadata schema, chat memory' and 'Ephemeral and cultural rules and identity'.

Figure 2.

ATLAS vector store database.

Figure 2 Long description

The interface shows a list of files with columns labeled 'Name', 'Size', 'Type' and 'Last Modified'. Each file entry includes a name, file size and type. On the right, metadata fields are visible, including 'Field', 'TTL', 'content', 'source', 'data' and 'page'. The 'content' field contains text, while 'source' and 'data' fields include links. The 'page' field shows a numerical value. The interface is divided into two sections, with the file list on the left and metadata details on the right. The background is dark and the layout is organized for easy navigation and viewing of file details and metadata information.

Figure 3.

ATLAS user interface.

Figure 3 Long description

The article discusses the imperial relations between Britain and its colonies in 1901, focusing on trade, tariffs, defense and autonomy. Key sections include 'Trade and Tariff Relations', 'Defence Contributions' and 'Imperial Solidarity and Autonomy'. The text is organized with bold section headings and a sidebar containing links and tags. The article begins with a summary of parliamentary research across Australia, New Zealand and the United Kingdom, examining trade agreements and defense contributions. It highlights New Zealand's 'policy of exclusion' and challenges to imperial defense. The sidebar lists various tags such as 'Imperial Relations', 'Trade' and 'Defence'. The document is formatted with a title at the top, followed by paragraphs under each section heading, providing a structured academic-style layout.

RAG system architectures are largely deterministic (traditional web applications), except for the foundation model (LLM) that provides the “brains” of the system, and include multiple configurable elements. The responses from the LLM are guided by system prompts that define the role the LLM should adopt in its answer. This constrains but does not override the different “personalities” of the different LLMs (Serapio-García et al. Reference Serapio-García, Safdari, Crepy, Sun, Fitz, Romero, Abdulhai, Faust and Matarić2023). The primary system prompt for ATLAS is configurable and can make a significant difference to the output. At the time of writing, its default prompt is 66 lines long, divided into sections defining its role, corpus selection, citation guidelines, evidence handling, uncertainty handling and edge cases. Its primary “identity” is defined in the following way:

Another system prompt is needed to guide multi-turn Q&A, instructing the LLM:

Using the system prompts, the LLM uses a “retriever” module, written using LangChain, to search across a vector database containing the transnational corpus of Hansard documents, stripped of their XML in the initial version to create a straightforward text corpus and divided into ∼95,000 “chunks.” Significantly, the open architecture of ATLAS allows for the implementation of multiple different retrievers, optimised for different vector stores and emerging forms of Model Context Protocol servers that can provide accessible programmatic access to data sources of various kinds.

One of the benefits of using open-source components, therefore, is the ability to create multiple vector store databases using the baseline Hansard corpus. As the primary data source for the system, the vector store has a crucial effect on the eventual output, implying the need for testing different approaches to vector store design. This is especially the case with the transformation of the document chunks into word embeddings that are stored in the database, a process that is needed because of the limited “context window” of LLMs. The LLM will only retrieve a limited number of chunks that match the user question, grounding its answers in them. Different document “chunking” sizes can therefore make a significant difference to the answers given by the LLM, depending on whether the chosen chunks support robust inferencing. The baseline ATLAS vector store adopts the widely accepted approach of chunking the Hansard debates into tokens of 1000, with an overlap of 200, resulting in ∼95,000 chunks with associated metadata. The number of chunks returned by the LLM (known as the “search-k” or “fetch-k” number) can also be set, with a range between 10 and 100 producing reasonable results depending on the LLM used. A “temperature” setting influences the shape of probability distribution through which an LLM generates a response, with a setting of 0 being the least random (or, most deterministic). Future work will assess whether larger or smaller chunks improve performance, and whether the ever-increasing context windows of LLMs will allow the analysis of HASS and GLAM-sized corpora without the use of RAG architectures.Footnote ³

This does not seem likely in the short term, although the pace of development is rapid. It is important to understand the trade-off with using RAG systems for the analysis of corpora, regardless. The fetch-k setting drastically reduces the amount of information the LLM has available to analyse, reducing the information space and evidentiary context of the resulting answer, but also drastically reducing the computational, financial and environmental cost of analysis. In the case of the default transnational Hansard corpus used in ATLAS, costs are reduced from over $USD1000 per query (depending on the model used) to a matter of cents.Footnote ⁴ The technique for achieving this relies on a radical reduction in the information space available to the LLM to analyse. Rather than analysing all 11,967,262 words (70,581,735 tokens) in the corpus, which ChatGPT-4o would need to do in 579 batches to fit the data into its 128,000 token context window at the time of writing, only the document chunks defined by the fetch-K variable are analysed. This monetary reality corresponds to environmental impact in important ways, with the cost of each query reflecting the computational intensity of the related processing.

The chunks are chosen using the Hierarchical Navigable Small World algorithm that selects chunks based on their semantic similarity to the query (making the choice of word embedding significant), then re-ranked using a standard process. This reduces the 95,000 chunks to 500 and then 30 that are analysed by the LLM, depending on the test target configuration. As with all LLM RAG systems, the information space is so reduced that a degree of serendipity is introduced, requiring careful attention to the response, but the underlying architecture guarantees a relationship between query, response and citations. The precise sociotechnical and epistemological nature of this relationship is a question of some complexity, especially given the probabilistic nature of the LLM responsible for analysing the document chunks, but parallels could perhaps be found in the analogue world of browsing library shelves and card catalogues. The point is not to assume the epistemological relationship between researcher and tool is stable or known (and certainly not to treat LLM RAG systems as oracles), but to problematise such systems in terms of critical technical practice and evaluate their potential role in the discovery and research process on that basis.

Aside from gaining the ability to extract and attach metadata to document chunks (potentially including concepts, entities and sentiments alongside basic information about the Hansard session, country of origin and so on), the primary advantage of having complete control over the vector store is the ability to use any number of different word embedding models in the development process, ensuring the semantic relationships represented in the vector space of the database correspond to historical rather than contemporary semantics (McGillivray Reference McGillivray, Schuster and Dunn2020). Most commercial LLM RAG products limit the word embeddings that can be used (if they even allow the user to see which embeddings have been used to transform the documents they have uploaded), as well as significantly limiting the number of documents that can be uploaded to the vector store. The primary ATLAS vector store uses word embeddings developed by the Living With Machines project. The embeddings are based on the well-regarded Bidirectional Encoder Representations from Transformers (BERT) model, fine-tuned using one billion tokens of content from the British Library’s collection of books from 1890 to 1901 (Hosseini et al. Reference Hosseini, Beelen and Colavizza2021). It is worth noting the foresight of that project in producing the embeddings and making them available for wider use. Future work could test our assumption that a vector store developed using the Living With Machines BERT word embeddings will perform better than vector stores developed using word embeddings optimised for contemporary content. While it is tempting to assume historical semantics will have a demonstrable effect on the quality of the output, the possibility remains that a sufficiently powerful LLM might be able to draw on its own background knowledge and the context of the corpus to choose appropriate chunks and infer appropriate meaning.

Pushing back against the opacity of many commercial AI products, ATLAS provides detailed technical information about the “calibration” of each system target (model version, word embedding, vector store characteristics, system prompt), as well as the source documents used by the foundation model to generate its response. The first version of the user interface only includes a simple multi-turn Q&A interface, with answers including citations to the document chunks used in the analysis. A privacy setting allows users to perform searches without their queries being captured. Data that is captured is fully anonymised, including the IP of the computer used for testing or any information about the user beyond their questions and feedback ratings.

To facilitate reproducibility, the user interface (UI) also includes export functionality that produces a .json file with the details of the system target and a record of the Q&A and citations (Figure 4). This is a rudimentary nod towards reproducibility but has been augmented with rich telemetry data that captures not only the input and output data but steps in the RAG chain, such as guardrail assessment (which can be configured to detect hate speech or language related to Indigenous culture or other sensitive topics), the retrieval and re-ranking process, and citation generation. By recording token counts for each query and response pair, a sense can be gained of the financial but also environmental cost of the system, and by sending test target metadata, it becomes possible to compare different configurations of the system.

Figure 4.

ATLAS .json export.

Figure 4 Long description

Monospace text in a code editor with line numbers, showing a JSON object. A field named date shows 2024 dash 11 dash 19 T 04 colon 17 colon 33 dot 813 Z. A config section lists settings including ALGORITHM set to FLAT, CHUNK OVERLAP set to 75, CHUNK SIZE set to 500 and score underscore threshold set to 0 dot 2. SEARCH TYPE is similarity underscore score underscore threshold. A chatHistory array contains two message objects. One message has sender user and message text asking to describe the documents, referring to similarities and differences between countries. Another message has sender ai and message text about documents referenced in the Hansard texts from 1981, mentioning government discussions about preserving local records, financial conditions and taxation across New Zealand, Australia and the United Kingdom.

In keeping with our focus on human responses to AI, ATLAS also includes functionality to capture human feedback on the LLM RAG output. We view this as crucial to the evaluation process, allowing subject domain and general users to enter feedback that is added to the telemetry sent to Phoenix Arize (Figure 5). This has been augmented with an inter-rating feature that allows feedback on Q&A sessions to be submitted by multiple users, improving the quality of the dataset. Although the data collected by ATLAS is anonymous, it should allow the project to develop a rich database of test configurations and associated user feedback for later analysis and possible model fine-tuning and LLM-as-Judge testing. AI “observability” frameworks and services (often also offering automated testing services), such as Phoenix Arize, are important for HASS and GLAM communities. Such tools have undeniable connections to the age of “surveillance capitalism” (Zuboff Reference Zuboff2019) but reflect a convergence between traditional notions of transparency and reproducibility in HASS research and the technology industry’s need for tools and methods capable of derisking probabilistic systems. Such tools meet the needs of commercial auditing and compliance requirements and provide software engineers and data scientists with large amounts of detailed test data. They also provide HASS and GLAM sector practitioners with a level of rigour appropriate to their research and public sector activities.

Figure 5.

ATLAS user feedback in the Phoenix Arize observability platform.

Figure 5 Long description

The image displays a dashboard interface with a trace status labeled as 'UNSET', a total cost of 90 and a latency of 17.15 seconds. On the left, there is a hierarchical list showing processes such as 'com.atlas.rag.pipeline' and 'com.atlas.rag.generation.response', each with associated time durations. The right side features a table with columns labeled 'name', 'annotator kind', 'user', 'label' and 'score'. Entries include 'Clarity', 'Factual Accuracy', 'Question Difficulty', 'Relevance Rating', 'Source Quality' and 'User Category', each annotated by 'HUMAN' or 'system' with corresponding scores. The interface is designed for monitoring and evaluating system processes and annotations.

Ultimately, the intention is to use the telemetry and feedback data for automated testing, whereby a second LLM is trained to assess the quality of responses and asked to grade the output of a range of different system targets. Despite this potential, it is important to note that testing with ATLAS requires substantial attention to human ethics and test planning, to ensure any subsequent operationalisation of the system is properly contextualised (Berman and Smithies Reference Berman, Smithies, Quercia and Constantinides2026). This crucial supporting material is under development at the time of writing. The test plan, used to guide focus groups and online interaction with ATLAS, will be made available along with the source code in due course. The plan uses principles for testing LLM systems informed by HASTRICT principles, referring to the need for testing to be H: Human-focused; A: Accountable; S: System-aware; T: Transparent; R: Reproducible; I: Iterative; C: Contextually Informed; and T: Task based (Smithies and Berman Reference Smithies and Berman2025a). These principles pair the technical ATLAS system with controlled human processes to generate actionable research data and will be the ultimate test of the harness.