2.1 Guard rails

The following example is like the example above, except that we ask ChatGPT to predict the future:

• • Input to ChatGPT:

  system: You are a helpful assistant.

  user: Who won the world series in 2035?

• • Response from ChatGPT:

  I’m sorry, but I am unable to provide real-time information or details about future events as I exist in the present. You may want to check the latest sports news or websites for updates on the World Series winners in 2035.

ChatGPT is constructed with guard rails designed to reduce “hallucinations.” “Hallucination” is a technical term for “making stuff up.” Chatbots have a tendency to spread misinformation. Guard rails help in easy cases like this, though it is not hard for an adversary to find small modifications to input prompts that will crash through the guard rails.

2.2 Timeliness

The next example replaces “2035” with “most recent.” The point of this modification is to make it clear that we are using an old model that was trained in 2021:

• • Input to ChatGPT:

  system: You are a helpful assistant.

  user: Who won the most recent world series?

• • Response from ChatGPT:

  The Atlanta Braves won the most recent World Series in 2021.

We used an old model because OpenAI charges more for more recent models. But even so, since training large language models (LLMs) is expensive, it is prohibitively expensive for OpenAI to continuously update their models. The following chat shows that phrases like “most recent” call out differences between training time and inference time. According to Google and Wikipedia, the Texas Rangers won the most recent World Series (in 2023), but ChatGPT-3 says the Atlanta Braves won (in 2021). It is easier to keep Google and Wikipedia up to date than ChatGPT. In Subsection 3.2, we will show how RAG can work around the need to continuously update models.

2.3 Hallucinations when asked to discuss content that goes beyond training data

The next example was designed to elicit misinformation. The prompt presupposes that we have been talking about an (unspecified) paper on psycholinguistics. One might hope there would be a guard rail that would catch this prompt and respond, “I’m sorry, but I don’t know what paper you are referring to.” As we will see below, there is a guard rail that catches similar prompts. Unfortunately, in this case, the response below starts with a pivot to a different question: “What is Psycholinguistics?” After that pivot, the response continues by hallucinating what might be in the unspecified paper.

• • Input to ChatGPT:

  system: You are a helpful assistant.

  user: Please summarize the paper on psycholinguistics.

• • Response from ChatGPT:

  Sure! Psycholinguistics is a field that studies the psychological processes involved in language comprehension and production. The paper discusses various topics in psycholinguistics, such as how language is processed in the brain, how we acquire language skills, and how language is used in communication. It also explores issues related to bilingualism, language development in children, and the impact of language disorders on individuals. Overall, the paper provides insights into how the human mind processes and understands language.

Robustness is a challenge for guard rails. A small change to the prompt can have a large impact. Consider the following chat where we replace a period (in the chat above) with a question mark (in the chat below). The guard rails are more effective with the question mark, at least in this particular instance:

• • Input to ChatGPT:

  system: You are a helpful assistant.

  user: Please summarize the paper on psycholinguistics?

• • Response from ChatGPT:

  I would need more specific information to provide an accurate answer. There are numerous papers on the topic of psycholinguistics written by various authors. If you have a specific paper or author in mind, please provide more details so I can assist you better.

More generally, hallucinations are a challenge for LLMs when asked to discuss content that goes beyond training data. A lawyer once found himself in trouble when he trusted ChatGPT to help write a brief.Footnote ^c

There was just one hitch: No one – not the airline’s lawyers, not even the judge himself – could find the decisions or the quotations cited and summarized in the brief.

A study on legal hallucinations from Yale and Stanford (Dahl et al. Reference Dahl, Magesh, Suzgun and Ho2024) reports that the problem is “alarmingly prevalent, occurring between 69% of the time with ChatGPT 3.5 and 88% with Llama 2, when these models are asked specific, verifiable questions about random federal court cases.”

3.1 Recipe

Four implementations of RAG are posted on our GitHub: (1) src/OpenAI/Rag.py, (2) src/LangChain/Rag.py, (3) src/transformers/Rag.py,and (4) src/VecML/Rag.py. They all use the following recipe:

1. 1. Upload one or more files

2. 2. Parse files into chunks

3. 3. Embed chunks as vectors in a vector space

4. 4. Index vectors with an approximate nearest neighbor (ANN) algorithm such as ANNOYFootnote ^k or FAISSFootnote ^l (Johnson, Douze, and Jégou Reference Johnson, Douze and Jégou2019; Douze et al. Reference Douze, Guzhva, Deng, Johnson, Szilvasy, Mazaré, Lomeli, Hosseini and Jégou2024)

5. 5. Retrieve chunks near prompt (using ANN)

6. 6. Generate responses

All four RAG.py programs share the same interface. They take a list of files to upload on the command line. Input prompts are read from standard input, and output responses are written to standard output. The programs are short, less than 100 lines; the VecML program is particularly short:

Some of the shorter RAG.py programs hide various steps in the recipe above behind APIs running on the cloud.

There are also a few programs, chat.py. These programs read prompts from standard input and write responses to standard output, but unlike RAG.py programs, the chat.py programs do not upload files.

3.2 Timeliness and a simple example of RAG

This paper will not attempt to discuss all of the opportunities raised above, except to suggest that RAG can help with a few of them. Consider the timeliness opportunity in Subsection 2.2; obviously, users do not want “day-old donuts” like who won the World Series when the bot was trained a few years ago.

On the web, information ages at different rates. Before the web, news was typically relevant for a news cycle, though breaking news was already an exception. These days, news typically ages more quickly than it used to, but even so, the half-life of news is probably a few hours. Some information sources age more quickly such as stock prices, and other information sources age more slowly such as the academic literature. Within the academic literature, some venues (arXiv and conferences) age more quickly than others (archival journals). Crawlers for search engines like Google and Bing need to prioritize crawling by trading-off these aging properties with demand. Since crawlers cannot afford to crawl all pages all the time, they need to crawl pages often enough to keep up with aging, but if that is not possible, then they should prioritize pages that are more likely to be requested.

How does RAG help with timeliness? Rather than use LLMs as is, and attempt “closed-book question-answering” with an “out-of-date book,” RAG uses R (retrieval/search) and A (augmentation) to update “the book” (knowledge base) just-in-time. For example, if we want to know who won the most recent World Series, then it might help to do a search for documents on that topic such as this.Footnote ^m RAG makes it possible to upload files before generating responses to the prompt. On the GitHub,Footnote ⁿ we work though this example in detail:

Without RAG, an LLM trained on 2021 data would likely hallucinate when asked about 2023. RAG fills in gaps in the knowledge base by uploading a pdf file, sample_files/World_Series/*pdf, a version of the text from footnoteFootnote ^m , and uses that content to update the knowledge base on this topic from 2021 to cover 2023. In this way, RAG works around the need to continuously retrain the bot, by creating a just-in-time mechanism for updating the knowledge base.

3.3 References to background context and RAG

RAG can also help with references to the discourse context. If we upload a paper on psycholinguistics, then it can address the prompt: “Please summarize the paper on psycholinguistics.”

The directory, sample_files/papers, contains two pdf files for Church and Hanks (Reference Church and Hanks1990) and Lin (Reference Lin1998). We would like to ask the bot to summarize (Church and Hanks Reference Church and Hanks1990), but since the bot is unlikely to understand citations, we will refer to the two papers with phrases such as

• • the paper on psycholinguistics

• • the paper on clustering

The program listing above produces the RAG summaries in Table 1. Summaries from RAG are impressive, at least on first impression. However, on further reflection, after reading a number of such summaries, it becomes clear that there are many opportunities for improvement.

Table 1. RAG summaries are longer than tl;dr summaries from Semantic Scholar

RAG summaries are probably worse than summaries from previous technologies such as tl;dr (too long; did not read) summaries from Semantic Scholar (S2).Footnote ^o Table 1 compares RAG summaries with tl;dr summaries; S2 tl;dr summaries are shorter, and probably more to the point, though it is hard to define “more to the point” precisely.

Figure 1. The query (top line) is followed by recommendations.

3.4 Scenario: compare and contrast recommendations

Figure 1 shows a recommendation engine.Footnote ^p We would like to provide users with explanations that work for them. Users probably do not want to know how the different recommendation engines work. It is probably not helpful to discuss details such as deep networks, embeddings, approximate nearest neighbors, and spectral clustering on citation graphs. Rather, we propose to offer users buttons like the “compare & contrast” buttons in Fig. 1. Users can click on one of these buttons if they want to know why a paper is being recommended. The implementation of these buttons is based on RAG. In this way, RAG can be used to address some of the explanation opportunities mentioned above. If a user clicks on one of these buttons, the relevant papers will be uploaded to a RAG system and then the RAG system will be prompted to not only summarize the relevant papers one at a time but also to compare and contrast pairs of papers at a time.Footnote ^q

3.5 Compare and contrast with VecML.com

Our GitHub provides several versions of RAG.py. In addition to the OpenAI version mentioned above, there is also a version based on VecML.com.Footnote ^r Here is an example that uses the VecML.com version of RAG.py:

The VecML code for RAG.pyFootnote ^s is even shorter than the OpenAI code for RAG.py.Footnote ^t As mentioned above, all of the RAG.py programs take one or more papers on the command line and upload them to the cloud. Then they read prompts from standard input and print responses to standard output.

If we run the VecML RAG.py program on two ACL papers, Church and Hanks (Reference Church and Hanks1990), Lin (Reference Lin1998), using the prompts below:

1. 1. Please summarize the paper on psycholinguistics.

2. 2. Please summarize the paper on clustering.

3. 3. What are the similarities between the two papers?

4. 4. What are the differences?

We obtain the following results:

1. Summarize (Church and Hanks 1990): The paper on psycholinguistics discusses the importance of word association norms in psycholinguistic research, particularly in the area of lexical retrieval. It mentions that subjects respond quicker to words that are highly associated with each other. While noun–noun word associations like “doctor/nurse” are extensively studied, less attention is given to associations among verbs, function words, adjectives, and other non-nouns. The paper concludes by linking the psycholinguistic notion of word association norms to the information-theoretic concept of mutual information, providing a more precise understanding of word associations.

2. Summarize (Lin 1998): The paper discusses a triangulation approach for clustering concordance lines into word senses based on usage rather than intuitive meanings. It highlights the superficiality of defining a word measure for clustering words without explicit preprocessing tools such as Church’s parts program or Hindle’s parser. The paper briefly mentions future work on clustering similar words and reviews related work while summarizing its contributions.

3. Similarities: The similarities between the two papers include a focus on analyzing language data, using distributional patterns of words, evaluating similarity measures for creating a thesaurus, and discussing the importance of smoothing methods in language processing tasks.

4. Differences: The differences between the two thesaurus entries can be measured based on the cosine coefficient of their feature vectors. In this case, the differences are represented in the relationships between the words listed in each entry. For example, in the given entries, “brief (noun)” is associated with words like “differ,” “scream,” “compete,” and “add,” while “inform” and “notify” are related to each other in the second entry. These associations indicate the semantic relationships and differences between the words in each entry.

Like summaries, output from the compare-and-contrast button looks impressive on first impression, but on further reflection, the responses can be long-winded, and sometimes seem to miss the point. It is not clear that the bot “understands” concepts such as similarities and differences. The bot’s discussion of differences is not addressing differences between the two papers, but rather differences within one of the papers among lexical items. We have obtained slightly better responses by rephrasing the prompts in various ways, but we will not discuss such complications here since this tutorial is intended to provide a gentle introduction (as opposed to SOTA-chasing).

That said, as an area chair and senior area chair for many conferences, we have seen many reviews from unmotivated (and unqualified) reviewers. It is not clear that RAG is worse than bad reviews, though there are obvious ethics concerns with using RAG (and unmotivated/unqualified reviewers) for high-stakes decisions with long-term implications for careers.

4.1 Problem decomposition

Uploading documents just-in-time addresses some of the opportunities raised above, but not all. Consider problem decomposition. Chatbots can easily add two small numbers, but they cannot add two large numbers. Children have the reverse problem. They struggle with memorizing multiplication and addition tables, but they have little trouble decomposing the sum of two big numbers into more manageable tasks. You do not have to teach a child fancy principles like superposition. They just get it.

A popular workaround to this problem is Chain-of-Thought (CoT) Prompting (Wei et al. Reference Wei, Wang, Schuurmans, Bosma, Xia, Chi, Le and Zhou2022). Since chatbots struggle with decomposing larger tasks into more manageable subtasks, the community has discovered that chatbots will be more successful if prompts come already predecomposed. In this way, CoT is seen by most researchers as an advance, but it might be more appropriate to think of it as a workaround. Just as a parent might cut up a child’s dinner to prevent choking incidents, so too, CoT Prompting cuts up prompts into more manageable bite-size chunks.

In short, chatbots are not magic. Chatbots will be more successful if we spoon-feed them. RAG spoon-feeds them by inserting the relevant content into the input before invoking the response, and CoT Prompting spoon feeds them by cutting up prompts into bite-size pieces to prevent hallucinations.

4.2 Opportunities for improvement

There are a couple of opportunities to improve the example above:

1. 1. OCR errors: garbage in $\rightarrow$ garbage out

2. 2. KISS (keep it simple, stupid):

   1. (a) It is safer to process fewer files at a time, and

   2. (b) To decompose prompts into smaller subtasks (CoT reasoning)

Consider OCR errors. RAG tends to process documents in very simple ways, typically as a sequence of chunks, where each chunk is a sequence of no more than 512 subword units. In fact, documents are much more complicated than that. Many documents contain tables, figures, equations, references, footnotes, headers, footers and much more. There are many OCR errors in older pdf files on the ACL Anthology, as illustrated in Table 2.

4.3 OCR errors and spaCy summarizations

It is remarkable how well RAG does on documents with OCR errors. Older technologies such as spaCyFootnote ^u are less robust to OCR errors, as discussed on our GitHub.Footnote ^v Table 2 shows that OCR errors are more challenging for spaCy than RAG.

Table 2. OCR errors are more challenging for spaCy than RAG