Raw opinion data

First, we collect all United States Supreme Court opinions available in the Harvard Case Law Access Project.Footnote ² We limit this historical court data to opinions issued in 1870 at the earliest, but this dataset only extends until 2014.Footnote ³ Court opinions and metadata from 2014 to 2024 are collected through the Supreme Court’s official website.Footnote ⁴ Then, using the Supreme Court Database’s list of cases (Spaeth et al. Reference Spaeth, Epstein, Martin, Segal, Ruger and Benesh2024),Footnote ⁵ we identify cases that are missing from these sources. We scrape missing cases from Justia using either U.S. Reporter citation or docket identifier.Footnote ⁶ Footnote ⁷ The combined dataset includes 24,890 unique cases and 36,430 opinions.

Over time, the number of cases in our dataset decreases, especially since the 1990s, as shown in Figure 1. The number of opinions issued each year has also gradually declined, with its peak in the 1980s, as shown in Figure 2.

Figure 1.

Number of court cases in every year in our dataset.

Figure 2.

Number of court opinions issued in every year in our dataset.

Creating the annotated dataset

After collecting the raw opinion data, we created a smaller dataset for hand annotation by sampling from the full dataset. First, we filter to opinions that likely perform statutory interpretation. Focusing on statutory interpretation was a strategic research choice. Where possible, we sought to ease the cognitive burden we placed on our annotators, and tracking categories of legal reasoning across different legal domains is taxing. The risk of this approach is that the model orients to statutory interpretation rather than law more generally. One reason we thought that this risk was tolerable is that much of the qualitative historical literature on trends in jurisprudence speaks in general terms that integrates over many different legal domains (Stiglitz and Thalken Reference Stiglitz and Thalken2024). As noted later, we evaluate this risk and find that model performance does not drop off in non-statutory domains, suggesting that the orientation for statutory interpretation is aligned with other legal domains (Stiglitz and Thalken Reference Stiglitz and Thalken2024).

To do this filtering, we select opinions that include any of the word tokens “statute,” “legislation,” or “act,” within 200 characters of the tokens “mean,” “constru” (i.e. construct), “interpret,” “reading,” or “understand.” We choose these words as a simple way to increase the likelihood that statutory interpretation is present in an opinion.

Of the cases that pass the initial statutory interpretation filter, we chunk the opinions by paragraph. Because we anticipate formal and grand reasoning to be relatively infrequent in court opinions, we up-sample paragraphs that are likely to have formal or grand reasoning by identifying the occurrence of seed words that are indications of formal or grand reasoning (Table 2).

Table 2.

Seed Words Used to Up-sample Paragraphs Likely to Exhibit Formal or Grand Legal Reasoning

This subsampled dataset was gradually created through periodic sampling from the full dataset during the annotation task. In the initial sample, we select 50% of paragraphs with no seed term, 25% of paragraphs with at least one of the grand seed terms, and the remaining 25% with at least one of the formal seed terms. This sampling procedure remains constant until the final stages of annotation, where we increase the proportion of formal and grand seeds to 40% in both classes (and 20% without any seed term) to increase the number of formal and grand examples in our annotated dataset.

The final annotated dataset includes 2,748 paragraphs. Of these paragraphs, 37% of paragraphs include a formal seed, 36% of paragraphs include a grand seed, and 28% of paragraphs contain no formal or grand seed. In this dataset, paragraphs from court opinions that were issued in the 1980s are overrepresented (Figure 3), yet the 1980s were also the decade with the largest number of opinions issued per year, as shown in Figure 2.

Figure 3.

Number of paragraph samples from every year included in our annotated dataset. Paragraphs from the 1980s are over-represented.

The annotation task

In the annotation task, our team assigned a paragraph one of three labels: formal, grand, or none. Formal and grand reasoning are our categories of interest, yet not every court opinion paragraph performs legal reasoning – many paragraphs recite procedure or facts, for example. For this reason, the “none” label represents an absence of legal reasoning or legal reasoning that is neither formal nor grand.

We created a codebook with definitions and examples of each type of legal reasoning to support the annotation team. Our final codebook, which evolved modestly in the first weeks of the task, is included in full in Appendix. This codebook includes a section for each of the three classes, and a definition and set of core and periphery examples for each class. The core class is intended to represent a canonical ideal of the class; the peripheral examples represent less central examples that still fit within a class.

We used the tool Prodigy as a simplified interface for the annotation task to reduce the cognitive workload of labeling data (Montani and Honnibal). This system allowed us to load the sampled paragraphs into a database system that presented annotators with a single paragraph and the class options (see Figure 4). Annotators were asked to assign a class label to the paragraph without any context (i.e. case title, year, author name, etc., were excluded). Additionally, we replaced any case citation with the token “[CITE]” using the eyecite Python package to identify citations (Cushman, Dahl, and Lissner Reference Cushman, Dahl and Lissner2021). Annotators were expected to choose a label using only their expertise in legal reasoning, the codebook, and the paragraph itself.

Figure 4.

Annotation interface for labeling. An annotator must assign a label to every highlighted sentence before assigning a label to the paragraph as a whole.

The annotation interface asks the annotator to consecutively label every highlighted sentence as formal, grand, or none. Then, after labeling every sentence in the paragraph, the annotator must assign a label to the entire paragraph. Though only the paragraph text and labels were used to train and evaluate the models, we included sentence-level labeling during the annotation task to ensure that every component of the paragraph was taken into account before the annotator reached a decision on the paragraph.

Annotation team training

The annotation task took place over the spring 2023 semester. After developing the initial codebook, we recruited four upper-year law students, all of whom had completed coursework relevant to identifying legal reasoning. Annotators were assigned background reading on legal reasoning to review before the initial training session (Schauer Reference Schauer1988), in which they were introduced to the classes and tested the annotation interface. All annotators were then given an overlapping set of paragraphs to label in the next week.

In weekly intervals, after labeling the set of assigned paragraphs, we evaluated the quality of the annotations with inter-rater reliability. We use Krippendorff’s alpha to account for the number of classes and annotators (Krippendorff Reference Krippendorff2011). Each week, the group discussed paragraphs with high disagreement to decide the paragraph’s best class label and form a better collective understanding of each class. Annotators were typically assigned anywhere from 25 to 200 paragraphs to label in a week. During the first weeks, all annotators labeled the same paragraphs to allow for careful assessment of inter-rater reliability at any point in the annotation task; in later weeks, after reaching sufficient inter-rater reliability, we continued assigning at least 25 paragraphs out of the total set to every annotator.

For the first four weeks, inter-rater reliability was both low and inconsistent from week to week. During this training period, we used disagreements during the weekly discussions to refine the codebook. The most successful codebook change was the introduction of a decision chart, which we believe reduced cognitive load in the labeling task by breaking down the classification of a given paragraph into a set of simple binary questions (Figure 5).

Figure 5.

The decision chart provided to the annotation team beginning in week 5.

After the introduction of this decision chart, inter-rater reliability increased and stabilized from week to week (Figure 6). Beginning in week five, the week following the introduction of the decision chart, we finalized the codebook and increased the number of paragraphs assigned to all annotators, so as to label as much data as possible.

Figure 6.

Weekly progression of Krippendorff’s alpha for paragraphs and sentences assigned the formal, grand, and none labels. The decision chart was added before week 5.

Low confidence annotations

During the annotation process, some paragraphs had such unclear legal reasoning that the annotator could not confidently assign a label without a group discussion of the paragraph. We added the “low confidence” label so that an annotator could signal this lack of certainty; this label could only be added in addition to the three primary classes.

If a paragraph received the low confidence label, the annotation team would deliberate the best label for the paragraph until forming consensus. Often, paragraphs with the low confidence label engage in an internal argument: for example, the paragraph might critique grand reasoning, but for the further improvement of that type of reasoning, not to promote formal reasoning. In these instances, we asked annotators to assign a label based solely on the paragraph text and to avoid making assumptions about the role of the paragraph in the entire opinion. Some of the peripheral examples in the codebook were added after discussing these low-confidence paragraphs (see Appendix).

Ending the annotation task

Annotation tasks are costly in terms of the time and energy of the annotators, as well as their compensation for performing the task. Every annotator in our project was also a law student, so we were careful to end the annotation task when the marginal improvement of additional training data became low. More data would certainly be better, as it would allow for more partitioning and testing of the models, but that must be balanced against the costs of human labor and the risk of diminished annotation quality as the task is prolonged.

After reaching our desired inter-rater reliability, we began evaluating model performance with each additional sample of data to identify the point at which additional labeled data had no empirical effect on model performance. We evaluate the performance of three fine-tuned large language models, BERT, LEGAL-BERT, and DistilBERT, to identify the point at which all three had plateaued on the various classes (Devlin et al. Reference Devlin, Chang, Lee, Toutanova, Burstein, Doran and Solorio2019; Chalkidis et al. Reference Chalkidis, Fergadiotis, Malakasiotis, Aletras and Androutsopoulos2020; Sanh et al. Reference Sanh, Debut, Chaumond and Wolf2020).

Based on the plateaus in Figure 7, we ended data collection after ten total weeks, the last six of which occurred after reaching adequate inter-rater reliability, once we found no substantial improvement in model performance with additional data. We included only data from the last six weeks in our annotated dataset and discarded all annotated data before the inter-rater reliability improvement.

Figure 7.

Model performance over 10-fold cross-validation across growing random samples of our annotated dataset, from 10% of the data to the entire dataset.

The annotated dataset

In total, there are 2,748 paragraphs in the annotated dataset, with an overall inter-rater reliability (IRR) of 0.63 Krippendorff’s alpha for annotations included in the final set. Although this IRR score represents an improvement over the annotator training stage, it remains below the ideal level for an annotation task. However, given the inherent complexity of classifying legal reasoning, particularly in cases where extracts may involve ambiguity, we regard this as an acceptable result.Footnote ⁸ We worried, moreover, that further amplifying annotator training or guidelines would reduce the task to a rule-following exercise, at the expense of the nuanced human judgment of our domain experts. Users of this dataset should recognize that the IRR score is lower than desired, but also that this is a task where a degree of expert disagreement is to be expected.

Table 3 shows the number of paragraphs assigned to each class label, as well as how many of those labels were also assigned the low confidence score. Though we upsampled paragraphs likely to use formal or grand reasoning, the “none” class was the most common. Grand reasoning is the second most common class, followed by formal reasoning as the least common class in our annotated dataset.

Table 3.

Number of Paragraphs Fitting Each Class Identified by Annotators, and Number Assigned to the Low Confidence (LC) Class by Its Initial Annotator

Evaluating models

Using the annotated dataset, we evaluate the performance of various models to optimize the quality of the predictions on the entire dataset. The objective is to find the model and training procedure that best align with the expert annotations in the annotated dataset. Because our object of interest is legal reasoning, we are most concerned with model performance in the grand and formal classes.

Related earlier work compares the performance of fine-tuned models, which update the model weights using training data, to generative models, such as GPT, prompted to identify formal, grand, and no legal reasoning. Full details of this model evaluation can be found in Thalken et al. Reference Thalken, EdwardStiglitz, Wilkens, Bouamor, Pino and Bali2023.Footnote ⁹

This work evaluates two fine-tuning procedures: the first procedure tunes a standalone model to classify a paragraph into one of the three classes. The alternative procedure breaks the classification task into two serial steps: first, classifying legal reasoning (i.e. a formal or grand label) and the absence of legal reasoning (i.e. the none label); then, for paragraphs classified as legal reasoning, whether the legal reasoning is formal or grand. The fine-tuned models include BERT-base, LEGAL-BERT-base, DistilBERT, T5-base, and T5-small (Devlin et al. Reference Devlin, Chang, Lee, Toutanova, Burstein, Doran and Solorio2019; Chalkidis et al. Reference Chalkidis, Fergadiotis, Malakasiotis, Aletras and Androutsopoulos2020; Sanh et al. Reference Sanh, Debut, Chaumond and Wolf2020; Raffel et al. Reference Raffel, Shazeer, Roberts, Lee, Narang, Matena, Zhou, Li and Liu2020).

Those fine-tuned models are compared to prompted encoder-decoder models. The authors use three prompting strategies, all of which mirror sections of our codebook (see Appendix). The first prompt uses descriptions of the classes, the second uses examples in a few-shot prompting strategy, and the third provides a chain of decisions that the model must make before assigning a label. The models used for prompting are all encoder-decoder models, including GPT-4, FLAN-T5-large, and Llama-2-Chat (7B) (OpenAI et al. Reference OpenAI, Adler, Agarwal, Ahmad, Akkaya and Aleman2024; Chung et al. Reference Chung, Le Hou, Zoph, Tay, Fedus and Li2022; Touvron et al. Reference Touvron, Martin, Stone, Albert, Almahairi, Babaei and Bashlykov2023).

We evaluate each model using five-fold cross-validation.Footnote ¹⁰ In each fold, we randomly hold out 25% of the data for evaluation and use the remaining 75% for fine-tuning the fine-tuned models (or omit this data in the case of prompted models). The same 25% subset is used to evaluate all models in a given fold. Final scores are reported as the average across the five folds. This procedure ensures that all models are evaluated on equivalent data, while preventing the fine-tuned models from gaining an advantage by having seen the evaluation data during training. Of all models and training procedures, a fine-tuned LEGAL-BERT model reaches the best performance in this task when fine-tuned to classify a paragraph into one of the three classes (complete results are in the Appendix). In general, fine-tuned models outperform prompted models; no prompted model performs a usable classification on legal reasoning.

Predictions

With this fine-tuned LEGAL-BERT model, we make predictions on all court opinion paragraphs in our entire dataset. The output of this fine-tuned model is a set of logits representing the model’s certainty in each of the three classes (“formal,” “grand,” and “none”); these scores are converted to probabilities across the three classes using the Softmax function. Our primary measurement of legal reasoning, used in Empirical Studies of Legal Reasoning, calculates a formalism score based on predicted class probabilities. At the paragraph level, formalism is weighted by class probabilities: $ {\overline{p}}_i=\left(1-{\pi}_{in}\right){\sum}_{k\in \left\{f,g\right\}}{v}_k{\pi}_{ik} $ , where class $ k\in \left\{f,g,n\right\} $ denotes the class of “formal,” “grand,” and “none” respectively; $ {\pi}_{ik} $ denotes the probability that the paragraph $ i $ is of class $ k $ ; and $ {v}_k $ is a value assigned to that class for the measure. We set $ {v}_f=1 $ and $ {v}_g=-1 $ . Thus, a paragraph with the full mass of probability on the formal class receives a score of 1, while one with the full mass on the grand class receives a score of -1. Paragraphs with indeterminate classifications return with scores closer to 0. Finally, for substantive analyses, we typically aggregate paragraphs to the opinion-level and normalize these scores as z-scores.

Analysis of models, in Using and Improving Models, uses the original probabilities derived from the model, given that the object of interest is the model’s certainty.

Validation

In this section, we present validation of our jurisprudence measurement. Some of this validation is already present in prior work:

• • A yearly analysis of jurisprudence based on our measurement aligns with theories presented by legal scholars (Stiglitz and Thalken Reference Stiglitz and Thalken2024). Prior work has explored this validation exercise and its contribution to debates regarding periods of legal jurisprudence in more detail below (Trends Over Time).

• • The classification performance of the best-performing language model remains relatively stable across the early, middle, and late periods of the same dataset (Stiglitz and Thalken Reference Stiglitz and Thalken2024).

• • Codebook exemplars score as expected by our measurement, with the formal example scoring substantially higher in formalism and the grand examples higher in grand reasoning (Stiglitz and Thalken Reference Stiglitz and Thalken2024).

• • Although the annotated training data consists solely of excerpts from statutory interpretation cases, we apply the model to all cases in the dataset, regardless of domain. In prior work, two annotators labeled 200 non-statutory examples. We evaluate our model, which was trained on statutory interpretation cases, against this non-statutory set and find that it performs nearly as well as it does on the original statutory examples (Stiglitz and Thalken Reference Stiglitz and Thalken2024).

In addition to these existing validation exercises, we present a new validation exercise where we evaluate the language that differentiates grand from formal reasoning. We use the Fightin’ Words algorithm to calculate this difference at the word level (Monroe, Colaresi, and Quinn Reference Monroe, Colaresi and Quinn2008; Hessel Reference Hessel2020). We begin by dropping any words that occur in more than 30% of all paragraphs or in fewer than 300 paragraphs in the full dataset. Then, for every remaining unique word token, we compare its probability in all grand paragraphs to its probability in all formal paragraphs. To do so, we calculate the log-odds ratio for each word, using an informative Dirichlet prior based on word frequencies across the entire dataset. Finally, we calculate z-scores to measure significance for each word. The result suggests how different language usage is between formal and grand paragraphs, at the word level.

Figure 8 shows the top 50 words that distinguish formal from grand, according to our measure. These results further validate our model, as formal words point toward segments of legal documents and the closed interpretation of a document’s meaning. In contrast, the grand words emphasize legislative history and legal and political actors beyond the legal text.

Figure 8.

Words that most distinguish paragraphs measured as formal to paragraphs measured as grand, calculated using Fightin’ Words.

Empirical studies of legal reasoning

Using these paragraph-level measures of legal reasoning, we can empirically analyze longstanding and novel substantive questions related to jurisprudence. In this introductory paper, we highlight three primary hypotheses: first, that the evolution of legal reasoning has now reached a point where formalism is the predominant mode of legal reasoning; second, that conservative ideology is correlated with formal reasoning; and finally, that a case that results in a conservative decision will have been decided with more formal reasoning.

Trends over time

In general, legal historians agree on the basic periods of legal reasoning, as it shifted toward and away from formal or grand reasoning: First, there is scholarly consensus that between the Civil War and roughly World War I, formalism was the predominant mode of legal reasoning (Thomas C Grey Reference Grey1983; Grossman Reference Grossman2007; Farber Reference Farber2015). Next, at some point in the mid-twentieth century, grand reasoning came to predominate.Footnote ¹¹ Legal scholars largely agree that after the 1980s, formalism once again became the most common mode of legal reasoning (Eskridge Jr Reference Eskridge1989; Thomas C. Grey Reference Grey1999; P. L. Strauss Reference Strauss2014). This lean toward formalism continues to the present (Doerfler Reference Doerfler2022; Stiglitz and Thalken Reference Stiglitz and Thalken2024).

Using the predictions from our model, we can compare these expectations regarding the periodization of legal reasoning to results from our computational model. We would expect formal reasoning to be the primary form of reasoning from the beginning of our dataset (1870) and through at least World War I; grand reasoning to rise in popularity during the early-to-mid 20th century, and then formalism to resurge in the 1980s into the present. To assess these historical assumptions, we take the predictions from the model at the opinion level, find the normalized average between formal (1) and grand (-1) reasoning across all paragraphs. We then calculate an average across all opinions issued in a year.

Our results support the core expectations of the legal historians (Figure 9). Formalism does, in fact, dominate from the 1870s and through the 1930s. There is then a rapid shift to grand reasoning following the judicial revolution of 1937, which remains dominant until the 1980s. Formalism surges during the 1980s and remains the primary mode of legal reasoning through all remaining years in our dataset, ending in 2024.

Figure 9.

Predicted legal reasoning across years in our dataset. Higher values are formal and lower values are grand.

Though these results support prior qualitative assessments regarding the general arc of legal reasoning, legal scholars disagree on the exact turning points when legal reasoning shifted from one form of reasoning to another. For example, legal scholars variously expect the early 20th-century shift from formalism to grand reasoning to have occurred as early as Lochner v. New York (Horwitz Reference Horwitz1992; Schauer Reference Schauer1988), around World War I (Llewellyn Reference Llewellyn1960; Gilmore Reference Gilmore2015; Kennedy Reference Kennedy2006), or decades later, at the “judicial revolution” of 1937 (Wiecek Reference Wiecek2001; Tamanaha Reference Tamanaha2008).

With the granular data provided by our model’s predictions, we can identify more exact turning points. In particular, more than any other year, the judicial revolution of 1937 occurred alongside an abrupt change in legal reasoning, from the formal to the grand reasoning. This result is visualized in Figure 9, and prior work has evaluated this turning point in detail and found that it is a statistically significant turning point (Stiglitz and Thalken Reference Stiglitz and Thalken2024).

Judicial ideology and reasoning

Common perceptions of legal reasoning align formalism with a conservative judicial ideology (Thomas C. Grey Reference Grey1999). Through our formalism measure, we can consider whether formalism does in fact correlate with conservatism. By way of preliminary inspection, consider the average formalism scores for recent justices. Figure 10 displays the average legal reasoning score used by every judge since 1973, excluding judges most recently appointed to the Court or those who wrote less than 50 opinions between 1973 and 2024.Footnote ¹² These results support the common categorization of Justice Scalia as a recent formalist (Segall Reference Segall1993). Justices Gorsuch, Thomas, Alito, and Kavanaugh also lean toward formalism. In contrast, Justices Marshall, Stewart, and Brennan used the most grand reasoning. These judge-level formalism scores further support that we live in a period of high formalism: all currently-serving Roberts Court judges included in our dataset lean more toward formal than grand reasoning.

Figure 10.

Predicted legal reasoning for every justice since 1973.

As indicated in Figure 10, judges commonly associated with more conservative ideology tend to exhibit formalism in our measure, and judges commonly associated with more liberal ideology tend to exhibit anti-formalism. To further evaluate this, we use a simple but common measure of judicial ideology, the binary party of the appointing President, and find that, on average, judges who are appointed by a Republican president use higher levels of formalism (Figure 11).

Figure 11.

Average formalism scores with 95% confidence intervals for judges appointed by a Democratic versus a Republican President.

Using and improving models

So far, the applications discussed were based on the final measure relating to Supreme Court opinions after Reconstruction. In addition to this measure, we also publish our annotated dataset and our best-performing model with adjusted weights.

That best-performing model can be used to extend the study of legal reasoning to other time periods or jurisdictions. For example, scholars may wish to study whether legal reasoning remains formal in future years. Or, they may want to understand the relationship between trends in legal reasoning on the Supreme Court and the lower courts. The model can be fed new data – for example, lower court opinions – and it will produce a measure of jurisprudence relevant to those opinions.

The annotated dataset, in turn, may be used for a variety of purposes. First, as new, still more sophisticated foundational models emerge, they can be fine-tuned on the annotated dataset. The measure of jurisprudence based on these newer models may outperform our best model. A superior measure may be able to address more fine-grained legal questions. These new measures derived from the annotation data will be of interest to scholars interested in substantive legal questions.

Second, the annotated dataset may be useful for those interested in language models or in the technical aspects of measuring legal reasoning. For example, the annotated dataset could be used for benchmarking model performance and to support the evaluation and comparison of both legal-domain and generalized models. We see our annotated dataset as a unique and robust benchmark for several reasons: the annotated dataset was assembled carefully, with extensive documentation requiring hundreds of hours of codebook refinement, annotation, and team conversations; the task itself is a challenging one, requiring both abstract reasoning and domain knowledge; the specific task would not be part of standard pre-training material for a language model.Footnote ¹³

Like other researchers (Orr and Kang Reference Orr and Kang2024; Walsh, Preus, and Antoniak Reference Walsh, Preus and Antoniak2024), we caution against the use of this dataset without attention to its domain-specificity and the known ambiguity of some labeled examples. Throughout the annotation process, we came across examples where two experts could arrive at different labels using valid human judgment. Precisely because this is a challenging task, this primes the annotated dataset for use in probing the limits of increasingly capable models. Indeed, though the ambiguous nature of many legal questions calls for caution, at core we see the ambiguity of the domain as an opportunity for the technical modeling of knowledge uncertainty.

To gesture in this direction, our initial analysis demonstrates that there is alignment between uncertainty on the part of our annotators (measured through the hand-labeled ‘low confidence’ class) and uncertainty on the part of the model, which can reasonably be measured through the probabilities assigned to each class. Using only predictions for paragraphs in the annotated dataset, we compare the distribution of class probabilities assigned by the fine-tuned LEGAL-BERT model to paragraphs with the low confidence (LC) label to non-low confidence (NLC) paragraphs. We expect the model to be more “confused” about the correct class on the paragraphs our team marked low confidence. This confusion will manifest in dispersion over predicted class probabilities.

We measure model uncertainty using the entropy of the class probability distribution. Entropy captures how evenly the model spreads its prediction probabilities across the possible classes. When the model is confident, it assigns a high probability to one class and low probabilities to the others, resulting in low entropy. When the model is uncertain, it assigns more even probabilities across the classes, resulting in higher entropy.

For this exercise, we begin with the model’s raw logit predictions for each class (“formal,” “grand,” and “none”). We convert these logits to class probabilities using the softmax function, which normalizes the outputs so they sum to one. Next, we compute the entropy of the resulting probability distribution for each paragraph. We then calculate the average entropy for LC (low confidence) paragraphs (0.789) and for NLC (non-low confidence) paragraphs (0.539), yielding an observed difference of 0.250. As expected, the LC paragraphs have higher average entropy, indicating that the model is more uncertain on these examples.

To assess the significance of this difference, we conduct a permutation test over 10,000 trials. In each trial, we randomly shuffle the LC labels, recompute the average entropy for the permuted LC and NLC groups, and calculate the difference between them. We then count how many times the permuted difference exceeds the observed difference in magnitude. Dividing this count by 10,000 gives the p-value. As visualized in Figure 12, the observed difference is statistically significant ( $ p<0.0001 $ ), providing strong evidence that model uncertainty is systematically higher on LC data.

Figure 12.

Results from the permutation test. The dashed line represents the observed difference between the average entropy of LC paragraph probabilities and the average entropy of NLC paragraph probabilities in the annotated dataset. The histogram shows the distribution of randomized differences generated through the permutation test.

Exercises such as this one may be exploited for a variety of questions. For example, technically this exercise draws a line between human uncertainty and model uncertainty. One concern with language models is that they can express false certainty, notably “hallucinating” about even simple facts or references (Magesh et al. Reference Magesh, Surani, Dahl, Suzgun, Manning and Ho2024). It is possible to ask language models to report their level of certainty, but the meaning of the reported quantity is unclear. The exercise in this section shows that it may be possible to recover meaningful assessments of model uncertainty.