Plain language summary

Historical documents often contain valuable information, but their unstructured and inconsistent formats make them difficult to use for large-scale analysis. Traditional approaches to information extraction, such as regular expressions and rule-based natural language processing tools, often struggle with these irregularities. This research explores how large language models (LLMs) can help scholars extract and organize data from challenging historical materials. We use the U.S. Department of Agriculture’s Plant Inventory books (1898–2008) as a case study. These volumes feature varied layouts, old fonts and inconsistent terminology.

Our approach involved a three-step process that combined the strengths of traditional and newer extraction methods:

1. 1. Optical character recognition: We used Azure AI Document Intelligence to convert scanned book images into raw text.

2. 2. Custom segmentation: We developed a specialized script to divide this text into logical blocks to handle entries that spanned multiple columns or pages.

3. 3. LLM testing: We evaluated commercial and open-source LLMs on their ability to extract specific information (e.g., plant names, type and origins) from the segmented text and structure it into an organized JSON format.

We found that performance varied across models. Open-source models, such as Llama, were constrained by context window limits when processing longer passages. While commercial LLMs showed improved accuracy in identifying and extracting relevant information, the newest models did not necessarily yield the best results. Overall, our findings suggest that LLMs provide a promising starting point for making unstructured historical data more accessible to computational humanities research.

Data

Our primary source of data is the Plant Inventory series, published from 1898 to 2008 by the USDA documenting the introduction of over 650,000 plant materials into the U.S. National Plant Germplasm System. While there was a longer tradition of introducing foreign plants into the United States, the establishment of the USDA’s “Section of Seed and Plant Introduction” in 1898 marked a turning point in centralizing and expanding this process. While the division underwent multiple changes over the years, including its name, it became an important organ of promoting the introduction of foreign germplasm by agricultural explorers, such as David Fairchild and Niels Hansen. The impact of these efforts can be seen throughout the American agricultural landscape, including the countless fields of hard winter wheat, soybeans and lemons. To systematically document newly imported plants and make them available to the growing number of experiment stations scattered throughout the United States, the USDA began publishing this information in booklets that would come to be known as Plant Inventory. Within these booklets, each plant was given a unique number to serve as a means of designation due to the complicated and multilingual context of global agriculture. Each entry also listed detailed information about the plants, including botanical and common names, collection location, collector’s name and descriptive notes about its potential uses or climatic suitability. This systematic documentation reflected the USDA’s effort to centralize agricultural experimentation and maintain a robust archive for plant genetic resources (Fullilove Reference Fullilove2019; Hodge and Erlanson Reference Hodge and Erlanson1956; Hyland Reference Hyland1977; Li Reference Li2022).

From 1898 to 2008, over 200 booklets were published as bound paper volumes. Initial publications were irregular, but they appeared quarterly from 1903 to 1941, shifting to an annual schedule in 1942. Beginning in 2009, they converted to a digital model. As the regularity of the publications shifted, so too did the format and information included. As seen in Figure 1, the differences could be dramatic, including the introduction of columns on the page or lumping together a range of numbers that might “continue” onto subsequent columns or pages. Such changes introduce additional complexities when trying to digitize the materials. These booklets, then, provide not only a window into the global origins of U.S. agriculture but also a case study for testing the limits of existing optical character recognition (OCR) tools and LLMs in processing complex historical documents.

Figure 1.

Page examples from Plant Inventory booklets.

Workflow

Given the complexity of our documents, which feature multiple formatting styles and irregular text structures, we developed a multi-stage workflow that includes the steps represented in Figure 2. The process begins by converting the PDF image into raw OCR output. This unstructured text is then embedded within a specific LLM prompt, which instructs the model to return the information as a structured final JSON output. A more detailed overview of this process is as follows.

Figure 2.

Pipeline for structured data extraction using OCR and LLM prompting.

Evaluation

To evaluate the prompt’s extraction performance, we created a ground-truth dataset by manually labeling 525 crop entries from 27 sample text chunks. Our evaluation criteria were designed to provide a granular analysis of performance, moving beyond a single similarity score that might obscure important distinctions between minor transcription errors and complete extraction failures. We therefore employed the following metrics:

1. 1. Output reliability: We measured the percentage of model outputs that could be successfully parsed as valid JSON. This metric assessed a model’s ability to consistently adhere to the required output format.

2. 2. Field-level accuracy: We calculated accuracy for each metadata field (e.g., name and type) by comparing the extracted value against the ground truth. For this, we first normalized both sets of text to account for minor differences. This normalization included converting text to lowercase, removing punctuation (e.g., “CUCUMIS MELO” vs. “Cucumis melo.”), and reconciling over- or under-extraction of entities (e.g., “Col. Robert H. Montgomery $\ldots $ ” vs. “Robert H. Montgomery”).

3. 3. Computational efficiency: We measured the average tokens consumed and the runtime per file for each model. These metrics provide insight into the computational cost and practicality of each approach alongside its extraction accuracy.

We evaluated a suite of state-of-the-art language models available in late 2024 on their ability to extract structured crop metadata from OCR-transcribed documents. Following an initial round of peer review, we updated our analysis to include several more recent models available by mid-2025. Our comparison included models from OpenAI (GPT-5, GPT-4o, GPT-4o-mini and GPT-3.5), Google (Gemini 2.5 Flash and Pro, and Gemini 1.5 Flash and Pro) and Meta (Llama3-8B), with the temperature set to 0 (when available) to ensure more deterministic outputs. While most models were accessed via commercial APIs, the Llama3-8B model was run locally on a Google Colab A100 High-RAM GPU instance. This local setup introduced additional overhead in managing dependencies and runtime stability, and both this and the Gemini 2.5 Pro failed to successfully process the full set of 27 test files.

An overview of the results, as shown in Table 1, reveals a clear tradeoff between model accuracy, speed and token efficiency. GPT-4o emerged as the top-performing model, achieving the highest accuracy in the fields of Name (98.66%) and Origin (97.13%). The Gemini 1.5 and 2.5 series models were highly competitive, though they showed variability in certain fields; for instance, Gemini 2.5 Flash scored lower on Presenter accuracy (74.19%). In terms of processing speed, GPT-3.5 Turbo was the most efficient model at 15.20 seconds per file, with Gemini 1.5 Flash following at 20.03 seconds.

Table 1.

Comparison of large language models on extraction accuracy, speed and cost

Note: “Success” refers to the number of files that the model was able to successfully process and return in the required JSON format. Subsequent results are based solely on the successfully-returned files. Overall accuracy is calculated by comparing the output from each model to a “gold standard” sample after standardizing for casing, punctuation and over-extraction of metadata.

A qualitative review of the discrepancies reveals several recurring error patterns that highlight the specific challenges of this extraction task. For instance, in the type field, models frequently struggled with semantic ambiguity; for a single entry, one model might extract a generic common name like “tree,” while another would extract the scientific family name, both of which could be technically correct but would fail a direct comparison against a gold standard expecting “acacia.” A second common issue was the over-extraction of structured metadata, particularly in the presenter field, where models would correctly identify a name but also include their title and affiliation (e.g., “Col. Robert H. Montgomery, Director, Coconut Grove arboretum”). Finally, we observed frequent cases of embedded descriptive phrases, especially in the origin field, where text like “at 12,000 feet altitude” was inserted into the middle of a place name. These examples demonstrate that failures were often not simple factual mistakes but rather issues of formatting, specificity and interpretation. While our post-processing framework was ultimately able to manage a majority of these errors, further prompt refinement and model fine-tuning would likely improve the consistency of the initial extraction (Brown et al. Reference Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal, Neelakantan, Shyam, Sastry, Askell, Agarwal, Herbert-Voss, Krueger, Henighan, Child, Ramesh, Ziegler, Wu, Winter, Hesse, Chen, Sigler, Litwin, Gray, Chess, Clark, Berner, McCandlish, Radford, Sutskever and Amodei2020).

Discussion

One key consideration in our process was the tradeoff between commercial and open-source language models. Commercial APIs (e.g., OpenAI and Gemini) typically offer ease of use, powerful performance, larger context windows and scalable infrastructure. However, they introduce concerns around consistency and reproducibility. Calling the same prompt twice may return different outputs, and proprietary models provide limited visibility into versioning, training data or system behavior. These challenges make it difficult to ensure transparency and replicability.

For this reason, many researchers in the humanities and social sciences have turned to open-source alternatives, such as Meta’s LLaMA models (Alizadeh et al. Reference Alizadeh, Kubli, Samei, Dehghani, Zahedivafa, Bermeo, Korobeynikova and Gilardi2025; Ziems et al. Reference Ziems, Held, Shaikh, Chen, Zhang and Yang2024). These models offer greater transparency and control, but they also demand significantly more computational power. As early-career researchers in Digital Humanities, our access to such resources was limited. We were able to run the LLaMA 3 8B model with a monthly subscription to Google Colab Pro, which was sufficient for small-scale evaluations. However, attempts to scale up to larger models like LLaMA 70B quickly ran into memory limitations, constraints that commercial API-based models typically circumvent. For researchers without access to institutional computing clusters, Colab Pro remains a practical option for working with smaller open models. Still, as of the writing of this paper, these models have their own drawbacks, including inconsistent performance on complex tasks and smaller context windows compared to commercial alternatives.

Moreover, our study provides a limited but productive glimpse into the trajectory of model change over time, along with how prompts can be model sensitive. When holding our prompt constant, the generational leap from older models like GPT-3.5 Turbo to the more recent GPT-4o is unambiguous as seen in Table 1, with accuracy scores jumping more than 30 percentage points in fields like origin. However, the progression among newer models is less linear. The performance of the “GPT-5” series, for instance, did not uniformly surpass that of GPT-4o on this task and came with a substantial increase in processing time. Factors, such as model size (mini vs. standard), intended function (flash vs. pro) and the incomplete runs of Llama3-8B and Gemini 2.5 Pro, are all important to consider when identifying models for information extraction. While this might point to a general yet uneven increase in capacity over time, it also underscores the challenge of prompt portability: a prompt optimized for GPT-4o will not necessarily yield promising results with Gemini 2.5 Pro.

Ultimately, LLMs are not a magic bullet for data extraction. They work best when paired with traditional methods, such as manual annotation, human-in-the-loop verification and domain-specific heuristics (Khraisha et al. Reference Khraisha, Put, Kappenberg, Warraitch and Hadfield2024; Schilling-Wilhelmi et al. Reference Schilling-Wilhelmi, Rıos-Garcıa, Shabih, Gil, Miret, Koch, Márquez and Jablonka2025; Schroeder, Jaldi, and Zhang Reference Schroeder, Jaldi and Zhang2025). By designing thoughtful validation strategies (e.g., checking field alignment, comparing PI numbers across pages and calculating similarity scores), we can build confidence in the outputs, even when working with the black-box nature of generative AI. Used strategically, LLMs can accelerate labor-intensive processes like historical metadata extraction while still preserving data integrity.