Digitisation and access

‘Digitisation’ is ambiguous: it could refer to the scans made of an original document or artefact; metadata about the object from a catalogue or finding aid; or searchable text or visual content from within it. However, each of these features represents different steps of processing and curation. In the case of textual sources, before there is text to search, an image of the document must be created, which might be a scan of a microfilm, a digital photo of a piece of paper, or a scan of a map. Creating a digital image and its metadata (and the conservation work that is often required before or after) may therefore represent only the beginning of a digitisation process, whereas for some materials it might also be its end. To create machine-readable information from an image (‘document processing’) might require further steps to produce searchable text. Each of these steps is computationally – as well as monetarily – expensive, not least because of the amount of labour required. Thereafter, cultural heritage organisations will also need to consider the costs of rights management (where necessary due to copyright or other restrictions), security, preservation, metadata curation, storage, discovery and data exploration interfaces. These will have implications for how the data is made available, and under what terms.

Broadly speaking, cultural heritage organisations have tended to provide access to digitised material in three ways (sometimes in parallel): direct access via institutional catalogue interfaces or linked digital library systems; online platforms offered as part of a third-party, subscription-based product (usually with free in-person access on-site); or data repositories, unlinked to catalogue records.

The example of the British Library’s (BL) newspaper collection combines a mix of all three approaches.Footnote ⁸ Although not a comprehensive or representative case study, it is a key national collection of enormous scale which, being familiar to many readers, makes it a useful object lesson. Over more than two decades, the BL has worked with public and private partners to support the gargantuan work of scanning and processing its collection of newspapers, which numbers c. 450 million pages. The result has been a patchwork of services and providers offering different access packages to different users. This patchwork allows us to see how the BL’s strategies have changed over time, in response to different partnerships, and how these in turn throw light on the range of possible strategies pursued by UK cultural heritage institutions at large.

The UK’s Joint Information Systems Committee (JISC) – a quasi-governmental non-profit, but which itself runs as a membership organisation – funded two rounds of newspaper digitisation in the mid-2000s, in partnership with the BL, as well as the academic publisher Gale.Footnote ⁹ These two projects set out to preserve and make available high-quality versions of historical British newspapers, amounting to c. 3 million pages selected according to criteria agreed as part of that digitisation project. However, the access arrangements were complicated from the outset, with libraries, universities and the general public offered different routes to the material, portending the splintering of the collection that has followed. Shortly after the conclusion of the second JISC project the BL entered into a partnership with a private genealogy company, FindMyPast, who have since gone on to digitise over thirty times as many newspaper pages as JISC (and counting), and which, outside of the Reading Rooms, are available to paying subscribers to the ‘British Newspaper Archive’ (BNA).Footnote ¹⁰

The BNA search portal is now the default way to access historical British newspapers; however, confusingly, this portal also provides access to ‘free to view’ material digitised in parallel by publicly funded projects. Alongside the paywalled newspapers, one can also find (if one knows that relevant filters exist) newspapers digitised ad hoc by projects within the library, as well as external partners, such the Living with Machines project, on which we worked 2018–2023.Footnote ¹¹ University researchers do not typically have access to the BNA because there is no institutional subscription option. Instead, they might have access to the eighteenth- and nineteenth-century collections digitised by Gale-Cengage (depending on the largesse of their library). These two platforms are emblematic of the types of products offered by commercial providers of cultural heritage collections across various document types: one caters to academic users (e.g. Gale or Proquest), where subscriptions are taken on by institutions, whereas the other serves a more general audience (e.g. FindMyPast and Ancestry), using a subscription model and search interface designed for the burgeoning genealogy and family history market. In both cases, users are restricted to browsing the collection or keyword search via a simplified web interface, rather than having direct access to the text of the newspapers in bulk – something that is necessary if historians wish to ask more nuanced corpus-wide questions than can be approached via simple search terms.Footnote ¹²

In parallel with the development of subscription portals, the BL has more tentatively made those subsets of its newspaper collection not under commercial licence available in other ways. As an important actor within the ‘collections as data’ movement, the BL has an established history of sharing open data, which since 2018 they have been making available via the public British Library Research Repository. For example, portions of the open-access digitisation efforts undertaken internally and with external partners can be downloaded as XML files (e.g. the outputs from the scanning and OCR process) directly from this repository, albeit without the wrapper of a user-friendly interface offered by the BNA. Likewise, the full text of the JISC newspaper collections, mentioned above, can be requested from BL Labs.Footnote ¹³ The example of newspapers shows how cultural heritage organisations have tried to square the circle: to get their collections digitised for the benefit of future generations with the help of commercial partners, while also carving out free-to-access sections where possible. The latter enables projects such as Impresso to access the ‘open’ BL newspaper titles (i.e. those that are ‘free to view’ via the BNA), so that they might be included in the construction of an international newspaper corpus.Footnote ¹⁴

That impetus can be seen in efforts across the sector, with repositories like the National Library of Scotland’s ‘Data Foundry’ leading the way. Through this platform the NLS openly shares not only its organisational data (e.g. catalogue metadata) and digitised collection files (e.g. scans of periodicals, gazetteers, and encyclopaedias), but also research datasets derived from these resources by third parties, including historians.Footnote ¹⁵ However, with resources in the sector scarce – especially for smaller cultural heritage institutions – progress in this direction remains piecemeal despite there being excellent precedents and models. The purpose of the UK’s Towards a National Collection Programme (TaNC) was to scope the best route to a UK-wide Digital Collection; but the study undertaken by Daniel Belteki, Arran J. Rees, and Anna-Maria Sichani within TaNC highlights the manifold barriers to scaling up current efforts due to inconsistencies between institutions’ operational practices and data standards as well as a lack of necessary technical or specialist skills (not to mention that AI has made navigating copyright ‘an increasingly time-consuming, yet vital responsibility for cultural heritage organizations’).Footnote ¹⁶ The creation of accessible digital assets requires ‘complex interactions across institutional structures as well as between software suppliers, cultural heritage organisations, and third-party partners’: which in turn require investments in technical and human infrastructure, as well as a willingness for cultural heritage institutions to collaborate and reach consensus.Footnote ¹⁷

Derived data

This denotes new information or datasets created from existing ones, for example where data is subject to a degree of abstraction or transformation such that it constitutes a new dataset, potentially thereby becoming free of any copyright in existence on the original. For example, if a library makes available images of book pages, and then OCR software is used to create full-text data, the latter would be derived data. If the library already provides that text, a new dataset could be derived by processing the text in one of several ways to enable different forms of research. This could include producing lists of named entities or word (or n-gram) frequencies. An n-gram is a sequence of n words which allows the analysis of texts based on word frequencies, but which does not preserve their position in the text and therefore prevents the original text being reproduced. An example of this is the n-gram data released for FindMyPast’s BNA titles (published by the LwM team, and a previous collaborative project), comprising time series for the most frequently occurring words from the corpus, but without the full-text from which they were derived, because it remains licenced.Footnote ¹⁹

Enhanced data

This denotes datasets that have been enhanced through processes such as cleaning, categorisation or enrichment with other features such as geo-coordinates, or linked open data, which link to stable identifiers on the web and ground references in sources to authoritative statements. A key feature of enhanced data is that it creates machine-readable data that can be processed computationally. An example of this is an enhanced version of the newspaper n-grams published by LwM colleagues mentioned above, which linked newspaper text to descriptive metadata for each newspaper title from which they were derived (such as places of publication, price and so forth).Footnote ²⁰ Another example of enhanced data that will be familiar to historians of the nineteenth-century is the Integrated Census Microdata (I-CeM): individual-level census records for Great Britain from 1851 to 1911 (England and Wales, 1851–61 and 1881–1921; Scotland, 1851–1901). This dataset itself has a history comprising multiple transformations, undertaken by different organisations over time with profound implications for access and reuse. The original census enumerators’ returns are held by The National Archives (TNA) and were scanned and processed by FMP in a similar arrangement to that struck with the BL regarding its newspaper collections. However, and in contrast to the case of the newspapers, these digitised census returns have been cleaned and coded by the I-CeM project (funded by the Economic and Social Research Council), which added consistent geographies over time and standardised coding schemes for many census variables, such as professions.Footnote ²¹

A further example is the ‘Tudor Networks of Power (TNoP) – Correspondence Network Dataset’, co-created by one of the present authors.Footnote ²² The history of this dataset spans several institutions: the original letters are among the Tudor State Papers, mostly held by TNA, but also within BL collections and Hatfield House, amongst others. These were digitised by Gale as ‘State Papers Online’, which links scans of most of the manuscripts with the Calendars of the State Papers, a long-running project to catalogue the manuscripts chronologically, begun in the nineteenth century (and still incomplete). The TNoP team gained access to the underlying SPO XML data and scans, from which they extracted documents likely to be letters (i.e. documents with a full sender and recipient metadata field), and cleaned and enriched the metadata for these records. They de-duplicated and disambiguated the names (sender and recipient) fields, creating unique resource identifiers and providing LOD identifiers where available. They did the same for place names, which were also geo-referenced; dates were rationalised to the Gregorian calendar, where possible; and ambiguous time-windows narrowed. Finally, the network of communications was reconstructed, with people (the nodes) connected by letters (edges).Footnote ²³

Aggregated data

This denotes something akin to the production of traditional reference works, involving work to aggregate or categorise data (or both). The names of some typical examples reveal their affinity with more traditional reference works: the World Historical Gazetteer, the Universal Short Title Catalogue and Early Modern Letters Online (which is described as a union catalogue) extend the long histories of cataloguing and the compiling of gazetteers or directories.Footnote ²⁴ An example of the latter from our own work is the Structured Timeline of Passenger Stations in Great Britain (SToPSGB): a dataset created by LwM.Footnote ²⁵ The underlying dataset was aggregated from a vast range of sources by Michael Quick and published in his Railway Passenger Stations in Great Britain: a Chronology. The book is a directory listing passenger station locations, as well as dates of opening and closure, making it a uniquely rich and detailed account of Britain’s changing railway infrastructure over time. With permission, the LwM project produced and released this dataset in order to provide the rich detail of this exceptional resource in a structured format, making it an open resource for historical research.Footnote ²⁶

Digital editions can be viewed as another example of data aggregation, bringing together documents that otherwise would not be accessible in one place. Editions traditionally published transcriptions of specific works or papers, collected according to some principle of selection such as theme or author. In both its analogue and digital manifestations editions thereby aggregate information by uniting scattered documents under one cover or web address. We might also note that digital edition projects are also examples of enhanced data as often these projects rely on the Text Encoding Initiative, which has, by virtue of setting common guidelines for publication, enabled editorial labour to enter the digital age, underpinning important initiatives such as The Women Writers Project (a collection of early-modern women’s writings) and the Old Bailey Online (which publishes London’s central criminal court Proceedings from 1674 to 1913).Footnote ²⁷

These brief examples of derived, enhanced and aggregated datasets issue from academic research projects led by historians (or other humanities scholars), and are designed for use primarily by historians. Sometimes these projects involved active collaboration with colleagues within cultural heritage organisations (e.g. the outputs of the LwM project), but others can be seen more as an asynchronous form of collaboration, building on the outputs of library and archive digitisation (e.g. the census microdata of I-CeM, or the Tudor letters dataset of TNoP), or cataloguing (e.g. the union catalogue of EMLO). These asynchronous engagements, however, often create new stages of life for the data, often away from the institutional homes in which the physical documents reside. In developing these datasets, historians must begin to address questions of how they are made accessible and maintained – practices that historians have not customarily been trained to perform. As the next section suggests, the variety of ways in which historical data is currently being made available to potential users makes for a complex set of practices, access arrangements and possible futures for the discipline, which requires us to re-imagine what a data ‘commons’ might look like.

For users of data

1. 1. Change citation practices to account for the nature of data. Historians are trained in the art of finding and citing research materials. However, best practices for which version of a non-unique document is cited remain inchoate, despite important research and advocacy around citation practices for physical versus digitised copies.Footnote ⁴⁶ For many, it is still the norm to cite the material document that sits behind a digital resource rather than the digital document or dataset (e.g. the edition of the early printed book, rather than Early English Books Online). This not only obscures what the researcher has actually observed in order to make their claim; it also hides how hugely reliant we are now as a research community on the products of digitisation and online portals. This occlusion means that it remains extremely hard to measure how much historical research data (in terms of both commercial platforms and data deposited on repositories) is actually being used and contributing to research. Historians must play a role in demonstrating the value of digitised collections by choosing to cite the digital sources. Doing so allows data creators such as cultural heritage organisations and research projects to measure the use of their data and demonstrate their impact.

2. 2. Choose open data over paywalled versions where possible. In addition to choosing to cite the digital version of documents, we suggest that where possible, researchers should seek out open rather than paywalled versions of data. We must, in other words, vote with our feet. This may mean some level of compromise, by asking whether a similar data source that is open access could be chosen instead of one that is perhaps more readily accessible, but behind a paywall. For example, if researchers want to access historical British newspapers, as we already outlined above, there are alternatives to the paid-for services of Gale-Cengage and the British Newspaper Archive (BNA) in the form of freely available full text versions, available on the British Library’s Research Repository.Footnote ⁴⁷ There will be equivalents for other sources and data types, although we recognise that this recommendation will not be straightforward for many practising historians, who may know neither how to manipulate and search such files, nor that this data even exists outside of the commercial platforms and portals to which they are ordinarily directed. However, with the right support and training where required, this may in fact involve a lower overall institutional cost; for example, where there is no or limited access to institutional subscriptions for paid-for platforms. The behavioural shift in seeking out open-access data will help provide cultural heritage colleagues and policy makers with evidence and encouragement in arguing for open-access digitisation and discovery platforms in future. As things stand, paywalled solutions – with their many limitations – will continue to be offered as a primary point of access.

3. 3. Educate yourselves and your students in open-access data. Demonstrating the value of open-access data is essential to change behaviour at a disciplinary level. Educators should be familiarising themselves with the data available through open-access repositories and passing this information on to their students as part of training in research skills. We suggest that the availability of new data should, to some extent, be steering our research and teaching, and that the availability of new open datasets could spark research and ideas for student projects.Footnote ⁴⁸ If a new generation of scholars change their behaviours by rejecting subscription models, and collaborating with colleagues in university libraries on initiatives to find and access open data, it has the potential to cause change.

4. 4. Improve digital skills and collaborate. How are historians to work with this open-access data once they have it? For most historians, accustomed to user interfaces with keyword search boxes, the prospect of working directly with data may feel daunting. However, open-access datasets can be rendered navigable with off-the-shelf tools, which allow researchers to move beyond basic search and browse functions, to perform more nuanced and powerful queries, tailored to their research questions. This is, perhaps, the bigger change in behaviour, and as such will require digital skills to be baked into the undergraduate curriculum. Established researchers who are amenable to such approaches will need either to undertake a degree of self-education, or to seek out and establish collaborations with new colleagues. There are a multitude of online and in-person opportunities available to the curious (such as The Programming Historian or the Oxford Digital Humanities Summer School), while those in universities may have access to advice from or opportunities for collaboration with colleagues in IT, Research Software Engineering, the Library, Computer Science, Linguistics, and Digital Humanities.Footnote ⁴⁹ Joining forces with colleagues with computational skills and experience may not require funding in the first instance if historians can seek out areas of complementary scholarly interest. The effort involved in such collaborations lies in establishing relationships, and working out the ‘trading zones’ between your disciplinary interests in ways whose value is hard to predict in advance, but which may be highly rewarding.Footnote ⁵⁰

For creators of data

1. 5. Create canonical datasets. FAIR research data principles recommend that data should be Findable, Accessible, Interoperable and Reusable.Footnote ⁵¹ To ensure that data is accessible, we encourage colleagues to work with their communities to develop canonical, open-access datasets. In many fields certain online databases have become, by default, the scholarly standard; however, where these are behind paywalls, they are not accessible to all. Not only is this not equitable, it also means that any historical arguments based on such documents are not necessarily reproducible (which is increasingly an expectation for rigorous digital historical research). We advocate community efforts to establish alternative datasets that are openly available to all. This may involve compromising on their size; or, having only derived, abstracted features, such as n-grams for research data that may otherwise be restricted, due to commercial interests or copyright. One of the additional benefits of focusing attention on building canonical datasets is that they can be selected for quality, or to meet certain standards of representativeness that a larger dataset might not be able to provide – issues that frequently dog historical collections, digitised or otherwise. Initiatives in this direction also enable collective effort to ensure iterative approaches to data cleaning or curation, and documentation. The benefits for scholarship should be clear: if research in digital or computationally inflected history is to develop a rich critical literature that parallels existing historiographies, then it must become possible for communities of scholars to gather around particular datasets (much as historians have returned to the same archives) in order to extend and challenge each other’s interpretations. This will only be possible if such datasets are freely open and available, without gatekeeping (such as making data available only on request). For a model of how this can work, we can look to computational literary studies, where Andrew Piper, Ted Underwood and their collaborators have published derived datasets from the Hathi Trust, which have been used at the centre of numerous studies.Footnote ⁵²

2. 6. Consider the discoverability of your data. We know that the proliferation of repositories is a barrier to data being findable (the first of the FAIR data principles). As the TaNC programme explores, having a national collections portal is a powerful way to facilitate discoverability (as well as interoperability); but the recommendations of TaNC will take time to be enacted, and there is unlikely to be the scale of funding to do for humanities data what DISSCO UK will do to unify scientific collections in a single platform. More realistically, we would recommend that sub-communities within history work to establish local norms by choosing to submit their research data to the same locations, which, crucially, should provide a durable Digital Object Identifier (DOI). The coverage of UKDS beyond ESRC-funded projects has been user-driven and could easily be expanded with coordination; and it has the benefit of resources to help users think about how qualitative data can be stored, contextualised and accessed.Footnote ⁵³ But equally a coordinated effort might opt for an institutional or disciplinary repository, the repo of the cultural heritage institution where the data originated, or even an open science repository such as Zenodo. As noted already above, efforts to create discipline or period-specific communities have been trialled by the Post45 and Nineteenth-century Data collectives. The fact that the latter has only three datasets goes to show that establishing new community practices can be a slow process, even with willing collaborators. We therefore suggest that any future efforts in this direction should seek the support of national and international associations and learned societies, such as the Royal Historical Society, perhaps underscored by updated requirements to publish data from the journals linked to these organisations.

3. 7. Create greater awareness of new datasets to drive reuse. Depositing data without signposting it is like burying it. One key way of raising the profile of new historical research data is to share it as a data paper in a venue such as The Journal of Cultural Analytics or JOHD. Data papers make datasets more accessible and discoverable – both practically and intellectually – to other historians. By providing an account of a dataset’s provenance, structure, affordances and value for future research, data papers pave the way for research data to be more readily reusable (the fourth of the FAIR principles). Perhaps just as importantly, these accounts also make visible the labour of the data creators for considerations such as promotion. Other modes of signposting may include the creation of published links to datasets’ DOI through cross depositing, indexing via reporting mechanisms (e.g. Researchfish, which populates the UKRI ‘Gateway to Research’, although this is to be shut down in 2027)Footnote ⁵⁴ and citations in linked publications. The more links we make, the more pathways there are to the data.

4. 8. Build communities around your data. The fourth of the FAIR principles – reusability – can also be supported by building communities around datasets Our experience collaborating on the Data/Culture project has demonstrated that some of the most effective ways to build and maintain a community is by delivering workshops and training, publishing tutorials and undertaking ‘community calls’ or other mechanisms for providing ongoing support to users of a given dataset.Footnote ⁵⁵ In our case the community call mechanism was trialled on a piece of software we developed on LwM (MapReader). We met online every month to share examples of ways our material was being reused by others, as well as offering technical support and critical discussion of the software itself. This format is common among software developers but could easily be adapted to build community and collaborations around a given historical dataset. The benefit of building such a community is that it supports the canonisation of research datasets by driving reuse and citation, as well as creating a community of practice.Footnote ⁵⁶

5. 9. Develop and support mapping exercises. While there is hope that future investment may provide a one-stop solution for finding historical data, it is likely to be steered to some extent by the UK government’s AI agenda, and will probably be delivered in partnership with cultural heritage institutions.Footnote ⁵⁷ Because these interests may not necessarily align with the interests and needs of historians, we suggest that mapping exercises are essential to record the development of the wider data landscape. This may need to be implemented from below by historians. The Data/Culture project suggested to the AHRC that existing reporting mechanisms could be harnessed for the purposes of indexing UK arts and humanities research data. UKRI-funded projects already have a reporting requirement through Researchfish. Despite known problems with the interface and the UK’s ‘Gateway to Research’ website, the reporting platform could be used to collect new datasets for a central index automatically, which could then link out to distributed repositories. Mapping data generated by completed projects is a harder task. One route would be to audit all projects funded over the last twenty years and to check which have resulted in data outputs, if they are still accessible, and if so, where they currently reside. Similar projects could be undertaken in other geographical areas in collaboration with other national funding bodies. More nuance could be provided to the resultant picture via a crowdsourced mapping exercise of data assets known and used within the broader communities whose work has been research-council funded. Comparing these two approaches would help show which projects have been effective at creating and sharing data, and which have allowed data to be forgotten or kept private; and relatedly, what factors distinguish one type of project from the other.

These strategies are needed due to the absence of systematic national policies and infrastructures for storing and accessing data created by and for historians. Of course, as we have gestured above, the slow progress towards existing recommendations might be set to change due to the big promises that are being made in the UK to support AI innovation. It might be that some of these endeavours will indirectly benefit the history community by pulling cultural heritage assets out from behind paywalls, and by financing the creation of new open datasets. Arguably our proposals are even more pertinent if those promises are delivered on, because they are being designed to serve different constituencies. Because the data landscape is likely to change rapidly in the coming years, we need to establish a set of practices that ensure the data we are creating and will rely on as a discipline is available for the entire community to use.