Introducing ‘ethos’ and the role of regulatory science

The term ‘ethos’ has a long history in philosophy, and is usually characterized by drawing on Aristotle’s definition of êthos, which denotes a person’s credibility as a function of his character. In science and technology studies (STS) and history and philosophy of science (HPS), the term has gained traction through the work of the sociologist Robert Merton (after whom was named the Robert K. Merton Center for Science Studies, a Berlin-based institution that has close ties to the Einstein Foundation Award). According to Merton, ‘ethos’ means a set of norms that contain ‘moral as well as technical prescriptions’ for how scientists should behave.Footnote ⁴ They are moral in that they give guidance about what is good, and they are technical in that they are ‘procedurally effective’. The Mertonian ethos is thus a set of qualities and values that are meant to secure the status, credibility and authority of the people and institutions who stand for this ethos. According to Judy Segal and Alan Richardson, Merton self-consciously appropriated Aristotle’s notion of ethos to apply it to the scientific persona. For them, the crucial question that a so-understood scientific ethos replies to is, ‘why should we believe science?’Footnote ⁵ We agree that this understanding of ethos is a more helpful framing of Merton’s concept than treating it as a particularly adequate sociological approach to describing academic professionalization.

A critical insight from the literature on objectivity is that there is no single answer to Merton’s question of why we should believe science. Rather, as Lorraine Daston and Peter Galison have argued, there are distinct forms of objectivity which are each responsive to specific fears about what might make science untrustworthy.Footnote ⁶ While Merton’s norms are attracting renewed attention in discussions of scientific reform, Bart Penders and colleagues have critiqued scientific reformers’ use of Mertonian norms in particular and virtue-theoretic frameworks in general as oversimplifications which do not recognize that there are different ways of enacting a virtue or that there are different moral programmes travelling together under the banner of reform.Footnote ⁷

STS and HPS scholars have instead offered analyses that give specificity to current scientific reforms, either by placing them in the context of broader historical trends or by emphasizing their continuity with previous styles of objectivity. Jill Morawksi, for example, has argued that the move towards ‘institutionalized regulative solutions’ to failures to replicate in psychology represents an extension of mechanical objectivity.Footnote ⁸ For her, the key continuity is the fear of subjectivity and the desire to eradicate the various cognitive biases that plague psychological science. Just as Arthur Waddington (a central actor in Daston and Galison’s description of mechanical objectivity) was distressed to find that his beliefs about the symmetry of nature had influenced his perception of splashing droplets, so too, she argues, are contemporary reformers distressed to see evidence of how confirmation bias or hindsight bias have influenced their data analysis practices. While Morawski sees reformers’ use of regulatory and auditing practices as part and parcel of mechanical objectivity, Penders treats contemporary reform movements as part of a broader history of scientific bureaucratization.Footnote ⁹ For Penders, the adoption of bureaucratic practices is not simply about reducing subjectivity, but distinguishing oneself and one’s science as being more advanced and civilized than other forms of science. The concept of an ethos speaks to status and authority as well as to epistemic concerns.

Daston and Galison’s description of mechanical objectivity shares many commonalities with our description of the regulatory ethos, such as the prescription that scientists should follow ‘a set of procedures that would … move nature to the page through a strict protocol’.Footnote ¹⁰ But, in contrast to Morawski, we argue that the fear that drives the regulatory ethos is not that scientists will impose their ‘unchecked will onto nature’; it is that they will do so in a way that is not traceable. As we will show, human-induced bias is not a problem per se as long as one can measure and document the bias. The fear is failing to report properly, committing fraud by not cohering to the preregistered plan, failing the audit.

Other scholars have already emphasized discontinuity with older scientific norms when studying regulatory science practices. Alberto Cambrosio and colleagues, for example, introduced the term ‘regulatory objectivity’ to describe what they argue is a distinctive set of norms.Footnote ¹¹ To them, the key distinction is that regulatory objectivity operates at the level of the collective – it is an ethos embodied not by an individual practitioner, but by a group. Cambrosio and colleagues’ description of regulatory objectivity is connected to our concept of the regulatory ethos, particularly in its focus on the development of procedures to coordinate collective action and the pursuit of compatibility and coordination over the pursuit of truth. Jeremy Freese and David Peterson describe what they call ‘statistical objectivity’ similarly, although they point to a shift from individual studies to groups of studies (rather than from individuals to groups of people) as their key feature.Footnote ¹² We argue, however, that the regulatory ethos has spread beyond the regulatory agencies, standard-setting organizations and consensus conferences that Cambrosio and colleagues studied, and has become an ideal to be embodied by individuals as well as groups. Cambrosio and colleagues argue further that collective emphasis of regulatory objectivity means that it is less concerned with the relationship between the observer and nature, and more concerned with coordinating action. As they put it, ‘under the regime of regulatory objectivity, biomedicine suspends the search for “true values” and replaces it with the establishment of conventions … it is less important to arrive at a truth (analytic or otherwise) than it is to ensure compatibility between different laboratories and different hospitals’.Footnote ¹³ This second point begins to depart from the very idea of a search for a mind-independent reality. This can be regarded as a further expansion of the concept of objectivity (as Cambrosio and colleagues argue), or can raise the question whether the norms and practices they describe should still be analysed under the rubric of objectivity. Our concept of regulatory ethos can be used as an analytic category to describe the phenomenon independently of this question.

The regulatory ethos sits at the intersection of regulatory objectivity, mechanical objectivity, bureaucratization and a Mertonian ideal of good science, making it at times difficult to see how the regulatory ethos is not just an enactment of one of these ideals. We argue that this is not a mere analytic difficulty for STS/HPS, but that the confusion of these overlapping norms and processes has in fact had a productive role in allowing the regulatory ethos to migrate outside regulatory science. From many different vantage points, the regulatory ethos can simply be mistaken as the scientific ethos, not a specific form of arguing for the trustworthiness of science to which there are viable alternatives.

A good deal of scholarship already exists in STS which attempts to define regulatory science based on the context in which science is conducted or its orientation towards policy, to highlight the wide range of activities which could plausibly count as regulatory science, and to contrast regulatory science with other types of science.Footnote ¹⁴ Our aim is not to contribute to this literature. Rather, we treat sites of regulatory action as separable from the ethos itself so that we can better understand how the moral and technical prescriptions of the regulatory ethos are increasingly not limited to regulatory science contexts, but extend to the evaluation of research and researchers in general.

Defining this ethos as regulatory is important because it allows us to identify links between regulatory science and contemporary reform movements in academic science, which in turn allows us to use the existing literature on regulatory science to better understand these developments. In the United States, agencies such as the Food and Drug Administration (FDA), the Environmental Protection Agency (EPA) and the US Department of Agriculture (USDA) have been key sites for the creation of regulatory standards for conducting scientific studies and analysing data, such as the Clinical Laboratory Improvement Amendments standard (which regulates laboratories performing clinical tests on human samples) and the Good Laboratory Practice standards (which regulates the production of data on the safety and/or efficacy of chemicals such as pesticides or pharmaceutical drugs).Footnote ¹⁵ The twentieth century has seen worldwide growth in agencies and committees dedicated to the quality assurance of products and processes. Work conducted by the World Health Organization’s International Agency for Research on Cancer, for example, intersected with discussions at the US EPA on whether (and, if so, which) Petri dish or animal test is sufficiently valid and feasible for qualifying that a chemical can cause cancer and therefore should be legally regulated.Footnote ¹⁶ Localizing the promulgation of the specific form of this ethos of science within the institutionalization of regulatory bodies can help to address the history, sociology and politics of the moral project and its epistemological implications.

Regulatory agencies have been successful in defining and disseminating a specific ethos in no small part because of their role as gatekeepers to the market – they can compel would-be participants to collect, analyse and disseminate data in a particular way. The presence of systems to monitor compliance has further reinforced regulators’ preferred mode of conducting science. Simply put, regulatory agencies have the institutional power to make others adopt their ethos. This does not mean that this ethos is confined to activities that regulatory agencies are legally empowered to regulate. As Daniel Carpenter has argued in his history of the FDA, the agency’s gatekeeper role has meant that ‘its mere suggestions and intimations induce compliance’ from drug developers, resulting in the widespread adoption of practices even in the absence of legal enforcement.Footnote ¹⁷ The diffusion of regulatory standards also takes place within institutions such as an individual company, where following standards such as Good Laboratory Practice standards can streamline operations within the company and provide documentation that a company adhered to the rules, thereby reducing the company’s liability. Indeed, as Jeremy Greene has shown, the establishment of a distinct ‘FDA science’ contributed to developing clinical trials into an industry for the newly founded contract research organizations in the late twentieth century.Footnote ¹⁸

Many instances of regulatory science may only partly embody the ethos that we describe, and our concept of the regulatory ethos is not meant to be synonymous with the actual actions of regulatory agencies, or of the scientists who produce data for regulatory purposes. This, we believe, is one of the strengths of the concept: creating some analytical separation between regulatory science and the practices and ideals commonly associated with this work allows us to more easily identify similarities with work in differently regulated sectors, and to consider how these ideals and practices might function in spaces without the same market incentives, legally authorized gatekeepers or compliance-monitoring systems. Taking this perspective can aid us in considering how, for example, the practice of preregistration functions differently in research regulated by the FDA compared to research regulated by institutional ethics committees, and can help us to elucidate which elements of a regulatory ethos are replaced and which might still be enforced in contemporary political appeals to so-called deregulation. The motivation of our characterization of the elements of this ethos, and our discussion of their enactment in various case studies in the twentieth and twentieth-first centuries, is to open up a broader debate about the origins of this ethos, its historical and contemporary forms and enactments, and the implications of and alternatives to this kind of thinking about good science.

The four features of the regulatory ethos: planability, uniformity, accountability, reliability

The first feature of the regulatory ethos is that, to borrow a phrase from Lucy Suchman, it favours plans over situated actions.Footnote ¹⁹ In this view, good scientists should specify in advance the steps that they will take (as in the case of a clinical trial) or follow a pre-specified series of steps (as in the case of test guidelines or standard operating procedures). Once specified, these plans take on rule-like qualities that govern scientists’ conduct, and the execution of the plan becomes an end in itself. In contrast to a situated model of action, where the trajectory of science is expected to evolve as scientists engage with the world, the unexpected moments that inevitably arise as scientists attempt to carry out their plans are seen as problems to troubleshoot.Footnote ²⁰ Rather than privileging iteration, improvisation or a more expansive notion of problem solving that involves finding new paths to get to a goal, the regulatory ethos encourages scientists to bring their actions back in line with the original plan to the extent possible and to document any deviations.

An example from medical science that illustrates the virtue of following plans is provided by Jingwen Li’s paper in this issue, which reconstructs a lawsuit brought by a customer against his optician in 1920s Singapore.Footnote ²¹ Unhappy with the glasses he had been prescribed, the customer sued, claiming that the prescription was faulty. As evidence that the doctor had behaved appropriately, the defence showed that he had followed proper procedures, analysing the patient’s eyes with a retinoscope and prescribing accordingly. The plaintiff’s lawyer, in contrast, argued that what distinguished a correct from an incorrect prescription was not the procedure followed, but how the patient felt when wearing the resulting glasses. A competent optician should conduct a retinoscopy examination but should also modify the calculated prescription as needed to produce a suitable pair of glasses for that patient. What this case and its interpretations show is that a focus on following rules tends to privilege specific kinds of evidence, namely evidence that is produced by mechanical, interrater reliable procedures, as well as a distinct kind of expert who knows how to follow the plan correctly. If the plan nonetheless does not work out, the responsibility can be delegated to external factors, such as the unruly customer. This feature of the regulatory ethos aligns closely with mechanical objectivity, in that it idealizes practitioners who behave in machine-like ways rather than those who exercise their trained judgement. Yet Li’s case study reminds us also of the limits of objectifying the subjective experience of vision correction.

A second feature is that the regulatory ethos favours uniformity over heterogeneity, difference and locality. Uniformity encompasses two kinds of prioritization: first, it privileges a broad accessibility of methods that does not depend heavily on individual performance and adjusting of technical and epistemic resources. Second, uniformity prioritizes framings of the objects of analysis that abstract away their idiosyncrasies, and focuses on objectified parameters. Uniformity goes hand in glove with strict adherence to plans, since alignment with pre-specified plans is more difficult to achieve when the conditions of work differ. In this view, the ideal plan is one that can be executed in a variety of locations and does not require access to highly specialized reagents, technologies or workspaces. Likewise, the ideal scientist is one who is interchangeable for any other similarly trained scientist, not one who has a particular aptitude for executing a technique or who has cultivated a unique skill set over the years. In this view, the world should be treated as (or remade into) a uniform terrain where any reasonably trained person can carry out a protocol and get the same result. The regulatory ethos has a democratizing impulse that changes the character of both science and scientists: expertise becomes a matter of discipline rather than skill, and specialized techniques become a liability rather than an advantage because they are heterogeneously distributed.

Colleen Lanier-Christensen’s study illustrates this ethos at work in the selection of methods of the Association of Official Agricultural Chemists (AOAC) for assessing the chemical composition of fertilizer.Footnote ²² In considering potential methods to adopt as a standard, the AOAC wanted methods that were accurate, precise and reliable, but also ‘appropriate for routine use’. Lanier-Christensen shows that when those two goals were in conflict, the AOAC prioritized uniformity. The United States Commissioner of Agriculture agreed, arguing that it would be preferable to use a ‘less perfect’ method than a more precise method that only some laboratories would be equipped to perform. Importantly, Lanier-Christensen stresses the difference between quality assurance as being only about the process, and quality control, which is directed at the outcome. She understands the introduction of Good Laboratory Practice standards in the 1970s as a form of quality assurance without quality control that has prioritized toxicological studies which follow these – contested but standardized – procedures over investigative studies that have tested alternative hypotheses but don’t have the same status in regulatory and juridical contexts as the (often industry-friendly) procedural studies do.Footnote ²³

Angela Creager’s study of work by federal agencies to evaluate potential laboratory tests for chemical carcinogenicity likewise shows a preference for accessible methods.Footnote ²⁴ A National Cancer Institute committee charged with evaluating tests indicated that they would not support a patented test when alternatives were available in the public domain. The fact that the inventor of the eponymous Ames test had begun sharing his bacterial strains even before publishing on them was a factor in its eventual adoption as a regulatory standard, since it meant that government, industry and academic scientists alike had access to the materials needed to carry out the protocol. Historical studies of the technique used to standardize the measurement of electrical resistance reveal a similar pattern. The British used metal coils as a means of measurement because they were well suited to use by telegraphists, even though metal coils were not as accurate as other available techniques.Footnote ²⁵ The German mercury-based technique offered greater precision but at the cost of radically decreased accessibility, and when the German Reich legalized the ohm in 1898, they chose a less precise but more widely reproducible version of the technique as the basis for their standard.Footnote ²⁶ In contrast to the examples discussed in outlining the first feature, the actors in all of these cases resist the pursuit of ever more refined means of producing mechanical impressions of nature. These examples align more strongly with regulatory objectivity, since they sacrifice potential precision for the ability to coordinate action across different domains.

A third feature of the regulatory ethos is that it treats documenting and reporting as processes that enable auditing rather than communication. Conceptualized as a communicative tool, a methods section is an imperfect means of transmitting information, and the shortcomings of this tool can be repaired through mechanisms such as face-to-face communication. Conceptualized as an audit trail, a methods section is a chronological record of actions that allows a third party to detect deviations from a protocol, problematic uses of methods, or even instances of fraud. Incomplete methods information is not a communicative gap to be repaired but a warning sign that may cast doubt on the credibility of the findings. This element of the regulatory ethos reflects the logic of accounting that has not only formed modern economics but also transformed evaluations writ large, including the evaluation of education, science and health care.Footnote ²⁷ Accounting pairs well with the aforementioned feature of planned action: in accounting, the yardstick of right and wrong is compliance to the initial plan. This is a very different register of trustworthiness than building trust on trained judgement, interpersonal relationships, practical experience or tacit knowledge, all of which challenge the exchangeability and uniformity of labourers that the regulatory ethos prioritizes. Reporting, documenting and compliance have a double role of delimiting sources of bias and creating a written trail of evidence that testifies that the research process has been done correctly.

Christopher Phillips’s paper exemplifies this aspect by showing how the FDA propagated off-the-shelf statistical tools for documenting that a rational and uniform procedure had been performed to determine the effects of a chemical.Footnote ²⁸ This use of statistical tools allowed regulators to define a so-called ‘virtual safe dose’ of a chemical despite the inherent uncertainties of such measurements. It did not eliminate the problems of identifying a practical zero residue or a lowest detectable amount, but it did allow regulators to move forward by documenting the assumptions that they were making and the steps they were taking in light of known uncertainties. The goal of such documentation was not to communicate to other scientists how they might perform similar assessments, but to transparently disclose how the FDA had acted in light of existing variability and uncertainty. This example illustrates the distinction we drew earlier between the fear of bias itself (which drives mechanical objectivity) and the fear that bias will not be properly disclosed, which is what we see as central to the regulatory ethos.

This leads us to the final feature: the belief that if every step of the research process has been done correctly – planning action, reproducible conduct of the plan, careful documenting and reporting – this will result in validated results and therefore valid knowledge. The fact that the process is validated vouches for the validity of its results (the vouching expression stems from Nancy Cartwright, who characterized a similar epistemic expectation with respect to the privileged role of randomized controlled trials to generate extrapolatable resultsFootnote ²⁹). In the most extreme case, the validity of a research result (i.e. that the result is actually informative of what it is intended to be about) becomes secondary to the validation of the process (i.e. that the result is reproducible).

Yingying Han’s examination of the rise and fall of the forced swim test is an example of how the validity of this behavioural test became secondary to its broad use as a reliable validation tool that provided comparable data for assessing antidepressants.Footnote ³⁰ Animal activists, however, successfully referred to scientific studies that questioned the validity of the relationship between an important symptom of human depression and the time rodents spent finding a platform in a water pool before giving up to make the case against the use of this test in depression research. According to them, the forced swim test led to invalid extrapolation from animals to humans, and therefore should lose its status as a standard test, and even be banned. In a similar vein, Creager argues that while scientists and regulators were able to validate the Ames test as a method, they were less successful in their efforts to validate it as a model.Footnote ³¹ They succeeded in developing standardized protocols that generated reproducible results across laboratories, but identified several limitations of the test as a tool for identifying compounds that might cause cancer in humans. A chemical could cause cancer without causing mutations, for example, which meant that the Ames test missed known carcinogens such as asbestos. Phillips’s actors likewise objected to the claim that the ‘virtual safe dose’ was synonymous with actual safety, or that the calculation could be straightforwardly mobilized in larger regimes of risk calculation to protect public health.

In Creager and Phillips’s cases, we see how tests became embedded in the regulatory apparatus without fully erasing the question marks around their validity. Lanier-Christensen’s case, in contrast, offers a clear example where actors prioritized validation over validity. Industry and government actors both recognized that a reliable, reproducible test could serve an important market coordination function even if that test was not the most accurate measure of the actual content of fertilizer. As long as each fertilizer was measured by the same yardstick, consumers could still make choices between products and companies could benefit from fair, predictable assessments of their products. These examples demonstrate two pathways through which validation can supplant validity – in the first case, because the practical necessity of producing science for regulation allowed validation to carry the day even in the absence of validity, while in the second case validity was deprivileged even though it was available because validation was more aligned with the needs and interests of the actors within the sphere being regulated. Han’s case demonstrates that this alignment is a reversible process and needs to be continuously maintained and hedged: with the diversification of interests and actors, the scientific legitimacy of a decade-long validation consensus between regulatory agencies, scientific communities and pharmaceutical companies came under pressure, and the never-resolved debate about the validity of the forced swim test gained new momentum.

Taken together, these four features and their illustrative cases highlight an important function of the regulatory ethos: it serves as a compass that guides knowledge production and decision making by striving to create trans-spatial and transtemporal processes that will generate generalizable facts. As Ted Porter and others have shown, early instances of regulation were important for building consumer trust in products that circulated beyond the environs in which they were produced.Footnote ³² Standardized bushels of grain or protocols for measuring antibodies allowed for commercial trade, public-works projects, medical practice and more to move forward in an increasingly democratized and globalized world. Standards, rules, bureaucratic processes and systems of measure worked because, as Porter put it, they ‘extract[ed] a severe discipline from [their] users’ and thus had a ‘powerful appeal to the wider public’ because they impl[ied] ‘personal restraint’.Footnote ³³ They are critical to establishing what Geoffrey Hosking calls ‘strong thin trust’ – that is to say, trust in institutions, of which we often know little, which define rules and enforce rule following.Footnote ³⁴ And the public display of that rule following serves as a technology of distinction that marks out good from bad, advanced from primitive, and civilized from uncivilized – both in science and elsewhere.Footnote ³⁵

Historians might question how new this development is and point to older means of establishing authoritative norms for science and technology through quasi-regulatory bodies such as the academies of sciences. Surely there are undercurrents to the advancement of the regulatory ethos that have a much longer history than the one that is connected to the rise of regulatory agencies in the past century, which we focus on in this paper. However, as Melinda Baldwin has shown with respect to the prehistory of peer reviewing – and Merton’s (and Harriet Zuckerman’s) misunderstanding of its epistemic role in academies’ periodicals of the seventeenth century – the same practice can be attuned to quite different ideals: while ‘refereeing’ originally meant receiving the advice of other scientists or scholars, the renaming of this practice as ‘peer reviewing’ in the 1970s reflected a shift towards gatekeeping and rules of procedure to assess the quality of scientific research.Footnote ³⁶ The long history, but also changing understanding, of the role and ideal of reviewing is thus not a counterexample but rather an additional illumination of the rise of the regulatory ethos of science. It is no coincidence that this ethos spread with the institutionalization of national or international bodies that manage and regulate science, such as national funding agencies (cf. Baldwin’s example of peer reviewing), or regulatory agencies that act as gatekeepers to market access and certifiers for the security or efficacy of chemicals and biologicals.

In the next section, we examine the recent history of reproducibility reforms to demonstrate how the slippage between the regulatory ethos and the scientific ethos writ large has facilitated the spread of norms and practices commonly associated with regulatory science into other sectors. Many of the features that we have defined as sitting in opposition to regulatory science – evolving plans, results that depend on highly specialized skills or techniques, reliance on interpersonal trust rather than documented rule following – are features that have come under heavy criticism in the last decade. We show how reformers have reached for techniques common in regulatory science, such as preregistration and reporting checklists, as a means of cultivating a regulatory ethos outside regulatory science.

The replication crisis and the regulatory ethos

The Dutch psychologist Eric-Jan Wagenmakers described 2011 as a ‘year of horrors’ for psychology.Footnote ³⁷ First, the Journal of Personality and Social Psychology (JPSP) – a top journal for the field – published an article by Daryl Bem claiming that people can see into the future. Second, a researcher whom Wagenmakers described as one of social psychology’s ‘brightest young stars’, Diederick Stapel, was found to have been fabricating data for decades.Footnote ³⁸ Both of these events, in Wagenmakers’s view, pointed towards problems with how psychologists conducted, analysed and peer-reviewed experiments. If standard processes failed to detect dozens of fraudulent papers but accepted the existence of extrasensory perception, what did that say about the integrity of those processes? ‘One may well wonder about the scientific status of a field in which an academic serial-killer can wander loose, undetected, for decades,’ Wagenmakers mournfully concluded.Footnote ³⁹

Wagenmakers was particularly galled that JPSP had refused to publish manuscripts that failed to replicate Bem’s results, arguing that knowledge of failed replications would have aided the field in detecting problems in both Bem’s and Stapel’s work, since ‘from presented work it is often completely unclear to what degree the data were tortured to obtain the reported confession’.Footnote ⁴⁰ While Wagenmakers focused on replications as a tool to identify problematic scientific results, others focused on disclosure itself as the core problem to be solved. If it was evident from the beginning exactly how much a data set had been ‘tortured’, then problematic findings could be identified even without conducting a replication.

Joseph Simmons, Leif Nelson and Uri Simonsohn took this second approach, focusing on documenting and making visible existing practices rather than changing them.Footnote ⁴¹ They designed an experiment to show that standard methods of statistical analysis could prove an impossible hypothesis: that research participants were younger if they listened to the Beatles than if they listed to a control song. The trick was that the participant’s age was only one of many dependent variables that they measured: participants’ father’s age, mother’s age, gender, political orientation, self-reports of how much they would enjoy eating at a diner and several other variables were also measured but showed no differences between the Beatles intervention and control groups. In a particularly impactful figure, they presented what they called a ‘requirement-compliant’ abstract that included all the variables in grey text with the selected information that would be included in a typical abstract in bold. If researchers were required to report all the greyed-out information rather than just the variables that produced statistically significant associations, it would be immediately obvious to readers that the one association was likely a chance variation rather than a true finding.

Simmons and colleagues’ interpretation of the forces driving this selective reporting of information is in line with Morawski’s analysis of the ‘psychologizing’ framing of what would come to be known as the replication crisis.Footnote ⁴² They argued that most psychologists were not wilfully attempting to deceive their readers; they were simply self-serving in their interpretations, and pointed to a large body of research from psychology and elsewhere showing that people are ‘remarkably adept at reaching justifiable conclusions that mesh with their desires’.Footnote ⁴³ Simmons and colleagues’ solutions, however, did not call for reining in these biases, but merely for documenting their presence. They proposed reporting requirements as a solution that left ‘the right and responsibility of identifying the most appropriate way to conduct research in the hands of researchers, requiring only that authors provide appropriately transparent descriptions of their methods so that reviewers and readers can make informed decisions regarding the credibility of their findings’.Footnote ⁴⁴ In a retrospective piece published several years after their original article, they noted that several journals had adopted their reporting requirements, although they noted with amusement one that had rejected their suggestions because it threatened to ‘dull … some of the joy scholars may find in their craft’.Footnote ⁴⁵

Reporting requirements are a widely used tool of regulators, and a tool that embodies many aspects of the regulatory ethos. FDA requirements for reporting are vast, and include, for example, reports on clinical trials testing investigational new drugs, adverse events or deaths from drugs or devices, or how much of a drug a manufacturer has produced or packaged. These requirements homogenize what might otherwise be unstructured and heterogeneous information. For example, because a chemical compound can be described by a structural formula, molecular formula, generic name, trade name and so forth, regulated companies are required to provide a CAS registry number that uniquely identifies around 300 million chemical substances. Reporting requirements are also built to enable auditing rather than communication, providing the FDA with information needed to manage risk and monitor the drug supply chain.

Reporting checklists were one of the first tools that reformers reached for in the early days of the replication crisis, and Simmons and colleagues were far from the only ones to make this suggestion. As Nelson and colleagues have shown, while written discussions of replication and reproducibility emphasize different causes and consequences of the problem, they tend to agree on transparency and open-science measures as the solution.Footnote ⁴⁶ An early conference at the US National Institutes of Health on rigour and reproducibility resulted in the publication of paper that the authors framed as a ‘call for transparent reporting’, which specified four features that should be reported in all biomedical papers.Footnote ⁴⁷ Nature Publishing Group followed suit in an editorial policy change announced in 2013, which eliminated space limits on methods section and introduced a reporting-checklist requirement for all authors.Footnote ⁴⁸ In their editorial announcing this change, they noted that while not all scientists had access to facilities to validate their reagents or the resources to increase their sample sizes, they did have the capacity to produce ‘a full report of how a study was designed, conducted and analysed that will allow reviewers and readers to adequately interpret and build on the results’.Footnote ⁴⁹

Preregistration is another favoured tool of replication reformers, as Penders has explored, and like reporting requirements it is both used by regulators and embodies the regulatory ethos.Footnote ⁵⁰ Preregistration takes reporting a step further by requiring researchers to state in advance how they will collect data, analyse it and judge the success of their experiment. Deviations from the preregistered protocol are to be avoided where possible and documented where unavoidable, and the presence of the preregistration – held by an independent third party and available for scrutiny by others – vouches for the integrity of the experiment and the knowledge that it has produced.

Until recently, formal preregistration practices were largely confined to the context of clinical trials, and regulatory agencies and quasi-regulatory actors played an important role in promulgating the practice. A widely acknowledged inflection point in the history of preregistration practices is a 2005 decision by the International Committee of Medical Journal Editors (ICMJE) to require preregistration in a public database as a condition of publication in member journals.Footnote ⁵¹ Since the ICMJE’s membership included very high-profile journals such as the New England Journal of Medicine and the Journal of the American Medical Association, this policy had an immediate effect: the largest registry of public trials at the time, clinicaltrials.gov, had registered a total of 13,153 clinical trials from its inception in 2000 until May 2005; in the five months following the ICMJE’s announcement this total jumped 73 per cent to a total of 22,714.Footnote ⁵² The ICMJE acted because of what they perceived as a failure of regulation: the lack of a system for tracking clinical trials meant that pharmaceutical companies could simply choose not to publish the results of trials that did not cast their products in a favourable light. The Lancet lambasted the FDA for overseeing a process that was ‘so easily manipulated by those with potentially massive financial gains’.Footnote ⁵³ The ICMJE’s quasi-regulatory mandate was superseded in the United States only a few years later by the FDA Amendments Act of 2007, which made trial registration compulsory for a more expansive set of clinical trial types and required the disclosure of trial results. The existence of the clinicaltrials.gov infrastructure itself is also a product of regulation: the 1997 Food and Drug Administration Modernization Act of 1997 mandated the creation of a publicly accessible database of clinical trials that were linked to an investigational new drug application to the FDA.

This is not to say that the regulatory ethos cannot spread in the absence of regulation, and psychology is a good case in point. The American social psychologist Brian Nosek has been at the forefront of what he and colleagues have called the ‘preregistration revolution’.Footnote ⁵⁴

In an early position paper, they pointed to preregistration in drug development as a practice to emulate, and argued that the model by which most science operates would be considered ‘laughably quaint for ensuring responsibility and accountability in state or corporate governance’.Footnote ⁵⁵ Under Nosek’s leadership, the Center for Open Science has pursued a strategy for change that is focused on building infrastructure and community to ‘make it easy’ and ‘make it normative’ for researchers to engage in preregistration and other behaviours they view as desirable.Footnote ⁵⁶ Penders describes this strategy as one that is based in appeals to the moral character of the scientist, as evidenced by public displays of emotions such as shame and embarrassment about having failed to use such practices in the past.Footnote ⁵⁷ And, arguably, this strategy has worked. While preregistration was ‘almost unheard of’ in psychology in 2012, a recent study of psychology articles estimated that 14 per cent of articles published in prominent journals now contain preregistrations.Footnote ⁵⁸

As we have argued above, one advantage of identifying the desired ethos of replication reformers as regulatory is that it allows us to leverage the existing scholarship on regulatory science to better understand how the practices associated with the regulatory ethos may function differently when placed in new institutional contexts. For example, the history of regulatory science suggests that change can be fragile in the absence of legal mandates. Creager’s study of the International Committee for Protection against Environmental Mutagens and Carcinogens (ICPEMC) in this issue is a case in point. The scientists who participated in the committee recognized the precarity of their work, fearing that the extensive (and expensive) data that they produced might be lost in time. As Creager puts it, they ‘recognized that certifying testing regimes was not sufficient for regulation’. This self-organized commission of scientists did not have the authority to negotiate legal change – or the responsibility to implement policy.Footnote ⁵⁹ And in the end, after more than a decade of work developing and validating screening tests for identifying cancer-causing chemicals, the committee folded in the 1990s due to lack of funding.

Reformers are not unaware of these risks. While Nosek and colleagues have long held that policy interventions to make behaviours ‘required’ should follow from earlier transformational work rather than acting as the catalyst for transformation, they have also long cautioned against reforms that rely solely on ideals. In their paper on the preregistration revolution, they noted that most areas of science have relatively weak incentives for adopting new practices, in contrast to the legal requirements for clinical trial registration, and called on funders and journals to exercise the same kind of quasi-regulatory power as the ICMJE to encourage the adoption of preregistration.Footnote ⁶⁰

Nosek’s own experiences with preregistration reveal the limits of transplanting the regulatory ethos outside its infrastructural ecosystem. In September 2024, Nature Human Behaviour retracted a paper because, although the authors claimed that the analyses in the paper were preregistered, a closer inspection of the preregistration itself revealed that the study had originally been designed to test a different hypothesis.Footnote ⁶¹ Ironically, the unregistered titular claim was that practices such as preregistration could increase the replicability of the social-behavioural sciences, and the paper’s authors included Brian Nosek and Charles Ebersole, both of whom were also authors on the aforementioned paper on the preregistration revolution. In a commentary about the retraction, the authors argued that the statement that all analyses had been preregistered was indeed erroneous, and that the article should be corrected to reflect that only some of the analyses had been preregistered.Footnote ⁶² But others saw the situation as much more serious because the unregistered analyses were key to the central claim of the paper. ‘It is one thing to forget to preregister’, one commentator wrote, ‘but it is entirely different to simply switch outcomes and pretend you were studying that thing all along’.Footnote ⁶³ The editors of Nature Human Behaviour agreed, citing ‘lack of preregistration for measures and analyses supporting the titular claim’ and ‘selection of outcome measures and analyses with knowledge of the data’ as reasons for retracting the article.Footnote ⁶⁴

It is not just the absence of legal enforcement which means that even the most ardent proponents of the regulatory ethos will sometimes fail to consistently embody it. Preregistration gains its power in large part because of the presence of regulators who are employed to analyse the data made available through this process and empowered to act on their findings – a labour force which is lacking outside the context of clinical trials. On its website, the FDA describes the 2007 Amendments Act which mandated trial registration as an act that would ‘ensure that FDA staff have the additional resources needed to conduct the complex and comprehensive reviews necessary to new drugs and devices’.Footnote ⁶⁵ In contrast, much of the spadework for the investigation of the Nature Human Behavior article was conducted by academics acting in a personal capacity. Joe Bak-Coleman, one of the scholars who was most deeply involved in the investigation, wrote a blog post titled ‘I did not want to write this blog post’, where he detailed the difficulties of doing this work outside his regular employment:

As scholars studying equity in open sciences practices have noted, the availability of data does not translate directly into benefit: potential users still need skills, time and other resources to extract value from the data, and some are better equipped with these resources than others.Footnote ⁶⁷ Likewise, as scholars studying reproducibility reforms have noted, the availability of a preregistered protocol has a very different value when the review process is managed by a largely voluntary workforce of academics than it does in the presence of a dedicated workforce of civil servants.Footnote ⁶⁸ This incomplete migration of the infrastructure that allows the regulatory ethos to work can ironically stabilize the enactment of this ethos in individual researchers, as they feel the need to personally make up for the lacking resources and engage themselves in auditing practices.

Some reformers have attempted to address the infrastructural gap by building their own version of a dedicated workforce. In her current role as editor of the prominent psychology journal Psychological Science, Simine Vazire has created a new category of editors – statistics, transparency, and rigour (STAR) editors – whose work takes place alongside and after that of traditional academic editors who manage peer review and make decisions on manuscripts.Footnote ⁶⁹ The STAR editors do the laborious work of checking the manuscript against its preregistration, checking that the analysis is computationally reproducible, and so forth. This is, however, an institutionally fragile model because of the extent to which it relies on volunteer labour. Reflecting on these and other limitations of preregistration, one recent opinion piece from political scientist Rose McDermott concluded that ‘preregistration represents a form of virtue signalling that is more performative than actual’ since overburdened editors and reviewers are unlikely to monitor compliance with a preregistered plan in most cases.Footnote ⁷⁰

Finally, McDermott’s characterization of reformers as engaging in ‘virtue signalling’ and ‘crush[ing] creativity’ demonstrates how common critiques of regulation take on a different character when the regulatory ethos moves into other areas of science.Footnote ⁷¹ Regulatory agencies have long been subject to critiques that they are slow, bureaucratic and overly cautious in their approaches. In the United States, for example, the FDA faced increasing pressure in the 1990s and 2000s from critics who asserted that the agency’s requirements were ‘stifling innovation’.Footnote ⁷² The Critical Path Initiative, introduced in 2004, aimed to ‘modernize’ the FDA’s processes by incorporating omic and imaging biomarkers into clinical trial designs.Footnote ⁷³ Somewhat ironically, while critiques of the scientific status quo have pushed academic science more towards the regulatory ethos, critiques of FDA bureaucracy have pushed this key regulatory agency away from it. Bayesian adaptive clinical trials for cancer therapeutics in the 2010s and accelerated approval of Alzheimer’s drugs in the 2020s are both examples of how the agency attempted to incorporate flexibility into its processes that would allow for the direction of a clinical trial or the evidence base for a new drug to evolve over time rather than being strictly specified in advance.Footnote ⁷⁴

More important for our argument, however, is how these critiques make visible existing scientific identities, values and practices that conflict with the regulatory ethos – that is, they make visible other forms of scientific life. When NIH deputy director Larry Tabak first presented the NIH’s plans for reproducibility reforms to the Advisory Committee to the Director in December 2013, he paused to joke about the ‘number of frowns he saw around the table’, and that he was well aware that many people saw some of the proposed interventions as ‘constraining innovation’.Footnote ⁷⁵ Members of the Advisory Committee reflected back his concerns. One academic biologist made a ‘plea’ to the NIH to ensure that there is good evidence for the efficacy of new requirements before imposing them, because the scientific community is already ‘drowning in endless bureaucracy’. Others voiced overlapping concerns that they ‘did not want to take away what is unique about academia’ and that we need to recognize the important role of intuition and not just ‘throw the old away.’ They pointed to important historical discoveries in biology built on close observation rather than pre-specified plans to underscore the importance of other styles of work.

The only member of the committee to speak positively about the proposed interventions was a senior executive at GlaxoSmithKline. He saw the differences in reactions – and the differences in academic and industry research – as a ‘cultural divide’. Over the fifteen- to twenty-year life cycle of developing a drug, he explained, many scientists will be involved in the process, and so practices aimed at providing continuity within a company were important. These practices might look bureaucratic or inflexible to outsiders, but, he argued, they were a fundamental part of the culture of industry. Implicit in his comments was the suggestion that this industry culture was also the more desirable culture. He spoke with frustration about how academics faced ‘no consequences’ when they failed to provide enough information for others to replicate their work, and if that work eventually proved to be irreproducible academics could ‘even write a paper about why the previous paper was wrong’.

As we have shown in this section, the practices being promoted by reproducibility reformers bear strong similarity to the practices used by regulators to shape knowledge produced for regulatory purposes. But this exchange in the NIH Director’s Advisory Committee meeting demonstrates that it is not simply a set of practices that is being promoted; it is an ethos that encompasses both practices and a set of beliefs about why those practices represent a right, ethical and proper way to be a scientist. In a proposed list of paperwork requirements, the pharmaceutical industry executive saw a familiar ‘culture’, while the academic scientists saw not just the pragmatic problems of increasing bureaucracy but a deeper mismatch between the spirit of what they saw represented in the reforms and what they saw as unique about their own ways of being a scientist.