In the late nineteenth century, agricultural chemistry was in chaos. According to Harvey W. Wiley – head chemist of the US Department of Agriculture (USDA) who later became known as the ‘Father of the Pure Food and Drugs Act’ after its 1906 passage – chemists ‘act[ed] in complete independence of each other in regard to methods of investigation’. Some followed the methods of German chemists, others French and yet others English. As Wiley recalled in 1899,
There was no unity of interest … nor any common system of work. The condition of analytical work may be truly described as chaotic … There was no standard of comparison or reference. Buyers and sellers were continually wrangling over analysis, which, made by different men following different methods, did not agree.Footnote 1
Methodological discord in agricultural chemistry was not an obscure scientific problem. It threatened the stability of agricultural markets, the profits of fertilizer manufacturers, the livelihoods of farmers, the expertise of analytical chemists and the regulatory authority of state officials. The conflict over fertilizer analysis exemplified a broader struggle to transform practices of science into reliable instruments of governance. The ensuing quest for methodological uniformity was at once an epistemic and a political project: it sought to stabilize not only results but also the authority to regulate commerce through science.
In 1884, chemists employed by state and federal government offices formed the Association of Official Agricultural Chemists (AOAC) to develop authoritative methods to evaluate manufacturers’ claims about fertilizer, and later other agricultural and food products. Through multi-laboratory collaborative studies, the AOAC developed, tested and approved standard analytical methods to enable consistent enforcement across states. In the process, they enshrined agricultural chemistry as the only reliable source of knowledge for commercial fertilizer regulation.
New anti-fraud legislation and scientific, business and legal demands for reliable chemical facts coalesced in the establishment of the AOAC as a source of regulatory knowledge. Regulators and industry trusted this knowledge because it was produced by standard methods that had been tested in multiple laboratories and judged to produce uniform results – a process that would later be called ‘validation’. In subsequent decades, AOAC methods became the foundation of US food and drug law. Validation here functioned as a mechanism of consensus building, transforming agreement among experts into a surrogate for legal authority.
Nearly a century later, officials confronted similar challenges with toxicity testing methods, then seen as insufficiently reliable for regulatory use. Rising to the challenge, AOAC scientists hoped that their practices for generating authoritative analytical facts about agricultural products could be applied similarly to generate authoritative facts about chemical risk. However, the AOAC failed to translate its monopoly over validated methods in analytical chemistry to toxicology. Where the AOAC failed, the Organisation for Economic Co-operation and Development (OECD) succeeded, bringing together scientists and regulators from across the industrialized world (including the US, Canada, much of Europe, Japan, Australia and New Zealand) to establish standard methods for evaluating chemical risk. There, epistemic qualms about standardization were overridden by social and political pressures to standardize practices to facilitate regulation and trade.
Upon its formation in the 1880s, the AOAC joined the ranks of hundreds of other professional societies newly established in this period, which exerted control by authorizing knowledge, issuing licenses and establishing professional standards.Footnote 2 The AOAC’s standardization efforts were hardly unusual in themselves, particularly at a time when new forms of codified measurement were replacing previously dominant norms of personal acquaintance and experience.Footnote 3 But their work contrasts with standard-setting efforts that have most often been voluntary, non-governmental efforts.Footnote 4 The AOAC’s work also reflected a larger shift in the late nineteenth century, as a moment when scientific expertise became more central to regulation. Before this period, regulation was ‘artisanal’ – based on experience and rules of thumb, not scientific training.Footnote 5 The shift was solidified as regulation enacted by expert-staffed public agencies became commonplace in the late nineteenth century.Footnote 6
The AOAC positioned itself as the supervisor of the agricultural trade not just state by state, but nationwide. It was an ambitious goal at a time when there were few precedents for national regulation of interstate commercial enterprises.Footnote 7 As one early AOAC president envisioned, their work would ‘lay a foundation so solid that every court in this land must respect its conclusions, and every analytical chemist … must be forced either to practice or admit the advantages and correctness of our system of analyses’.Footnote 8 Despite lacking legal authority, the AOAC successfully achieved this vision.
The AOAC – and later the OECD – story exemplifies business’s and government’s shared interest in creating the rules by which markets operate and how science has been made useful to governance. Business and government have a mutual interest in stabilizing markets and have long enrolled scientific expertise in these efforts.Footnote 9 Scientists have used their expertise to inventory natural resources and render goods calculable. For example, varietal names create market categories, metrological practices calculate goods’ weights and volumes, and valuation practices transform measurements into prices. As such, scientific expertise, including chemistry, has been a useful resource for the state and corporations to rationalize markets.Footnote 10
Where regulation is concerned, not all science is created equal. Since at least the late nineteenth century, governments and industries have favored a form of scientific practice that values predictability and uniformity. In Lorraine Daston’s and Peter Galison’s terms, it is a science prizing mechanical objectivity over trained judgement.Footnote 11 Scholarship on agencies such as the Food and Drug Administration (FDA) and the Environmental Protection Agency (EPA) has described this orientation as ‘regulatory science’, distinguishing it from the more open-ended inquiries of ‘research science’.Footnote 12 STS scholars have shown how knowledge practices organized around public decision making under uncertainty depend on standardization to preserve authority and stabilize expertise.Footnote 13 Building on this work, Alberto Cambrosio and colleagues identify a key feature of this approach: under regulatory conditions, compatibility across laboratories often takes precedence over truth value.Footnote 14 In this volume, Nicole Nelson and Lara Keuck have introduced the concept of the ‘regulatory ethos’ to capture how these practices and values have spread beyond formal oversight into non-regulated domains.
The AOAC offers an early instantiation of this ethos in action. Its work illustrates how validation practices, as part of the longer history of regulatory science, became central to producing reliable, administratively actionable knowledge.Footnote 15 Building the regulatory state required building measurement regimes: metrology furnished a shared technical language through which states could govern trade, health and safety. In the nineteenth century, scientists and bureaucrats joined scientific methods – e.g. bacteriological tests, chemical assays – to expanding governmental mandates, enabling laboratories, bureaus and inspectors to translate measurement into law. Standardization and quantification underwrote this legitimacy.Footnote 16 Later systems of public health and environmental governance inherited this infrastructure even as risk logics evolved.Footnote 17 This paper argues that, beyond these broad continuities, validation through inter-laboratory testing remained a crucial means of producing administratively actionable facts amid contestation and uncertainty. Yet the contrast between the AOAC and the OECD highlights the shifting politics of validation: where the AOAC’s validation practices safeguarded public authority against industrial influence, the OECD’s later framework legitimized industry dominance under a transnational regime of regulatory science.
Bringing uniformity to chaos
As American farmers began to use chemical fertilizers in the early nineteenth century, they faced a challenge: they had no expedient way to know what was in the fertilizer. Farmers found that the time from fertilizing to harvesting created openings for fraud, making it difficult to trust fertilizer manufacturers’ claims. This created a market for analytical chemistry in agriculture, with American farmers relying on the chemical valuation of fertilizer from the 1830s.Footnote 18 By the late 1870s, now routine chemical work performed at agricultural experiment stations had created a direct link between fertilizer’s chemical content and price.Footnote 19 This coincided with a push by a generation of agricultural chemists to increase agricultural productivity and contribute to economic progress. However, reliance on chemical accounting practices created additional openings for fraud, as manufacturers used the direct link between alleged chemical content and price to adulterate products and cheat farmers.Footnote 20 Farmers in the eastern US were particularly eager to accept chemists’ analytical help with artificial fertilizer, which promised to help them compete with more fertile farmland farther west.Footnote 21
State anti-fraud legislation solidified the role of chemical analysis through the establishment of new inspection, testing and labelling systems. These laws empowered state chemists – chemists appointed by state governments, who usually also worked at agricultural experiment stations – to determine nutrient values in fertilizer through laboratory analysis, to record findings on product tags and thereby to inform farmers of product quality.Footnote 22 Fertilizer became one of the first products to be labelled with certified analyses.Footnote 23
How these analyses should be performed, however, was disputed. Different chemists used different methods, which produced different results. Which were correct? Or, more to the point, which should be the basis for market valuation and regulation?
The problem was that chemists disagreed about what they were measuring – or should be measuring. Fertilizer contained three key nutrients: nitrogen, phosphorous and potassium. Manufacturing chemists generally favoured methods yielding the highest possible values; more nutrients commanded higher prices. State chemists typically reported only constituents from which a plant could derive nutritional benefit – only a percentage of total nutrients. Different chemical forms (e.g. water-soluble or reverted) provided varying nutritional value. But state chemists disagreed about how much of the partially useful forms to count. While analyses varied, state chemists generally derived lower nutrient values, resulting in lower product valuations.Footnote 24
Connecticut state chemist Samuel Johnson – also a professor at Yale and later president of the American Chemical Society – campaigned against fertilizer fraud by using his analyses to challenge commercial claims of product quality and price. He made no small charges. For example, in 1850s tests of six superphosphate fertilizers that were all priced at forty-five dollars per ton, Johnson and his students calculated that they were worth between $3.80 and $25.22 – the lowest because it contained no soluble phosphate at all.Footnote 25 Johnson and other state chemists publicized their findings in fertilizer valuation tables, which translated nutrient percentages into local dollar values. The chemists’ stated aim was to help farmers get the best value for their money. Yet Johnson grew so bold as to lead an effort to create a single valuation table for most of the Northeast – in doing so, effectively seizing from manufacturers the power to set prices.Footnote 26
Methodological discord destabilized the fertilizer market and left regulatory efforts on unsteady ground. When disputes surrounding the accuracy of nutrient values faced litigation, judges struggled to adjudicate which laboratory was correct. Whether employed by the buyer, seller or state, all chemists – often indistinguishable by training – defended their methods as scientifically valid. Moreover, commercial chemists emphasized, state chemists could not claim a disinterested higher ground that produced more reliable results: their own measurements sometimes varied by a factor of two or more.Footnote 27 Per one prominent chemist, the ‘fallibility of methods’ was exposed as soon as they were tested by multiple analysts, which undermined chemists’ justification for interceding on the public’s behalf in the fertilizer trade.Footnote 28
Fortunately, the solution was simple – or so many chemists thought. The ‘great and regrettable divergency’ in chemists’ results left the impression that analytical chemistry, said one chemist, was ‘not a reliable or exact science, and … cannot produce in practice what it expresses by equation’. To remedy the situation, chemists simply needed to adopt standard methods and convince manufacturers to operate on that basis. If they did so, ‘all divergency of results should disappear’.Footnote 29 Uniform methods would beget uniform results, which would help to restore trust in chemists’ technical acumen and to stabilize the fertilizer trade.Footnote 30
Over the course of three assemblies between 1880 and 1881, government and manufacturing chemists attempted to agree on a new uniform system for measuring nitrogen, phosphorous and potassium in fertilizer. The first attempt failed. This was largely due to conflict over the phosphorous method, which produced results ‘absurdly at variance’ – even when performed by chemists in the same lab using the same sample, and so the methodological stalemate continued.Footnote 31 Manufacturers continued employing methods that yielded higher results, state chemists lower ones. The National Fertilizer Association (NFA) – newly formed in 1883 – campaigned against what they portrayed as state chemists’ capricious and inaccurate methods.Footnote 32
In 1884, government chemists again convened and began executing a series of politically savvy maneuvers. To consolidate their standing, they founded the Association of Official Agricultural Chemists (AOAC), aspiring ‘to secure … uniformity and accuracy in the methods and results of fertilizer analysis’.Footnote 33 With the AOAC’s formation, chemists transformed the nitrogen, phosphorus and potassium committees into year-round operations. Members would survey literature to assess available analytical methods, test new techniques and have peers confirm results, which would be presented at annual meetings. In what historian Alan Marcus described as ‘a political masterstroke’, the AOAC adopted a phosphate measurement method developed by a chemist who worked closely with fertilizer manufacturers in place of their prior embattled method.Footnote 34
The AOAC not only excluded manufacturing chemists by design; its formation also constituted a direct assault on the NFA, whose main founding aim had been to abolish the whole system of state chemists and all fertilizer legislation.Footnote 35 Nonetheless, at the AOAC’s second annual meeting in 1885, an NFA spokesman congratulated the ‘official’ chemists, attesting to their ‘benevolent influence’.Footnote 36 The NFA rationalized that the AOAC’s efforts had brought about analytical results more in accordance with their own. While some differences would always exist, they argued that the AOAC’s methods approached ‘close correspondence within the unavoidable limits of variation’, and hence manufacturers no longer had reason to complain about their analyses.Footnote 37
Manufacturing chemists had effectively ceded control over the scientific standards that would determine the value of their products. Why? The AOAC’s decision to supersede the inconsistent phosphorous method with an industry-endorsed technique certainly helped, but it was ultimately a calculated move to shape regulation in industry’s favour. While the NFA had campaigned against legislative efforts, by 1884 they concluded that state laws were ‘incontestable’.Footnote 38 The association grew increasingly supportive of uniform fertilizer legislation – ceding not just the benefit but the necessity of ‘[u]niform registration of brands, uniform phraseology in expressing the guaranteed analysis, uniform procedure in case analysis difficulties arise, [and] uniform provisions for sampling’.Footnote 39 They recognized that the AOAC’s codification of standard methods – and their use by official chemists across all states – would serve a beneficial market coordination function. Manufacturers would have preferred that states not issue rules at all. But if there had to be rules, better that they be uniform rules, nationwide. The AOAC could help them achieve that.
Manufacturers also hoped to achieve more favorable outcomes acting as partners instead of opponents. They quickly did just that, leveraging their cooperation to persuade the AOAC to stop issuing fertilizer valuation tables.Footnote 40 Government and manufacturing chemists’ interests were ultimately congruent: the AOAC legitimized official chemists’ expertise and authority while simultaneously producing definitive results for product labels, facilitating commerce.
In 1886, Science reported that the AOAC’s work ‘brought about greater harmony’ between manufacturing and government chemists – results ‘so satisfactory’ that the AOAC expanded their efforts to other agricultural products, including soils, cattle feed and dairy products.Footnote 41 The NFA and federal government spokesmen had both urged this expansion. The AOAC’s standard methods were taken up as part of the battle for pure foods. Delivering products free of adulteration required authoritative methods; in even ‘simple’ matters like milk adulteration, officials would struggle to secure a conviction in court without accepted standards to measure adulteration.Footnote 42 The AOAC worked to establish itself as the authority in all matters of agricultural analytical methods.
An instructive example comes from dairy science. In 1890, Stephen M. Babcock of the University of Wisconsin’s experimental station published a technique to measure the butterfat content of milk. The AOAC adopted the test as an official method in 1909. By providing a rational means of determining how much to pay producers, the test helped to transform the American dairy industry.Footnote 43 Previously, milk quality had been judged by weight, which created an incentive for producers to dilute milk or skim its cream. This business tactic became unprofitable once milk was evaluated by fat content, leading one Wisconsin governor purportedly to quip that the test ‘made more dairymen honest than … the Bible because of the summary verdict which it rendered’.Footnote 44
The AOAC developed standard methods for regulatory purposes that came to be treated as legally binding absent any law stipulating their use; as long as they presented a unified front, explicit legal authority was simply not needed. As Marcus argued, AOAC efforts hinged on the fact ‘that each association member possessed state power, which he was willing to wield widely, and that the success of each depended upon his colleagues’ unity and consistency’.Footnote 45 By creating a national standardizing device, state chemists simultaneously protected themselves and provided de facto legal authority.
The AOAC’s de facto regulatory role was solidified in the coming decades and their methods became the foundation of US food and drug law. At the turn of the twentieth century, AOAC methods allowed government officials to verify consumers’, farmers’ and journalists’ claims of food adulteration. AOAC chemists helped to draft the legislation that would become the Pure Food and Drug Act of 1906, which in turn reinforced their methods.Footnote 46 In subsequent decades, AOAC methods were central to the USDA’s prosecution of food adulteration and misbranding. By the 1930s, the regulatory authority of this professional association of government scientists had become so solidified that the FDA commissioner proclaimed in 1936 that AOAC methods were ‘the Law and the Prophets as far as Food and Drug Act enforcement is concerned’.Footnote 47 As of the 1980s, no court had ever failed to enforce the use of AOAC methods.Footnote 48
Collaborative methods testing
Today, analytical chemistry is widely trusted to produce facts for regulation. But that took work to achieve: it did not produce definitive facts until people found a social means to agree on what would count as definitive. The core of the AOAC’s approach was based in testing methods across multiple laboratories, a process typically referred to as collaborative methods testing, or later inter-laboratory testing. By the 1960s, the process had come to be known as ‘validation’.Footnote 49
When the AOAC was formed, AOAC chemists recognized that they could never achieve absolute uniformity; instead they created a consensual process for agreeing when methods produced results that were uniform enough. They modelled the process on the evidentiary practices that had been persuasive in preceding years’ methodological disputes between official and manufacturing chemists. Contests had been won and lost based on multi-laboratory demonstrations, where multiple accomplished chemists employed the same materials and methods. For example, after state chemists selected the original phosphorous method in 1880 (mentioned above), manufacturing chemists orchestrated one of the first formal inter laboratory trials in the US.Footnote 50 Their results convincingly demonstrated the method’s inadequacy. The same mechanism of inter-laboratory trials would now be adopted by the AOAC.Footnote 51
The process worked as follows.Footnote 52 Scientists designated as referees oversaw the review process. After reviewing available literature and protocols, a referee either selected an existing method or developed a new one and subjected it to intra-laboratory testing to the point where they judged it to satisfy four criteria: accuracy, precision, reliability and appropriateness for routine use. Ideally this would be done by one or two ‘guinea pig’ chemists in the referee’s lab who were unfamiliar with the method.Footnote 53 The referee then codified the protocol and enrolled fellow AOAC scientists as ‘collaborators’ to run the test in different laboratories using the same materials and methods.Footnote 54 The referee analysed the results and – if they judged them sufficient – submitted them to a review committee that would recommend the protocol for acceptance, rejection or further study. If it was endorsed, the referees would attempt to convince their peers at the annual meeting, where it would be put to a vote.Footnote 55 No official criteria specified when collaboratively tested methods should be deemed good enough. In the end it was a social decision: the method was good enough when the experts said it was good enough.
Until at least the 1960s, there was no statement of policy or official guide; records from the AOAC’s the early years only specified when implicit standards were unmet.Footnote 56 Nonetheless, regularities emerged early on and were reiterated over the organization’s first hundred years. Much changed in this time – including the rapid expansion of product and test types as well as methodological shifts (e.g. volumetric to colorimetric analysis, automation), but the vision of what made a good AOAC method remained consistent. Reading across decades’ of AOAC proceedings and publications, I found that the AOAC used several epistemic values for assessing potential official methods: uniformity, workability, representativeness and conservatism.
Above all, the AOAC prized uniformity. Consistent, uniform standards brought about predictable and fair markets. That meant that all chemists had to use the same methods. In 1885, the US commissioner of agriculture inveighed against the goal of devising ‘the absolutely best method’ (i.e. the most accurate or most precise). It was ‘better that a less perfect method be uniformly employed than … the best method … used in one locality and the poorest in another’.Footnote 57 As decreed by the AOAC constitution, the organization was committed to both ‘uniformity and accuracy’ of methods, but uniformity outweighed absolute accuracy or precision.Footnote 58 One reason was legal: regulated parties, as the opposition in court, needed to have the opportunity to employ the methods; if their results differed, regulators’ enforcement efforts would fail.Footnote 59
The chemists aimed to eliminate individual judgement through formal rules, which would limit the discretion of the analysing chemist. The emphasis on uniformity also helped to cultivate a sense of ‘placelessness’, which, as Robert Kohler has argued, helps to accord laboratories epistemic authority.Footnote 60 Collaborative methods testing fostered the belief that results derived by AOAC methods would have been the same had they been produced in any other lab – government or manufacturing – following the same protocol. Therefore, one result could stand in for many and be accepted as definitive.
A universal method also had to be simple and fast enough for routine use. The thinking was that ‘chemists would make a mistake if in their zeal for absolute accuracy they made the methods so complicated as to run the risk of great variation in the hands of the less skilled’.Footnote 61 For a method to be ‘valid and workable’ (as described in later decades) it should employ commonly used laboratory tools, not highly specialized or complicated apparatus.Footnote 62 If the AOAC adopted too complicated a method, labs would find workarounds, making AOAC methods obsolete and undermining uniformity.Footnote 63
The testing laboratories defined the range of method applicability. The AOAC could only make inferences about methods if the collaborative tests were representative of laboratories, test conditions and substances for which a method was intended. For example, if designed for one material, a test could just include that one material; wider use required that all intended materials be tested.Footnote 64
The AOAC anticipated the need to continually update its methods in step with evolving experimental practices and instrumentation, but called for caution. As the AOAC president stated in 1891, they aimed to maintain a ‘rigorous system of testing’ and recommended ‘holding fast to all that is good either in old methods or in new ones, being thus always both duly conservative and vigorously progressive’.Footnote 65 Conservatism was crucial: they wanted to avoid premature method adoption. That is, they wanted to avoid another phosphorous method debacle.Footnote 66
By the mid-twentieth century, the AOAC faced calls to further specify the parameters for inter-laboratory studies, yet still had not defined criteria for method adoption.Footnote 67 The AOAC purportedly adopted methods only when they were shown to produce uniform results. However, no two labs’ results would ever be truly identical – some variation was inevitable. How much variation could be tolerated for a method to be said to produce uniformity? They never specified a threshold. A 1961 analysis found coefficients of variation as large as 10 per cent.Footnote 68 William Youden, a Bureau of Standards statistician and AOAC member who in the 1960s created a statistical manual for the AOAC, thought the lack of specification was ‘just as well’ because it afforded members flexibility to deal with the great diversity of agricultural materials and associated analytical methods.Footnote 69
As FDA chemist and long-time AOAC leader William Horwitz argued in 1964, ‘Specific criteria for acceptability of analytical methods cannot be imposed’. The decision was an ‘administrative scientific judgment’, similar to ‘judgments … made in determining whether analytical results support a legal action or a scientific hypothesis’.Footnote 70 Another FDA scientist noted that ‘the accuracy, the reproducibility, the specificity, and the sensitivity of a method must be directly related to the need in the interest of safety’ of a given substance.Footnote 71 A toxic pesticide or carcinogenic food additive would require considerably more sensitive and reliable methods than a more innocuous substance. That is, there could be no explicit criteria because what was good enough for one method might not be good enough for others. Despite the AOAC’s commitment to the discipline of formal rules in laboratories, the organization itself insisted it could not abide by the same. Expert judgement was at some level inescapable.
Despite these limitations, the AOAC’s project succeeded. Official AOAC methods underwent collaborative testing and were judged to produce sufficiently uniform results; the process authorized all subsequent results derived by the methods to be considered definitive. From the 1880s onwards, government and industry found it useful to agree that these official methods produced facts about the content of agricultural and food products: doing so facilitated both regulation and commerce.
‘Toxicologists need help’
Once an analytic method garnered the AOAC’s ‘official-method’ stamp of approval, regulators and the regulated each could use it with confidence that the results would be accepted by the other and stand up in court. Designation as an official method achieved closure, rendering its results authoritative as facts. The AOAC first developed its process to create consensus around methods used to generate facts about the contents of fertilizer; it quickly expanded to generate facts running the gamut of agricultural products. How much fat was in milk? Nicotine in tobacco? Aflatoxin in peanuts? Nitrites in meat? PCBs in fish? AOAC methods were – and are – the authority on all this and much, much more.
In the post-war era, regulatory agencies asked whether there could be equivalent methods for toxicology. Regulators sought standard methods for generating authoritative facts about the amount of risk posed by chemical products. In 1971, the AOAC was asked, presumably by the FDA, to contribute to this effort by establishing toxicological procedures ‘on a firmer basis’ by improving their ‘reproducibility and reliability’.Footnote 72 Chemists believed that they could do that. However, the toxicologists involved resisted, on the ground that biological systems were far more variable than chemicals. If they were right – that biological systems were so variable as to resist analysis by standardized methods – it would be a critical roadblock, because standardization was a necessary precursor to validation.
The problem of standard methods had at least two key components. First, methods would be chosen as exemplars (‘the standard-bearers’).Footnote 73 Second, once so designated, methods would become ‘the things that everyone uses’.Footnote 74 In general, a validated method is one that people accept as appropriate for use. How that appropriateness is demonstrated varies, but in the cases discussed here, it involved inter-laboratory testing. Demonstrating uniform results in inter-laboratory tests authorized the method to be used over alternatives, providing the warrant for widespread use as the reference method. Beginning the process necessitated choosing a methodological exemplar as the standard. For analytic methods, the desirability of exemplary methods in wide use was not disputed; the dispute generally concerned which particular method it should be, not whether there should be one. In 1970s toxicology, however, this was in question.
Researchers had begun standardizing guidelines for testing chemical toxicity in animals in the 1940s and 1950s, when the US FDA issued guidelines for evaluating the toxicity of chemicals in food, drugs and cosmetics.Footnote 75 By the 1970s, some standardized assays – such as National Cancer Institute rodent bioassays – had entered common use. A small number were even codified in regulatory standards, such as the FDA’s guideline for testing eye irritation, known as the Draize test.Footnote 76
Toxicologists, however, questioned the project of methodological standardization. Part of the problem involved incentive: toxicity test results did not play a market-coordinating function, so what benefit did it bring if everyone conducted tests in the same way? Some toxicologists argued that regulators should embrace methodological plurality. Because toxicology labs had ‘attained a degree of internal consistency’ using unique protocols, they thought standardization was unnecessary, if not undesirable.Footnote 77 Their arguments were based in part on inter-laboratory testing comparing standardized reference procedures such as those offered by the FDA against individualized procedures unique to specific laboratories. Some of these studies found that the results derived from reference versus non-reference methods did differ, but never to the extent that it would change the interpretation of the hazard. They concluded that ‘competent’ laboratories could arrive at ‘nearly the same’ results using varying procedures.Footnote 78 These views reflect the pragmatic orientation of science done for regulatory purposes. Though the results might not be as consistent as they might have ideally liked, they were consistent enough for the purpose at hand of differentiating between risky and non-risky substances.
A larger number of toxicologists, while not contesting the desirability of standardization, simply thought it impossible. The problem was the state of the discipline. It was widely believed that toxicology was ‘in a state of transition’ and that the push to standardize was premature.Footnote 79 They conceived of the problem of standardizing methods across laboratories much as sociologist Harry Collins has written about replication: absent ‘a well-worked-out set of crucial variables’, even apparently trivial changes in the experimental set-up may cause ‘invisible but significant changes in conditions’. In such cases, ‘scientists just do not know enough to be able to guarantee that an experiment which looks just the same as another is the same in essence’.Footnote 80 Toxicologists widely believed that more work was needed to ensure they had an adequate grasp of the crucial variables – each of which would need to be controlled in a standardized study.
Yet even if all crucial variables were worked out and a standardized protocol were developed, two sources of variability remained: the biological test system and the scientist. For example, as one USDA scientist complained, it was often assumed that animals in different labs or kept in the same lab under varying conditions were all similar. They were seen as ‘“inanimate objects” rather than delicate, biologic systems responding in diverse ways to alterations in internal and external environments’. Belief in the value of standardized procedures was premised on this and led government scientists to promulgate ‘fixed methods’.Footnote 81 But standardized protocols would inevitably fail to produce uniformity with living organisms that themselves were not standardized.Footnote 82
Scientists themselves also introduced variation. Toxicologists typically did not invoke this in terms of observer bias or the ‘personal equation’.Footnote 83 They most frequently attributed this to the influence of training and expert judgement – crucial aspects of tacit knowledge and embodied technical skills or ‘know-how’.Footnote 84 This would, of course, be true across scientific disciplines; however, toxicologists argued that scientific judgement played a more significant role in their work compared to analytical chemistry.Footnote 85 As an MIT pathologist put it, chemists ‘work[ed] very hard, often for long periods, using difficult methods and complicated instrumentation to develop answers’. When finished, they could ‘usually relax in the satisfaction’ that the results were accurate with ‘reasonable assurance’ that data from multiple labs would be comparable.Footnote 86
Toxicologists argued that biological variation made training and expert judgement more central. The MIT scientist argued that conducting the test was just the beginning: when they ‘ha[ve] the answer in hand, [their] major job has just begun; the data must be interpreted, and the biological variations inherent in animal systems often make this a difficult task’.Footnote 87 Because of this, a toxicologist from Albany Medical College argued, scientists’ ‘training, skill, experience, and dedication have a decisive influence on the reliability and value of even the most completely standardized biological test’. The LD50 – an acute toxicity test to determine the dose of a substance lethal to 50 per cent of the animal population – was frequently cited as one of the simplest and most standardizable toxicity tests. Yet even it required ‘so much more in the way of accurate and detailed observation’ than any set of guidelines for a standardized technique could provide.Footnote 88 Ultimately, the interpretive flexibility of toxicological studies was greater than that of analytical studies. Analytical chemistry studies could be standardized and validated but toxicological ones could not; analytical expertise was a matter of rules, but toxicological expertise was a matter of judgement.Footnote 89
The AOAC thought that toxicologists had simply not tried hard enough. ‘Toxicologists need help’, observed FDA chemist and long-time AOAC leader William Horwitz.Footnote 90 He argued that it was inconsistent with scientific principles to expect standardization to lead to anything other than decreased variability. Uniformity of methods would surely beget uniformity of results. After all, it had for the AOAC.
Horwitz believed that the AOAC’s century of experience controlling ‘inevitable physical variability’ in analytical chemistry could similarly reduce the ‘inevitable biological variability’ in toxicology. Toxicologists simply had ‘not yet spent enough time discovering the sources of variability’. Once they identified and reduced that variability, inter-laboratory studies could be used to validate the methods – to demonstrate that the newly standardized methods produced more uniform results. Ultimately, ‘The work of toxicologists is much too important to be left to the toxicologists’.Footnote 91
The AOAC forged ahead against toxicologists’ warnings. In 1971, the AOAC appointed an FDA scientist as referee for the peer review and collaborative study processes for toxicological methods. They also partnered with the Society for Toxicology and the Environmental Mutagen Society to develop methods important to regulatory agencies.Footnote 92 Both societies had collaborative studies under way, which the AOAC would co-sponsor.
So far, Society of Toxicology studies had demonstrated that even the most common procedures, such as the LD50, yielded significant variability, which cast doubt on their reliability.Footnote 93 Here was a toxicity test that was supposed to be one of the most standardizable, yet standardization had not decreased LD50 variability.Footnote 94 Why? Incompletely standardized protocols due to unknown, crucial variables? The inevitable variability of biological systems? Scientists themselves? The answer, the AOAC posited, was incompletely standardized protocols. They argued that the sources of variation – the crucial variables to control with experimental design – had simply not yet been worked out.
The AOAC confidence was not borne out. Throughout the 1970s, the AOAC endeavoured to bring methodological uniformity to toxicology, only to fail resoundingly: the toxicologists appeared to be right. The AOAC attempted numerous collaborative studies to validate toxicological methods including for the LD50, aspiration, eye irritation, skin irritation and skin sensitization tests, as well as in vitro ones such as the Ames test.Footnote 95 Ultimately, the AOAC successfully conducted collaborative studies on only a handful of methods; they established official methods for even fewer.Footnote 96 For example, in 1973 the Draize eye irritation test was adopted as ‘official first action’ based on an AOAC collaborative study. However, this move was called into question by a duelling study that found ‘extreme’ inter-laboratory variability.Footnote 97 The Draize test never received the AOAC’s full endorsement. In 1975, perhaps to salvage their wavering efforts, the AOAC proposed a formal joint committee between it and the American Society for Testing Materials (ASTM), but the ASTM rebuffed the proposal, in part over concern about the way the AOAC had tried to ‘take over’ the toxicological work of one of its pesticide committees.Footnote 98
In time, the AOAC acknowledged that living systems presented ‘problem areas’ seldom seen in chemical assays. Many variables – particularly relating to live animals and their handling – eschewed full standardization or control.Footnote 99 The AOAC conceded that biological variability was perhaps more intractable than they had originally presumed. Still, they maintained that the collaborative study process readily applied, and that standardization was more feasible than toxicologists believed.
Ultimately, the AOAC monopoly on developing validated analytic methods did not translate to toxicological ones. By the 1980s, the AOAC reported, funding for validation studies was going to other (unspecified) organizations. Moreover, the AOAC recognized that the perceived reliability of toxicological data had been undermined more severely by the results of recent government investigations into toxicology laboratories, which led to the rejection of studies that had been used to demonstrate the safety of chemical products.Footnote 100 Instead of standardizing and validating methods, the AOAC shifted its focus to quality assurance.Footnote 101 (Previously, validation practices and quality assurance were assumed to be one and the same; now there were new demands to provide assurance that analysts were carrying out methods reliably.Footnote 102) The AOAC continued some minor efforts, inquiring with a variety of domestic and international industrial, professional and regulatory authorities – including the OECD – about their interest in collaborative studies on toxicological methods, but these efforts did not gain traction.Footnote 103
Conclusions
The AOAC’s failure to translate the validation practices it had developed for analytical chemistry might seem to prove toxicologists’ point: that standardized methods could not yield reliable results in living systems. Yet generating reliable knowledge from organisms was hardly a new problem – the history of experimentation in the life sciences is one of scientists developing practices to manage biological variation. Moreover, toxicologists in the same period collaborated with regulators elsewhere for the express aim of standardizing and validating test methods. The OECD’s Chemical Testing Programme, launched in 1977, brought together hundreds of scientists – largely from government and industry – from member countries to create a ‘how-to’ manual specifying how laboratories should conduct toxicity tests required by governments.Footnote 104 Method validation was – and continues to be – a key criterion of becoming an official OECD test guideline.
Why did the OECD succeed where the AOAC failed? Disciplinary competition played a role: toxicologists were willing to standardize and validate their methods; they just did not want chemists to do it for them. However, these debates – like those that led to the founding of the AOAC – did not happen in a vacuum. By the end of the 1970s, new chemical control laws and growing concern over inconsistent national testing requirements created administrative and economic incentives to harmonize methods internationally. The OECD became the key venue for this standardization. Although some members initially resisted, their concerns were ultimately overridden by ‘the administrative need to adopt standardized tests’.Footnote 105
Another part of the answer as to why the OECD succeeded lies in the organizations’ different epistemic standards. While the AOAC historically lacked explicit criteria for determining when inter-laboratory test results demonstrated sufficient uniformity to warrant approval, they always held firm in the requirement that an inter-laboratory study be conducted. The OECD aspired to similar validation practices, hoping to conduct inter-laboratory tests to provide members with assurance that a test performed internationally was the same as one conducted according to the same protocol domestically. However, the OECD felt it had too narrow a window of opportunity to harmonize methods before national statutory deadlines. Therefore, in place of validation by inter-laboratory methods testing, they accepted ‘validation through tradition and experience’.Footnote 106 This applied particularly to the initial fifty-one test guidelines adopted in 1981, which were based on established methods, generally widely accepted as valid means of measuring toxicity – that is, as the OECD also later put it, ‘classical [methods] for which experience had widely been gained’.Footnote 107 That existing experience with the methods provided authorization for their use absent formal inter-laboratory testing. Though dressed in the trappings of validation language, the process ultimately came down to expert judgement, to the chagrin of the AOAC.Footnote 108
The AOAC did for the Gilded Age agricultural trade in the US what the OECD later did for the Cold War chemical trade across industrialized countries: each established standard methods for evaluating chemical products to facilitate regulation and commerce. Validation was central to both, functioning as a consensual mechanism of agreeing when methods were good enough for regulatory use. AOAC methods were adopted not based on a priori epistemic standards but on members’ collective agreement that methods produced results that were sufficiently alike to be called uniform. The OECD’s test guidelines – at least in the programme’s early years – were adopted based on a looser standard. In both, consensus produced regulatory legitimacy by aligning the needs of science with those of markets and the state.
Today, validation remains a key criterion for OECD test guidelines, now under stricter requirements for inter-laboratory testing. Regulators and industry argue that validation ensures that test guideline studies – generally performed or funded by industry – are more reproducible and therefore more appropriate for regulatory decision making than the non-standardized studies more often conducted by academic toxicologists. The distinction matters: validated test guideline studies tend not to report adverse effects at current human exposure levels, whereas the more varied and evolving approaches typical of academic studies more often identify harm, implying the need for stronger regulation.Footnote 109 Yet test guideline studies are rarely replicated in practice. In one recent and unusual instance when a study was repeated, the results conflicted, calling into question whether these ostensibly reproducible studies – validated precisely to guarantee such reproducibility – are in fact so.Footnote 110 Nonetheless, regulators have largely continued to rely on them, even when substantial bodies of independent scientific evidence come to contrary conclusions.Footnote 111 Thus, where the AOAC’s validation practices defended public control against industrial influence, the OECD’s framework consolidated industry’s role by restricting regulatory legitimacy to ‘validated’ studies alone. This dynamic exemplifies how government and industry’s enrollment of science privileges stability, predictability and uniformity – even at the expense of validity or truth.
Taken together, the AOAC and OECD cases show how validation evolved from being a means of defending state authority and coordinating markets to being merely a tool of the latter. Validation linked epistemic authority to administrative power, enabling scientific procedures to serve as instruments of regulation. Yet the locus of that authority shifted – from the state’s claim to govern commerce in the nineteenth century to industry’s claim to govern evidence in the late twentieth.
By historicizing validation as a practice that both expressed and reorganized relations between expertise, commerce and governance, this paper has extended scholarship on regulatory science and the politics of expertise. It has shown that validation has been not a measure of truth but a technology for managing epistemic and political uncertainty – one that aligns scientific procedures with the administrative need for stability. In doing so, it has highlighted how the authority of science in regulation has depended less on the pursuit of truth than on the social management of agreement.
Acknowledgements
For their helpful comments and insightful critiques on earlier versions of this article, I am grateful to Hannah Conway, Angela Creager, David Jones, Lara Keuck and Naomi Oreskes; to participants of the Max Planck Institute for the History of Science Validation and Regulation in the Sciences of Health Workshop Series and Author’s Workshop, Harvard University’s Modern Sciences Working Group and Columbia University’s Center for the History and Ethics of Public Health; and to two anonymous referees for the journal. This research was supported by the National Science Foundation (award no. 1754980) and the Harvard Global Health Institute Burke Fellowship.
Competing interests
The author declares none.