The Background

Science depends on hypotheses, which may contain elements of truth. Hypotheses are tested using experiments that may fail to reject the absence of these truths, thereby strengthening the next hypothesis. In turn, if not rejected by the results of more experiments, the repetitive formulation of hypotheses allows more elements of truth to be assembled, like mosaics, until a picture of reality begins to emerge.

The beauty and importance of that reality depends on the ingenuity of the hypotheses that are framed by scientists and the validity of their experimental results. Even clever ideas may be wrong; a beautiful hypothesis slain by an ugly fact.Reference Huxley ¹ However, no matter. Just come up with another one. But even when a series of hypotheses are not rejected, the passage of time, which runs serially with this process, can shape its truth. The precision of experimental measurements usually improves with time, enhancing the clarity of that emerging picture — from abstraction, through impressionism to expressionism and ending in realism.

While this is the essence of the scientific approach to discovering truth, it is also well recognized that incorrect conclusions can be drawn, due to flaws in experimental methods or in the interpretation of the results they yield. A mistaken failure to reject a hypothesis may result from inappropriate controls, inadequate sample size, simple mistakes of interpretation or flawed methods of measurement. A fundamental strength of the scientific method is this recognition of the provisional nature of results until they have been replicated, ideally by many investigators, using different approaches, many times. Thus, the most important test in science is the test of time: the validation of one’s discoveries by others, gradually widening their acceptance as truth. This point is often missed in public discourse, most recently during the COVID-19 pandemic when scientists were criticized for inconstancy when they revised their public health guidance on masking, social distancing and disinfecting fomites as more evidence on viral transmission emerged.Reference Moffitt and McCloskey ² As Paul Samuelson summarized John Maynard Keynes’ thinking, “When the facts change, I change my mind. What do you do, sir?” In the public square, this feature of the scientific method is poorly understood, never mind the concurrent competition from “alternative facts.”

The Problem

While replication is fundamental to the integrity of a scientific discovery, a perception has emerged of a “replication crisis” in biomedical science. Attention was broadly drawn to this issue by publication of efforts by the private sector to recapitulate scientific results published in the literature, often from academic institutions.Reference Frye ³

Pharma and biotech have a strong motivation to try and replicate published work before investing in drug discovery and development based on such foundations. It is a logical and common deployment of their resources. Such internal studies are rarely subjected to the rigor of peer review and emerge in the literature. Besides commercial competitors, this puts the biomedical community at large and arguably decision-making within that company at a disadvantage. Thus, failure to replicate may also result from a failure of the replicator. In the COX-2 inhibitor saga, early replication by the sponsors of our studies showing depression of prostacyclin biosynthesis, which we suggested might be predictive of a potential cardiovascular hazard, were never published.Reference Grosser ⁴ Thus, acceptance of this reality was postponed while the sponsors argued publicly against the validity of our observations.

Academic investigators lack resources for replication studies. A recent survey of academic investigators reported a lack of funding, institutional priority and competitiveness for publication of replication studies compared to those addressing a novel hypothesis.Reference Cobey ⁵ In the rare situations when replication studies are performed by academic investigators and accepted for publication, this lack of priority can delay publication of such “negative results” — in this case by 18 months.Reference Skarke ⁶ Thus, a failure to replicate claims about so called “pro-resolving mediators” (SPMs) published in 2015 was essentially ignored and when the methodology underlying the original claims was shown to be flawed nine years later, the emergence of that paper took 18 months from the time of submission.Reference O’Donnell ⁷ These features reduce the incentive of investigators to embark on such studies. The delays and difficulties in publication may jeopardize their career advancement, despite their ultimate validation with the passage of time.

Publication bias exists in various forms. These include the bias towards “positive” results in studies funded by industryReference Lundh ⁸ and the bias of journals to avoid or delay papers that are negative. The most impactful journals are particularly on the lookout for novel, newsy discoveries.Reference Marder ⁹ While the incentives not to publish papers that question their own previously published studies are clear, we lack the data to determine whether this is also a problem.

Leaders of academic institutions are also not motivated to address the issue of scientific validity in a transparent way. The institution may have both fiscal and reputational investments in the experimental outcome.Reference Cappola and FitzGerald ¹⁰ Institutional hesitancy may also relate to the conflation of validity with fraud. The community at large is concerned in the first instance with validity — whether they can believe a set of published results — rather than, if they cannot, the reasons that lie beneath. However, such concerns with validity then raise questions about the rest of that author’s portfolio. Pursuit of such broader concerns of systematic misconduct are more time consuming and blend into consideration of outright fraud. We lack hard data on the prevalence of invalid data in the literature or of the frequency of fraud.

Just as journals may fear for their reputations being tarnished by evidence that invalidates their published papers, academic institutions are wary of the issue of fraud. Clearly, the responsibility to investigate this issue in a fair and rigorous manner and to apportion sanction rests with them. Those investigating such cases bear a responsibility to the accused, who is innocent until proven otherwise, to the whistleblower, and to their institution.

Such investigations usually involve a stepwise process — first by the relevant internal faculty committee, then by a broader committee that may include outside experts with domain expertise and finally, the decision by a separate body on how to proceed with respect to potential sanctions based on their findings.

Institutions vary as to their criteria for progression through this process, when to inform funders of the existence and progress of such an investigation (whether this is obligated varies between countries, although in the US, investigations of National Institute of Health-funded studies must be reported to the Office of Research Integrity (ORI)), whether to inform the whistleblower of the results of the scientific investigation and how to decide on sanctions. Both funders and journals tend to cede their decisions to the outcome of these institutional investigations despite their inconstancy and lack of transparency.

These processes are not only variable and non-transparent but are extremely slow. Anecdotal evidence shows how cautiously institutions approach these issues. Recently, investigative journalism by a TV station and a student newspaper pushed the Karolinska Institute and Stanford University respectively to broaden their investigations of research integrity. Both institutions then revised their initial conclusions, leading to the departures of their Dean and President respectively.Reference Ward and Anderson ¹¹ A claim of a faculty member’s exoneration after consideration of a large portfolio his papersReference Sinha ¹² was removed from the website of Queen Mary’s University of London only after publication of the actual conclusions of their own investigative committee of experts that upheld all concerns of the whistleblowers.Reference Sinha ¹³ An alleged systematic pattern of scientific misconduct over decades at both the University of California San Diego and the agency by an influential leader in the National Institutes of Health has recently been reported.Reference Piller ¹⁴

Some Solutions

This issue of ensuring scientific truth has many stakeholders, each with their own incentives, budgets and investigative powers. The approach has been highly variable, lacking transparency. It is inadequately prioritized and therefore, resourced. Addressing it is thus highly complex and requires the major players — academic institutions, journals, funders and industry — to act both collectively and in concert and for each seriously to up their game. This is a big ask. However, our procedures for assessing the quality of the biomedical literature are not fit for purpose.

A few suggestions:

• • Segregate the approaches to questions of fraud and validity.Reference Thorp ¹⁵ This is perhaps the most fundamental recommendation. Implementation of the recommendations below will let us get quickly to the question posed most commonly by the biomedical community — do I have reason to doubt the validity of the data in the paper I am reading? Institutions, journals and funders each have distinct obligations to answer that question. By contrast, answering a tougher question — was fraud involved? — is a more complex one to address for a narrower audience, is led by the employer and is more prolonged, usually taking years.