1 Imperfect reference standard—human RoB 2 assessments

We acknowledge that using published Cochrane RoB 2 judgments as a reference standard has its limitations, given the modest inter-rater reliability reported for the tool.Reference Minozzi, Cinquini, Gianola, Gonzalez-Lorenzo and Banzi ^{3 ,} Reference Minozzi, Dwan, Borrelli and Filippini ⁴ However, this limitation was already discussed in our article. As we suggested, future work could employ a purpose-built expert reference standard (e.g., from a panel of experts), although this would also require considerably more time and resources. Given the high methodological quality of most Cochrane reviews, the assessments by various Cochrane authors were arguably best suited for our project to establishing an external and feasible reference standard.

2 Augmentation rather than replacement

We agree that a “human-in-the-loop” or semiautomated approach may currently be the most practical way to harness the judgments made by a large language model (LLM). Our study did not test such a workflow, but we welcome further investigations. Plausible scenarios include using an LLM for an initial, rough pre-assessment or for flagging potentially critical sections of a report, which are then verified by human reviewers. Likewise, an LLM could assist with extracting or highlighting data relevant to the risk of bias assessment, as suggested in our discussion.

3 Information parity between model and human reviewers

In our study, the LLM received the full-text article together with a compressed protocol or registry entry (when available). Human Cochrane reviewers, in contrast, could consult any supplementary material, contact study authors, and examine complete protocols. This consequently resulted in a slight imbalance in the information available to LLMs and human reviewers. However, providing every piece of ancillary information for the 100 randomized controlled trials we examined would have been unfeasible. Moreover, as described in our article, Claude struggled with very long protocols. Therefore, we implemented a refined approach by supplying a compressed version to ensure that the model assessed the study itself rather than an extensive protocol.

4 Exploration of performance heterogeneity

Claude’s agreement with human judgments varied across RoB 2 domains, that is, highest Cohen’s κ for Domain 3 (“missing data”), followed by Domain 4 (“outcome measurement”), and lowest for Domain 5 (“selective reporting”) in our main analysis. This suggests that there is likely some heterogeneity in performance across the various RoB-2 domains. As described in our article, we conducted several additional analyses to investigate possible influencing factors, including, e.g., sensitivity analyses using two alternative prompts and one alternative model (Claude 3), subgroup analyses by intervention type (pharmacological vs. other—nonpharmacological, nonsurgical—interventions), and subgroup analyses by the availability of a published protocol or registry entry. Of course, these were not exhaustive. Although κ values fluctuated modestly in our additional analyses, none indicated consistently strong performance. We would therefore urge caution and advise against placing overly strong emphasis on individual κ values.

5 Temporal validity and rapid model evolution

We share the concern that evaluations of single LLMs can become outdated quickly in this rapidly evolving field, underscoring the importance of approaches that support reproducibility and continuous reassessments. Our study, conducted in February 2024, contributes an early evaluation within this dynamic landscape. Building on this, we emphasize that future validation studies should be strengthened by incorporating plans for reevaluation and by transparently reporting key details such as model version, run date, prompts, and data used to facilitate replication studies. The RAISE (Responsible Use of AI in Evidence SynthEsis) StatementReference Flemyng, Noel-Storr and Macura ^{5 ,} Reference Thomas, Hair and Noel-Storr ⁶ provides valuable guidance in this regard. Complementary initiatives, including the Digital Evidence Synthesis Tool EvaluationsReference Bond, Finnerty and O'Mara-Eves ⁷ and our currently ongoing living scoping review,Reference Eisele-Metzger, Bond and Gopal ⁸ are actively advancing the coordination and tracking of evaluation efforts.

We appreciate Sahu’s insightful appraisal and his constructive proposals. We look forward to further constructive discourse and collaboration with the systematic review community.