1 The motivations for introducing pre-results review

As we pointed out above, pre-results review is meant to tackle various forms of questionable research practices. Having said this, our primary motivation for promoting pre-results review is to fight publication bias, which we consider the prime reason for replication issues in psychology and economics. In our view, far too many studies end up in the ‘file drawer’. These studies are not written up due to a lack of incentives because the (perceived) chances of publishing a null result or even inconclusive results are just too low. However, coming back to the above argument, if a stringent selection is applied during the pre-results review phase, the published results, positive or negative, will be among the most credible in the journal. And because the profession rightly is moving away from the idea that one study is generally enough to decide a question, even high-powered inconclusive results are important if the question is important enough.

Widespread usage of pre-results review would have positive side effects. In particular, it would lead to a better-targeted use of resources. First, eliminating the publication bias would strongly reduce the multiplication of efforts of multiple researchers trying to demonstrate a non-existent relationship. Second, false-positive results would become rarer because power calculations are required in Stage 1, which are also reviewed. Consequently, the number of follow-up studies based on false-positive results would be reduced, too. Third, there would be fewer experiments with flawed designs because referees (and sometimes potentially even the authors themselves) would spot bad design choices before the experiments are run. Fourth, acquiring funds may become easier for young researchers if they demonstrate to a funding organisation that the research will be published.Footnote ⁶ Fifth, while the up-front investment needed for starting the project will increase strongly, obviating the need to re-write a study time and again is likely to decrease overall time costs. In Chris Chambers’ words: “Because the study is accepted in advance, the incentives for authors change from producing the most beautiful story to the most accurate one.”Footnote ⁷ In this context, the “most accurate” story means a neutral presentation of the data rather than a selection of analyses to fit a particular story. Sixth, in some cases, the procedure may even lead to a wiser choice of projects to be pursued because researchers think about a project more carefully before they start it, and the higher up-front costs are likely to deter them from working on less important research questions. Seventh, as Dufwenberg & Martinsson (Reference Dufwenberg and Martinsson2019) have pointed out, even the incentives to cheat are reduced. The basic argument is that if the article will be published irrespective of the results, there is a lower incentive to report particular outcomes and less of a need to fabricate a perfect story. Dufwenberg & Martinsson suggest results-free review exactly as a solution to the problem of existing incentives to cheat. And eighth, pre-results review would allow going back to a meaningful double-blind process for the initial submission, which might attenuate or even eliminate a seniority bias.

2 Common reservations and our experience

Having pointed out the advantages and benefits pre-results review has, we now also discuss potential downsides. In the following, we present the most common reservations we have heard and juxtapose them with our experiences where relevant.

The first reservation is that we might receive only low-quality submissions because people might prefer not to submit ‘revolutionary’ ideas. They would like first to conduct the study and decide on the submission target when they can judge the scope of the result. Two things can be said about this first reservation before we look at the data. On the one hand, the up-front costs increase, which should discourage low-quality ideas. On the other hand, for revolutionary ideas, authors may hope to publish the study in a top general-interest journal, which may indeed discourage high-quality submissions. Yet, it is not clear why a discouragement should happen even under the somewhat restrictive terms of the Symposium (that authors cannot hand in the final paper at another journal and fall back on the in-principle acceptance at Experimental Economics after a rejection).Footnote ⁸ Under the published terms, it would have been possible to submit, run the experiment, and re-tract the study for the Symposium in case of revolutionary findings. If authors were to follow this path in substantial numbers, this would indeed be a problem for Experimental Economics as a journal, as we would be bearing the costs while another journal would be reaping the benefits. Yet, for science as a whole, the pre-results-review procedure would be beneficial, as it would ensure that the results would be published irrespective of the results. The downside would be that the editors and referees whose time and effort were used to advance science would likely become resentful that they put in hard work as long as they cannot get any credit for it.Footnote ⁹

Judging by the submissions and the associated reviewer recommendations, there is little evidence of low-quality submissions. Twenty-one submissions is a substantial number in light of the window for submissions. Judging by the recommendations, the submissions had a quality comparable to those of standard submissions to Experimental Economics. At the same time, note that Soderberg et al. (Reference Soderberg, Errington, Schiavone, Bottesini, Thorn, Vazire, Esterling and Nosek2021) ran an experiment in which 353 scientists rated a sample of published, partially blinded publications submitted either under the standard procedure or under pre-results review. The participating scientists had to rate the studies on 19 characteristics. Numerically, the publications that had gone through pre-results review outperformed the standard publications on all characteristics, and significantly so for 12 out of 19 characteristics, amongst them methodologic rigour, how much can be learnt from the article, and overall article quality.

Related to the concern that too few high-quality ideas might be submitted is the concern that people might use pre-results review only as a cheap improvement device for their studies. In the case of our special issue, we chose not to have an ‘entry bar’ in terms of desk-rejecting some of the submissions. The main reason for sending out all submissions for review was that we were inexperienced with the format and wanted to give all submissions their best chance of being included in the Symposium. We probably would change this policy if pre-results review was introduced as an additional standard submission option. Having said this, we did not have the impression that any of the submissions were rough sketches in the state of a preliminary draft. Virtually all of the submissions gave testimony of a substantial amount of thought and effort by the authors. Given that, the “improvement device” (if any) is by no means “cheap”.

Suppose a journal offering pre-results review were to note at some point in time that a non-negligible share of authors is using pre-results review as a cheap improvement device. In that case, one would, of course, need to think of measures that could be taken to address this problem. A first measure would be to introduce desk rejections, as pointed out above. A second measure could be to introduce submission fees, although a submission fee would probably need to be very high to serve as an effective entry bar. A measure that would solve the problem would be not to provide feedback for rejected papers. However, such a measure contradicts transparency requirements and leads to inefficiencies because reviewers’ thoughts would get lost. After all, the most relevant cost may be time, which might be particularly high for innovative ideas in the air. However, time costs are already present if we do not aim for the extremely short turn-around times that some general-science journals work with.

A third reservation is that laboratory research does not need pre-results review because one can easily re-run studies. In our view, this is a good argument for why the pre-registration of hypotheses (which is part of the pre-results-review process) may not be as crucial for lab experiments as it is for field experiments. Laboratory experiments, particularly those considered to be important, are naturally replicated in follow-up experiments, and null results in these replications may eventually find their way into the journals (even though it typically takes more resources to correct a published result than to publish a result without precedent). However, the main aim of introducing pre-results review is to address the publication bias before a (false-)positive finding has been published. It guarantees that null results are published before so many attempts are made that a “statistically significant” effect is recorded.

A fourth reservation is that scholars might try to cheat. The concern was that scholars might run an experiment, obtain a null result, and submit the paper results-free as a proposal. The proposal gets accepted based on the research question and the design. There are two possibilities: if the submitting scholars report the old data, that runs under fraud. This is something we cannot prevent (similar to, e.g., faking the data under the standard process). If, however, the scholars re-run the experiment (which they have to do anyways in case reviewers suggest any changes to the design), then new evidence concerning this research question is generated.Footnote ¹⁰ The important research question has received an answer, which is an important piece of information. As Dufwenberg & Martinsson (Reference Dufwenberg and Martinsson2019) argue, pre-results review even reduces the incentives to cheat, particularly in reporting incomplete data sets or tweaking the data (see above).

A fifth reservation is the problem of an increased seniority bias. Again, there are several answers. If pre-results submissions were handled under a double-blind protocol, then any seniority bias should be reduced or prevented. Without a double-blind review, it is an empirical question. Of course, we do not have sufficient data to assess the seniority bias statistically. Nevertheless, two out of three Revise-and-Resubmit decisions went to author teams containing scholars ranking in the top 10% of the Repec Ranking (all publications). At the same time, there were only seven such scholars among the 27 submitting authors. In this light, it might be reasonable to switch back to a double-blind review. This is much easier with pre-results-review submission because nobody will publicly post their research proposals before submission. In contrast, authors are often identifiable in standard submissions because working papers are available online.

A sixth reservation is the risk that authors provide low-quality work at the second stage of the process, in the data collection or the data analysis. Generally, the paper should have high quality because quality is ensured in the first submission stage, and the second submission stage ensures that authors will perform well on the promised analyses. Also, authors are likely to provide high-quality work because their reputation is still at stake. As pointed out above, the evidence from other fields gathered by Soderberg et al. (Reference Soderberg, Errington, Schiavone, Bottesini, Thorn, Vazire, Esterling and Nosek2021) suggests that the quality does not suffer in any dimension and improves in terms of how much can be learnt from the article. However, it is entirely conceivable that there will be less explorative data analysis. While this is an open empirical question, there is a potential trade-off here. By providing an institution that ensures that analyses become more reliable, we lose some handle in incentivising more detailed exploratory analyses.

A seventh reservation is how to roll out pre-results review on a larger scale. This Symposium provides the first evidence of how it works for the journal—what kind of submissions we can expect and how the refereeing process works. As our discussion above shows, we did not encounter major problems and received good quality. However, implementing pre-results review on a larger scale could affect these outcomes. For example, authors, reviewers, and editors involved in the Symposium may have been particularly motivated. There are also specific issues that would arise. First, implementing pre-results review as a standing option would require another editor on the editorial board. It is less evident whether more reviewers will be needed. On the one hand, the number of initial submissions would increase to a certain degree (our experience suggests an estimate of roughly 30 submissions per year). On the other hand, fewer revisions will be the standard in the second stage because most parts of the second-stage submission have already been reviewed (in the case of the two articles in our Symposium, there was no need to re-invite the reviewers in the second stage altogether).Footnote ¹¹

It would definitively be necessary to extend the editorial system to add additional default texts and to take into account the additional states (most prominently, “in-principle accepted”; for the Symposium, we technically had to mark in-principle-accepted submissions as “revise and resubmits”). A larger roll-out also requires—in particular at the beginning—more instructions for editors and referees to explain the specific features of the new reviewing process.

The most prevalent concern, however, is what the above quote already hinted at: that the journal might end up publishing ‘too many uninteresting results’. But then, again, shouldn’t it be more important to us to get the true answer to an important question, examined through a well-suited design, than to read about ‘interesting results’ that have a far higher probability of being false positives? We need to know ‘uninteresting results’ if they are answers to important questions. And pre-results review is the best way to ensure that.

Overall, we are convinced that pre-results review is an option that solves many problems concerning the replication crisis. This Symposium also shows that it is feasible in principle. For Experimental Economics, we see two reasons to consider it risky to roll it out at the current stage. First, more experience is needed to know how the process scales up. Second, pre-results review is still uncommon in economics, and so Experimental Economics might lose in-principle-accepted papers to top journals.