Guest Editors: Urs Fischbacher and Irenaeus Wolff
Experimental Economics will publish a Symposium of pre-results-reviewed papers that will be guest-edited by Urs Fischbacher and Irenaeus Wolff. The Symposium will consist of 5–7 papers to appear in the autumn or winter issue 2021 of Experimental Economics. The idea of pre-results review is that authors submit their fully motivated and fully specified research plan before gathering the data. This submission procedure allows both authors and referees to focus on the importance of the research question and it provides incentives to publish the results of well-crafted experiments even after a null result. The deadline for submissions is January 15th, 2020.
1 Details
Recently, there has been a growing concern with respect to the (non-)replicability of scientific findings. The Symposium “Pre-Results Review” will consist of a collection of papers in a 2021 issue of Experimental Economics. It pilots an approach with the potential to alleviate the replicability crisis: reviewing experimental papers before the data has been collected.
Papers to appear in the Experimental Economics Symposium “Pre-Results Review” will go through two stages, a proposal stage and a full-paper stage. In both stages, there will be a review process, although the criteria for the two stages are different. A proposal is similar to a full paper without a results section. In place of the results section, authors specify the statistical analyses to be executed once the data are collected (after the proposal has been accepted). The proposal further includes a power analysis and a brief (three-sentence) strategy-method conclusion. Proposals will be assessed based on the importance of the research question, on whether the proposal specifies all details of the experiment and subsequent analyses, and whether the design allows one to answer the research question, which includes the issue of whether the study will have sufficient power. The editors may ask for changes to the proposed experimental design and procedures at this stage if they make a revise and resubmit decision.
Once a proposal has been accepted, the authors run the experiment and write the paper. For the full-paper review, there are two cases: the authors submit a ‘minimal version’ of the paper that does exactly what the authors had promised, or the authors submit an ‘extended version’ that includes additional parts. If the authors submit a ‘minimal version’, the full-paper review merely serves to check that the experiment and statistical analysis indeed were exactly as specified in the proposal (notably also that the authors did not leave out any pre-specified analyses). This is a possibility that will always be open to the authors, even if they first try to submit an ‘extended version’ which is rejected in the full-paper review stage. If the authors instead submit an ‘extended version’ with additional treatments, additional statistical analyses or any other additions, the full-paper review will fall into two different parts: the part of the paper that coincides with the ‘minimal version’ will only be checked as outlined above. All additional parts have to be marked as such, and will be subject to the standard review criteria that also apply for standard submissions.
2 Advantages for submitting authors
• Scholars can focus on high-impact questions rather than high-impact results, which need not be equivalent.
• Given experimental funds only have to be available after the acceptance decision, the acquisition of funds will be much easier (as risk is eliminated for the funding institution; this may not apply to the present Symposium due to the time schedule we will have to follow).
• Contrary to what intuition may suggest, the required effort for the publication of a paper may decrease: the analyses contained in the published version would have been necessary also under the traditional regime; on the other hand, it is no longer necessary to re-write full papers time and again to make them fit a different ‘story’. Also, the procedure obviates the need for many of the usual robustness checks.
• In contrast to the pre-trial registry that many people think may solve the problem of the publication bias, pre-results review does not suffer from a lack of incentives to complete an article after a null-result.
• Problematic experimental-design choices can be dealt with before a lot of money has been spent, for two reasons: proposers need to think about design issues more carefully than is currently the case, and reviewers can spot the few remaining issues before the experiment has been run.
3 What has to be submitted at the proposal-submission stage?
A project proposal submitted for publication in the Experimental Economics Symposium “Pre-Results Review” in principle should be an anonymised finalised paper with empty spaces where the prospective data is to be filled in. Conclusions also should be spelled out, with text parts to be included contingent on the results. Most importantly, the paper should include the following:
• Introduction and review of relevant literature.
• Presentation of the experiment to be analysed.
• Hypotheses derived theoretically or from earlier evidence (including a report on any pilot experiments that motivate the research proposal).
• A detailed experimental protocol of the study proposed.
• Plan of statistical analysis, including a power analysis, and
• A brief “strategy-method” Conclusion (contingent on the outcome of the statistical analysis).
A ‘template’ for a proposal is provided below.
4 Review criteria for proposals
1. The importance of the research question(s).
2. Does the proposal include a replicable description of the design?
• Would the clarity and degree of methodological detail be sufficient to replicate exactly the proposed experimental procedures and statistical analysis?
• Is the description of the methods sufficiently clear and specific to prevent undisclosed flexibility in the experimental procedures or statistical analysis.
3. Does the design allow the research question to be answered?
5 What has to be submitted at the paper-submission stage?
• The full paper; if any details of the proposal are taken out of the final paper, authors should include the accepted proposal as supplementary materials.
• It is important that the results of all hypotheses pre-specified in the proposal must also be included in the paper. Authors are free to perform exploratory analyses (not fully pre-specified in the proposal) and report the corresponding results in the paper. However, these analyses must be clearly marked as exploratory. In case a hypothesis from the accepted proposal turns out to be logically flawed, authors are allowed to move it to an Appendix (if particularly lengthy) or include it as a footnote.
• Papers should follow the same publishing policies and formatting rules as regular articles submitted to Experimental Economics. In particular, authors should adhere to the guidelines and policies on Publishing and Ethics, Data Sharing instructions, Copyright policy and Artwork and media formatting instructions. If there were any changes to the design between the accepted proposal and the paper, authors have to report them clearly and justify them. In this case, a proposal acceptance can (but need not) be revoked.
6 Review criteria for full papers
1. Did the authors adhere to the proposal (motivation, hypotheses, experimental design, and statistical analysis)? Is the paper complete in reporting all of the analyses specified in the accepted proposal?
2. Did the authors clearly mark any additional parts (in particular, additional statistical analyses) as exploratory? These parts are subject to the same review criteria as in standard submissions.
Note that the paper review under pre-results review only serves to ensure that the study under review meets the criteria above. While reviewers may suggest further exploratory data analyses in the process, the editor will ordinarily not consider these to be compulsory. In case a paper is rejected on the basis of the above review criteria, the authors keep the right to submit a minimal version of the paper that fulfills criterion 1.
7 Timeline for the special issue
• Deadline for submissions to the Special Issue: 15th January, 2020.
• Stage-1 reviewing process (to be completed by 15th July, 2020): will need to be relatively active (requires responsiveness on behalf of referees as well as authors).
• Gathering of data and completion of Stage-2 submission (submission deadline: 15th March, 2021).
• Stage-2 reviewing process (to be completed by 31st July, 2021); in case the authors submit a minimal version of the paper (i.e., without any changes in the experimental design and procedures as well as without any additional analysis parts), the final decision shall be made by the editor within 1 month after the submission date.
• Symposium to appear in Experimental Economics autumn or winter issue of 2021.
8 “Template” of a proposal
1. Introduction (Research question: background, literature, importance; also describe any relevant pilots that you have conducted: note how they differ with respect to your proposed experiment, and briefly describe the data/results). Authors are required to affirm that no experimental sessions that will be reported in the paper have been conducted in advance of the submission of the proposal. Any pilot data already gathered may not be included in the data set analysed in the paper.
2. Research strategy/design
• The proposed experiment: description; the instructions and screenshots at the very least of any critical screens should be included in an Appendix. We recommend submitting authors to have the experimental programme finalised and ready before submitting.
• Hypotheses on explicitly specified statistical objects.
3. Power analysis
• Please include statistical power calculations to justify the proposed sample size. Specify (and justify) the (expected) effect size and the targeted statistical power (useful information and software tools for power calculations can be found, e.g., under https://www.povertyactionlab.org/research-resources/software-and-tools).
4. Data analysis
• What statistical methods will be used to analyze the data and what are their underlying assumptions?
• Will missing values be an issue, and if so, how will you deal with them?
• How do you define and handle outliers?
• Specify exactly the variables of interest and the statistical tests/models you intend to use on them (if applicable, including its functional form).
• How will the study address false positives from multiple hypothesis testing?
5. List of references References can be in any style or format as long as the style is consistent.
