Journal information | Journal of Experimental Political Science

The Results-Blind Review Track: JEPS Accepts Registered Reports

Review Process for Registered Reports

Review process diagram

Registered reports are a form of results-blind review. The review process for traditional papers happens after the study has been completed and the data analyzed. To the extent that reviewers and editors lean too much on the results to judge the importance of a paper’s findings, it creates perverse incentives to publish positive results. In the traditional review process, registering pre-analysis plans makes research more transparent and more credible, but it does not change the way research is evaluated. Consequently, it does not fully resolve the problem.

As the figure above shows, results-blind review addresses the problem by placing the beginning of the review process before researchers conduct an experiment (or set of experiments) and asking reviewers for an ex ante evaluation of a proposed study’s novelty, importance, and quality. Registered reports that meet these high standards are “accepted in principle” and researchers are given the green light to conduct the study. After the study is conducted and the results are added to the registered report, the stage 2 review focuses on the execution of the experiment(s) and not the results.

Frequently Asked Questions

How does the review process for registered reports work?

(Adapted from Figure 2, p. 7, from Chambers, Feredoes, Muthukumaraswamy, and Etschells. 2014. “Instead of ‘Playing the Game’ It is Time to Change the Rules: Registered Reports at AIMS Neuroscience and Beyond.” AIMS Neuroscience, 1 (1): 4-17.)

In Stage 1, reviewers evaluate the importance of the research question and quality of the proposed design. At the conclusion of this stage, the editor may reject the submission, request revisions that may be sent back to reviewers, or accept the submission “in principle.”

If a paper is accepted in principle, the author(s) are asked to register the report on a registry (e.g., EGAP, OSF) and given sufficient time to conduct the study, data analysis, and write up the results.

In Stage 2, researchers resubmit papers that have been accepted in principle after the proposed research has been conducted. Reviewers are asked to evaluate whether the researchers carried out the research in the way they proposed in Stage 1 or adequately justified any divergences from the pre-analysis plan. We will make every effort to recruit the same reviewers from Stage 1, but if reviewers are unavailable, we may recruit a new reviewer and provide this person with all of the details from Stage 1. Submissions that clear this stage are accepted for publication and subjected to the same publication process as other submissions (e.g., checks for reproducible results, etc.)

Like a traditional paper, the Stage 1 review process could take 3-6 months and more if extensive revisions are requested, and author(s) should account for the review process when contemplating their study timeline.

What are the criteria for evaluating registered reports?

Stage 1

Reviewers will be asked to review the registered report along five dimensions:

Importance of the research question
Soundness of proposed hypothesis
Soundness and feasibility of proposed design (including statistical power)
Whether proposed experiment offers an adequate and appropriate test of hypothesis
Whether proposed methods are appropriate and sufficiently detailed

Like all manuscripts submitted to JEPS, AEs have the option of desk rejecting the registered report if s/he believes that it is fundamentally lacking on any of the five dimensions. Registered report sent out for review will be handled in a similar way. The AE may conditionally accept the registered report, ask for revisions, or reject it. AEs are encouraged to set a high bar for R&Rs. The proposed revisions should be doable and straightforward as well as promise to be reasonably successful at addressing reviewers’ concerns if taken seriously by the authors.

Stage 2

Registered report that are accepted-in-principle are subject to a second stage review where, to the extent possible, the original reviewers are re-contacted and asked to assess the manuscript along four dimensions listed below.

Research question and rationale for hypothesis did not change from Stage 1.
Experimental procedures detailed in the registered report were followed closely and any departures are noted and justified.
Unregistered post hoc analyses are clearly labeled, justified, methodologically sound, and informative.
Conclusions are justified by the data. (Considerations of data quality fall here).

Reviewers are asked not to consider the perceived importance, novelty, or clarity of the empirical results.

If the reviewers and AE conclude that the manuscript did not adhere to the registered report, the manuscript will be either rejected or, in the case that the AE deems a revision to be eminently doable, given a chance to make revisions.

Accepted registered reports must meet guidelines for publishing other manuscripts at JEPS.

How does writing a registered report differ from a traditional research paper?

A registered report only differs from a traditional research paper in that the proposed experiment has not yet been conducted. Otherwise, a registered report should look very much like a traditional research paper. A registered report should consist of the following elements:

Introduction that summarizes the research question.
A brief literature review that summarizes previous research, making clear what gap(s) in the literature the proposed research will address.
Clearly stated hypotheses as well as the theoretical basis for them.
A discussion of pilot data if any have been collected.
A detailed description of the proposed experimental design and procedures, with all materials (e.g., scripts, question wording, etc.) placed in an appendix.
1. Rationale for sample size (e.g., an a priori power analysis
2. Inclusion and exclusion criteria
A detailed explanation of how aspects of the experiment (design and measurement) relate to the hypotheses under investigation.
1. Full explanation of all measures as space permits (you may also include an appendix)
2. Explicit discussion of the empirical predictions implied by the hypotheses
Detailed explanation of the methods that will be used to analyze the experimental data.
1. Describe methodological approaches (e.g., statistical models)
2. Criteria for statistical inference (e.g., alpha-values, distributions of priors, etc.)
3. Criteria for manipulation checks (if there are any) and how author(s) plan to handle observations that do not "pass"
4. Corrections for multiple comparisons (if applicable)
5. Approach for dealing with missing observations
6. Explicitly state covariates and how you plan to use them in analyses
7. It is advisable (though not required) to simulate data and analyses so that reviewers get a concrete idea of the type of evidence that the authors will provide
Deposition in a registry before fielding study (e.g., EGAP, OSF, etc.).

Submissions that do not meet these criteria will be desk rejected. Well-organized papers that anticipate questions and concerns about the proposed research are more likely to be accepted in principle than papers that do not.

What is the word limit and can I have an appendix?

The word limit for registered reports is 4000, and authors are encouraged to use an online appendix as a way to provide details that reviewers need to evaluate proposed experimental designs.

Can I conduct my proposed experiment(s) before or during the Stage 1 review?

No. The proposed experiment(s) can only be conducted after the Stage 1 review process is complete and the proposed design reflects the reviewers’ comments.

Can I report results from previous research, even if it wasn't registered?

Yes. Previous research (registered or not) can be incorporated in a registered report in the same way it would be in a traditional paper — as a preliminary study in the main text or pilot study in the appendix. It is important that authors make clear in the text which results were registered and which were not.

Can I submit one aspect of a larger registered project as a standalone registered report?

Yes, but this should be made clear in the main text and an anonymized version of registered pre-analysis plan for the larger project should be provided in the online appendix, so that reviewers can consider the hypotheses to be tested in the registered report in the broader context of other proposed tests in the larger pre-analysis plan.

How detailed do I need to be when describing the research design?

As detailed as possible. Please use an appendix as a way to provide full details of experimental conditions, measures, and the like. Report everything that you would in a traditional paper, including power analyses, experimental stimuli, question wording, details about measurement (e.g., recoding, factor analysis, etc.), rationales for excluding data, details about statistical models, thresholds for inferential statistics, etc. See How does writing a registered report differ from a traditional research paper?

As is the case with a traditional paper, the burden is on the author(s) to organize the information in a way that is clear and accessible to reviewers.

Should I simulate the results of my proposed experiment(s)?

While this is not required, it is an excellent habit. It will help diagnose potential issues with the proposed design and offer a way to more clearly communicate to reviewers what the eventual paper will look like. Applications like Declare Design could be helpful in this regard (https://declaredesign.org/).

If my registered report is rejected, but I run the study anyway, can I submit it later as a traditional research article that features a registered analysis plan?

Yes. We recommend that the author(s) explain in the cover letter how the pre-analysis plan (and thus study) was modified in light of reviews from Stage 1.

If my registered report is “accepted in principle,” how long do I have to conduct the study?

One year, but researcher(s) can negotiate longer timelines if necessary.

If my registered report is “accepted in principle,” do I have to publish the registered report with JEPS?

Yes! Authors will be asked to agree at the outset of the review process that if their registered report is accepted in principle, they will continue on to Stage 2 of the review process and ultimately publish their paper with JEPS if it clears the second stage.

When do I need to register my experiment(s)?

Before fielding the study.

Where do I register my experiments?

On an established and recognized public registry for pre-analysis plans (e.g., EGAP, OSF, etc.).

What if I need to make unforeseen adjustments once I am in the field?

You will need to note and justify in the main text of the registered report any divergences from the pre-analysis plan that was accepted in principle.

If unforeseen circumstances cause the authors to make major deviations to the design and protocol from the accepted-in-principle registered report, authors are encouraged to reach out to the editorial office for guidance. Associate Editors have the option of consulting with reviewers in the first stage.

What if the results do not support my hypotheses?

That is fine! The strength of your paper should rest on the importance of the research question and quality of the research design (assessed in Stage 1 of the review process), not the perceived novelty of the results.

Can I conduct analyses that I did not register in the first stage of the review process?

It is best to register all of the planned analyses. Nonetheless, if an additional analysis only becomes apparent after the experiment is conducted, it is permissible to do additional post-hoc analyses, as long as you offer a defensible substantive or statistical rationale for the additional analysis and flag these as post hoc when writing up the data analysis. One of the strengths of registered research is that it makes transparent the distinction between confirmatory and exploratory analyses.

What if some of my analyses depend on the preliminary results?

You may register contingencies in your pre-analysis plan (e.g., if A result is obtained, B method will be used).

How do I avoid receiving a negative outcome after the Stage 2 review?

Any divergences from the proposed pre-analysis plans that weaken the research design (e.g., reduce its power, vitiate the causal identification strategy, or alters the estimand) will likely lead to a negative outcome.

Are there any resources I should consult before submitting a registered report?

The Open Science Framework offers a number of resources, including checklists, for registered reports: https://cos.io/prereg/

Nosek, et al. (https://www.pnas.org/content/115/11/2600) offer helpful reflections on challenges to registered work in practice.

Declare Design help researchers formalize their research design and simulate data (https://declaredesign.org/).