Giving vague answers to precise questions when preregistering a typical study

Imagine we were interested in whether workload during the workday affects exhaustion the next morning using experience sampling methodology (ESM). Upon filling the blanks of the preregistration template, it is straightforward to state that workload predicts higher levels of exhaustion the next day. However, the going gets tough when it comes to calculating test power and optimal sample sizes. The first task is determining our anticipated effect size. Specific ESM studies (Prem et al., Reference Prem, Paškvan, Kubicek and Korunka2018) or meta-analytic evidence (Crawford et al., Reference Crawford, LePine and Rich2010; Alarcon, Reference Alarcon2011) may provide average effect sizes of the bivariate links between workload and exhaustion. However with few exceptions (McCormick et al., Reference McCormick, Reeves, Downes, Li and Ilies2018), the meta-analyses on this topic speak to the between-person level of analysis, most studies refer to proxies of our focal measures, and random effects estimates typically suggest that a wide range of effect sizes is plausible (Brannick et al., Reference Brannick, French, Rothstein, Kiselica and Apostoloski2021). Hence, based on the literature, a calculation of the standard meta-analytic effect size is imprecise at best. Design features might also change our effect size. For example, the effect size might vary when studying workload and exhaustion with a time lag of a few hours compared to measuring constructs concurrently (Pindek et al., Reference Pindek, Arvan and Spector2019). Without knowing unique temporal effects, we can only assume that the effect sizes are similar across these two design choices. Because we plan to study changes in exhaustion from one day to the next by controlling for exhaustion, we would need coefficients from a study that has studied changes in exhaustion across a similar period—another feature that may change the anticipated effect. Given that such a study is rare or may not exist at all, we end up giving vague answers to specific questions about power analyses upon trying to preregister a typical ESM study.

Ambiguity in preregistration might also arise in trying to detail the type of analyses we will be conducting. For example, we might document planned analyses, such as (serial) mediation, a bi-factor structure, accumulation effects, parallel change, a three-level data structure, (multilevel) group comparisons, or cross-level interactions. However, if the factorial structure is not supported in our measurement model, we may need to revise the measures used for hypothesis testing. Perhaps there is too little variation within-person on our exhaustion measure, or perhaps we were unable to get enough participants to conduct our hypothesized multilevel group comparisons. The decision trees on the analyses, even for testing rather simple models, might soon appear infinite. Moreover, our decisions are still full of uncertainty, depending on the amount and quality of the data we will end up getting after study completion. Even with the clearest of intentions, a final analysis plan cannot be established without knowing the true properties of our data once it is collected. Again, we are therefore likely to give vague answers to specific preregistration questions beforehand.

Being honest about what we know in advance

For researchers in I-O psychology, preregistration tends to create a catch-22 scenario: We have to ground our study in specific sample and data handling information. Typically, however, neither existing theory nor the available empirical evidence is precise enough to justify such specific study design decisions. If precise information is well-documented, we run the risk that our planned research is less novel. We therefore propose being more honest about what we may know in advance and what we may not, and presenting the study as such in the preregistration and paper. We can simply admit that our design and analysis plans are guided by our best educated guesses of the nature of eventual data, collected within our anticipated context. Moreover, we can clearly communicate the developmental stage of our research programs in an effort to clarify the rationale for inductive and deductive approaches, and to ensure early-stage research programs are not held to the same standards of precision as more developed research programs.

We argue that opening doors explicitly for inductive elements in quantitative empirical research would improve transparency and help to refine theories—after all, inductive testing is just as much part of the research cycle as deductive testing (Hollenbeck & Wright, Reference Hollenbeck and Wright2017). Once we explicitly acknowledge how our theories are deficient, permitting ambiguity and induction can help to fill these gaps in a programmatic way (Bamberger, Reference Bamberger2019). For example relevant to ESM preregistration, inductively testing and reporting effects across different lags might advance theories that are mute regarding temporal issues (George & Jones, Reference George and Jones2000; Shipp & Cole, Reference Shipp and Cole2015). Moreover, while deductively obtained non-significant findings may illustrate that our prior scholarly thinking needs revisiting, inductive exploration of those findings allows us to examine how these views need to be refined (Spector, Reference Spector2017; Spector et al., Reference Spector, Rogelberg, Ryan, Schmitt and Zedeck2014). We contend that we will not make progress as a discipline unless we leverage the richness of information inherent in the data we analyze. Being honest about the specific gaps in our research, and the type of goals we intended to reach with this research (inductive versus deductive testing) will contribute to create the conditions open science depends on.

Overcoming misconceptions and taking ownership of open science practices

Thinking through a study to plan the approach and contingencies likely facilitates doing better research, even when there are still vague answers. However, many of the existing templates and materials rooted in basic research pressure researchers to fill each and every blank and to provide precise answers when only vague ones are possible. They may also signal that research should be exclusively deductive in nature. We would like to set these misunderstandings straight. Rather than forcing the deductive paradigm on open research in I-O psychology, we should identify and refer to templates that fit our way of doing research (for an adequate example see the PRP-QUANT template by Bosnjak et al., Reference Bosnjak, Fiebach, Mellor, Mueller, O’Connor, Oswald and Sokol2022). Even with a good fitting preregistration template, we may only be able to fill in a few blanks as exemplified above. However, providing some detail is always better than providing no information at all. Being honest about what we know and what we cannot know in advance would align well with open science. Filling out a preregistration form should not become such a constraint that researchers try to avoid it. Allowing for blanks, ranges, or educated guesses that are declared as such, can help transform the preregistration process from a time-consuming and frustrating experience to a developmental stage (Daft, Reference Daft1983) that helps researchers anticipate problems and create contingencies.

Of note, open science practices that unveil some degree of ambiguity in decision-making would need to be understood by editorial teams. In our view, allowing for induction in preregistration during the early stages of a research program may open the door to more novel and relevant research, that can lead to a robust and replicable body of evidence when refined and improved in follow-up research. Preregistration can help make this happen, by providing a platform for open communication among researchers about what they aimed to find in their studies, and ultimately observed.

Reconciling confirmation and discovery

Being aware that deductive, abductive, and inductive research complement one another is not sufficient. We need to get involved in designing, adopting, and adapting open science practices in a way that they fit the unique nature of our discipline. As I-O psychologists, we need to capitalize on the unique opportunities we have: While embracing more flexibility in open science, we can still provide evidence that is real in naturalistic field settings where not all variables are held constant. Moreover, we can quantify, rather than cancel out, the role of context. We can study phenomena that emerge in the world of work and that may not (yet) exist according to theory, but that are familiar to millions. In this way, we can move open science from confirming what we believe we already know to discoveries that matter.