Incompatibility #1: Disclosure and sharing

For research reports emanating from practice settings, the principles of transparency and sharing can become obstacles to their entry into the peer-reviewed literature. Data sharing is an especially problematic issue. Consider this example of research conducted in a fast-food business involving over 500 restaurants and over 20,000 employees. It used a prospective research design to investigate the effects of multiple current-period workforce and workplace attributes on three measures of next-period restaurant performance (profit, customer experience, speed of service) for each of 12 consecutive months (Erland et al., Reference Erland, Gross and Guzzo2017). A large data set was amassed integrating observations from multiple sources. Some of the variables served as controls relevant to variance in performance measures such as the population density, median income levels, and unemployment rates of a restaurant’s local community. These variables came from publicly available sources. However, most of the variables came from proprietary sources including the company’s human resources information system (HRIS), financial, process control, and customer experience databases, much of which cannot be disclosed or shared. The company’s competitive business advantages could be harmed by divulging fully transparent, quantitative details about the critical dependent variables, its restaurants’ monthly performance metrics, or about certain performance-relevant covariates, such as the incidence of kitchen equipment repair. Employees could be exposed to risks of revealed identities if HRIS and workplace data are made available to others, and employers could be exposed to litigation risk to the extent that they are responsible for harm to employees through disclosure. Gabriel and Wessel (Reference Gabriel and Wessel2013) discuss how data-sharing requirements can place vulnerable populations at risk and how such requirements can stop research from ever being initiated. Risks may be reduced by efforts to alter the data to make at least some of it shareable (e.g., tactics such as renaming, perturbing, and sharding). Banks et al. (Reference Banks, Field, Oswald, O’Boyle, Landis, Rupp and Rogelberg2019) recognize the incompatibility between the principles of disclosure and sharing and research in organizations, and they suggest tactics to overcome the incompatibility such as delaying the release of original data and anonymization. However, none of the tactics mentioned here can guarantee that a clever data scientist could not identify at least some individuals through recombining identity-relevant variables in a data set (age, work location, job type, hours worked, hiring and termination information, etc.), especially if joined with local-area data from sources such as tweets, voter registries, and high school yearbooks. Adequate anonymization is an unresolved, multidisciplinary concern, and organizational researchers should be involved in the search for effective solutions.

Sometimes data from organizations will be fully shareable, and sometimes what is shareable is partial or is an abstraction of underlying data, such as a correlation matrix. Such a matrix may have variables deleted from it to avoid disclosures that are potentially harmful to an organization. Harm could come from reporting named variables that give strong identity clues or from reporting relationships, even among innocuously named variables, that are of value to competitors. Unfortunately, the greater the omission or obfuscation of variable names, the less valuable the shared matrix is for researchers’ reanalysis or replication. Nonetheless, some data sharing is preferable to none. The Journal of Business and Psychology, for example, communicates a clear “research data policy” that encourages sharing of all materials and raw data described in a manuscript and makes compliance easy by directing authors to relevant data repositories and by providing scripted “data availability” statements for authors’ to use. The journal’s policy, although strongly pushing for sharing, acknowledges that not all data can be shared. An insistence on complete transparency and full data sharing is not an absolute requirement in widespread use today in order to publish. It is undeniably true, however, that there is movement in that direction.

Looking back, it is interesting to speculate how many landmark studies in the organizational sciences might have failed to “score high” if subjected to open science criteria. The influence of the Hawthorne Studies (Roethlisberger & Dickson, Reference Roethlisberger and Dickson1939), the AT&T management progress studies (Bray et al., Reference Bray, Campbell and Grant1974), and the early performance appraisal studies at General Electric revealing how “split roles” for the appraiser can be useful (Meyer et al., Reference Meyer, Kay and French1965) might never have registered.

The threat that we see from disclosure and sharing is to hinder how frontline research informs science. Banks et al. (Reference Banks, Field, Oswald, O’Boyle, Landis, Rupp and Rogelberg2019), for example, observe that an organization may decline to participate in research that requires complete transparency and data sharing. We believe that imposing requirements of transparency and sharing such that they become obstacles to initiating research in organizations and barriers to the write-up of practice-based research reports for contribution to the scientific literature will have profoundly adverse consequences for the growth of realistic and useful knowledge (Rousseau, Reference Rousseau2012).

It is not just the need to protect individuals’ identity that is at issue. Additionally, companies cooperate in doing research such that organizations’ identities are known to researchers so that relationships can be established between, for example, employee survey data and company-level performance data (e.g., Schneider et al., Reference Schneider, Hanges, Smith and Salvaggio2003, on The Mayflower Group data). Companies will be appropriately loath to share such identifying information beyond the immediate team of researchers.

Furthermore, there is the important issue of intellectual property. Researchers in consulting firms and in-house research groups develop analytic tools and various selection, appraisal, and survey measures. Those organizations often may not wish to divulge the algorithms and/or measures to the public because of the investments they have made in their development. The research literature is replete with studies that have moved the field forward without such full disclosure and sharing being necessary. A recent paper by Pulakos et al. (Reference Pulakos, Kantrowitz and Schneider2019) provides an example of how this has typically been done but would no longer be possible if our journals required full disclosure and sharing. So, the Pulakos et al. article on agility and resilience in organizations provides a few sample items of the measure they developed and does not provide or make access available to the full measure or to the identity of the sample of organizations across which the new measure was validated successfully against organizational financial performance.

Lest we appear as mere rejectionists, each of this paper’s authors has been involved in research efforts that could have complied with open science practices had those practices been in effect at the time. Examples include Guzzo and Waters (1982), Dieterly and Schneider (Reference Dieterly and Schneider1974), and Nalbantian and Schotter (Reference Nalbantian and Schotter1995, Reference Nalbantian and Schotter1997). We also have extensive records of research involving people at work in privately held companies, the locale of almost all of our data-based work. Under open science rules regarding disclosure and sharing, we would certainly have had far fewer opportunities to contribute to science through peer-reviewed publications than we have had to date. We estimate that the number of our data-based publications would actually be reduced by at least 75%. We would not be able to share data that came from HRIS systems in companies, where individuals were identified so that follow-up studies for panel analyses were possible (Schneider, White, et al., Reference Schneider, White and Paul1998), where proprietary data bases were used (the Center for Creative Leadership in Schneider, Smith, et al., Reference Schneider, Smith, Taylor and Fleenor1998; A national insurance company’s insurance agents and agencies in Schneider et al.,   Reference Schneider, Ashworth, Higgs and Carr1996), or the study of the drivers of voluntary turnover at Fleet Bank based on the modeling of a panel data set constructed from the company’s HRIS data and focused on actual stay/quit events (Nalbantian & Szostak, Reference Nalbantian and Szostak2004), and so forth. The requirements for sharing and disclosure are but one issue that would inhibit the publication of the kind of research we do.

Incompatibility #2: Overidentification with the hypothetico-deductive model

Open science principles and practices are imbued with attributes of the classical hypothetico-deductive approach. Preregistration of hypotheses is a critical practice that promotes conformity with this way of building theory, as does the emphasis on replications of hypothesis tests. These types of open science practices explicitly rely on the hypothetico-deductive model of theory creation at a time when significant change is happening in the nature of science–practice research, especially research conducted in applied settings. That change involves “big data,” and it has big consequences for how best to advance theory.

Big-data organizational research is characterized both by lots of observations and by large numbers of variables. For example, Illingworth et al. (Reference Illingworth, Lippstreu, Deprez-Sims, Tonidandel, King and Cortina2016) describe selection research involving 125+ variables and a sample of 167,000 monthly assessments. They also describe research on coaching involving 80 to 120 variables. The previously mentioned Erland et al. (Reference Erland, Gross and Guzzo2017) case involved 78 variables in its analyses of the performance of 500+ restaurants and 135 variables in separate analyses of factors influencing voluntary turnover among 20,000+ employees. The latter is consistent with Hausknecht and Li’s (2016) account of the multiplicity of variables and often-large samples in studies of turnover. The breadth of topics in I-O psychology addressable by large numbers of variables from multiple sources is described well by King et al. (Reference King, Tonidandel, Cortina, Fink, Tonidandel, King and Cortina2016).

The availability of big data in organizational settings enables “the accelerated development of existing theories to account for more relevant factors and to become more complex, specific, and nuanced—and ideally more accurate and useful as a result” (Guzzo, Reference Guzzo, Tonidandel, King and Cortina2016, p. 348). A deficiency of the hypothetico-deductive approach is that it is, by definition, theory first. When 100 or more potentially meaningful variables are available for a research study in an organization of, say, 100,000 or more employees, there are many moderators, boundary conditions, levels of analysis, and causal pathways that are open to empirical tests. However, existing theories have not anticipated—could not be expected to have anticipated—such complexity and nuance. Consequently, existing theories offer too few hypotheses relative to the number of potentially meaningful relationships that can be rigorously tested in big-data research.

The emphasis on theory often leads to characterizing research as being of either two types, confirmatory or exploratory (Nosek et al., Reference Nosek, Ebersole, DeHaven and Mellor2018, speak of prediction or postdiction), the former being aligned with theory testing.Footnote ⁴ Big-data research exposes the limitations of this distinction. The absence of sufficiently complex theories in a big-data world makes thoroughly hypothesis-driven prediction impossible, thus “confirmatory” is impaired as a referent category. “Exploratory” is an old term loaded with connotations of being the poor relative of confirmatory research. Furthermore, the label obscures the value of alternative, orderly approaches to theory generation and testing such as induction and abduction, discussed below. The presence of big data also breaks the confirmatory–exploratory mold by creating within-study opportunities to advance and continuously test and retest explanations, say, with randomly selected samples of 10,000 or more from a larger data set—essentially enabling multiple replications and/or meta-analysis processes within a single study’s large data set. McShane and Böckenholt (Reference McShane and Böckenholt2017) present examples of single-paper meta-analysis in consumer behavior research. Within-study multiple replications and meta-analyses that are made possible with big data are at this time admittedly more aspirational than realized in publications. Indicative, though, is the work of Guzzo et al. (Reference Guzzo, Nalbantian, Parra, Schneider and Barbera2014), which reports results from 34 distinct replications, involving nearly a million employees, of investigations of the effects of various forms of compensation on voluntary turnover. Also indicative is Guzzo et al.’s (Reference Guzzo, Nalbantian and Anderson2022) single-paper meta-analysis of 23 studies, each averaging over 40 measured variables, investigating the influence of employee age and tenure on work unit performance. Existing theories and processes of discovery are concurrently embedded in those works. Indeed, with the proliferation of continuously refreshed and expanded data in organizations, the empirical testing that becomes possible constitutes an approach to theory building beyond anything achievable with traditional methods tied to the hypothetico-deductive approach. Practices that elevate traditional hypothetico-deductive approaches over alternatives that are better suited to big-data realities will hold back the advancement of knowledge. What are such alternatives?

Some research scholars propose that an inductive model where one explores interesting questions and obtains significant unexpected findings for later pursuit is a superior process for theorizing (Locke, Reference Locke2007). Locke illustrates how inductive processes based on research in both laboratory and field settings fueled the development of goal-setting theory, including the identification of moderating conditions that influence goals’ effects. The inductive model, indeed, may be quite useful for research in applied settings where questions emerge and data are available to explore possible answers to those questions, especially in our contemporary world that permits repeated testing of possible answers (cross-validation in I-O terms) on large data sets. McAbee et al. (Reference McAbee, Landis and Burke2016) also assert the value of inductive research, arguing that the organizational sciences would benefit from a “shock” to rejuvenate empirical advances and that big-data analytics in organizations joined with inductive inference deliver that needed shock. They also make it clear that an embrace of big-data analytics does not imply abandonment of foundational principles of measurement and causal inference.

It is safe to say that much of the work done by practitioner-applied psychologists has not been of the hypothetico-deductive model but of the inductive model, which, in our experience, proceeds from a question not a theory. For example, one of our streams of research on service climate (Schneider, Reference Schneider1973; Schneider et al., Reference Schneider, Parkington and Buxton1980) began with this question raised based on the 1973 paper: if customers in a bank’s branches say that they get lousy service because the employees are not service oriented, I wonder what kinds of conditions could be created that would encourage them to deliver superior service? This is how the stream of research on service climate began (see Hong et al., Reference Hong, Liao, Hu and Jiang2013, for a meta-analysis of this line of research).

Another model for theorizing is abduction, sometimes referred to as “inference to the best explanation” (Douven, Reference Douven2017). Unlike deductive or inductive inference making, abduction is about reaching credible explanations based on ensembles of facts, often involving multistage data analysis to arrive at the best of competing explanations (Haig, Reference Haig2005). It is especially well suited to big-data research. The authors’ research-based consulting often embodies the spirit of abduction. For example, when studying internal labor market dynamics (Nalbantian et al., Reference Nalbantian, Guzzo, Kieffer and Doherty2004), investigation is made by statistically modeling drivers of a set of outcomes: turnover, promotion, internal mobility, performance, and pay. Then the constellation of results is integrated to determine which of a set of preexisting theories relevant to these outcomes, in combination with empirical uniquenesses of the studied organization, best explains the patterns observed. Deploying theories this way helps uncover the story in the data in a compelling way that is easy to communicate to decision makers and easy for them to act on. It is also a more practical way of connecting theories to practice than starting with the presumption of one particular theory or model and then just testing it alone.

Prosperi et al. (Reference Prosperi, Bian, Buchan, Koopman, Sperrin and Wang2019) discuss the value of the joint presence of abductive and deductive processes in such research. They also directly challenge the wisdom of prevailing precautions against HARKing that are part of the open science movement, asserting that they conflict with the “modern reality” of researchers’ “obligations” to use big data (Prosperi et al., Reference Prosperi, Bian, Buchan, Koopman, Sperrin and Wang2019, p. 1). Behfar and Okhuysen (Reference Behfar and Okhuysen2018) also assert that abduction is a needed complement to hypothetico-deductive reasoning in the organizational sciences. Heckman and Singer (2017) make a case for abduction in economics, saying that it helps overcome shortfalls in doctrinal theories, and it challenges prevailing schools of thought that are unable to account for new and surprising facts. The abduction approach also is naturally more aligned with the systems view of organizations, a point we elaborate below, and opens the door to new theorizing in ways that traditional theory-driven testing would not encourage.

In the next section we will discuss the paradox of replication where we will note that qualitative case methods represent the antithesis of the requirement for replication (Pratt et al., 2019). Here we note that such methods inherently begin with the goal of discovery not validation, the opposite of the hypothetico-deductive model. Indeed, projects in single organizations, by either qualitative methods or the use of a variety of sources for quantitative data, can both be thought of as case studies: Explorations of data from a single organization with no or few prior hypotheses are as much a case study as though it were done qualitatively. In other words, a primary open science principle devalues two essential streams of research in organizational psychology, potentially rendering them inappropriate for subsequent publication: qualitative case methods and quantitative case methods.

A bottom line to our thinking on the issue of theory and theorizing is that (a) our field has simply overemphasized the necessity for having a theory to test before one begins research and that (b) the open science movement concretizes this overemphasis. In the early 2000s the Journal of Applied Psychology (JAP), felt compelled to issue a call for theory-driven research. The editors of the JAP special issue on theory (Klein & Zedeck, Reference Klein and Zedeck2004) put the need for theory this way: “Research in the absence of theory is often trivial—a technical feat more likely to yield confusion and boredom than insight. In contrast, research that is guided by theory, or that develops theory, generates understanding and excitement” (p. 931). It follows that problem-focused, inductively oriented work is trivial and of course uninteresting to JAP—at least in 2004.

Economics research is theory driven far more than problem driven. During the mid- and late-20th century, important theoretical developments appeared such as human capital theory, the operation of internal labor markets, and personnel economics. Unfortunately, the practical application and demonstrated value of these developments have fallen short of their theoretical promise. A major reason for this is inadequate access to organizational data and the resulting inability to inform theory by research that deals with practical issues. A recourse to laboratory experimentation ensued to test theories of incentives, decision making, and pricing mechanisms in organizations. As one of the authors (Nalbantian) can attest from personal research experience, experimental economics offers powerful insights on narrowly defined, technical questions. However, it does not come close to generating the learning that arises from rigorous examination of real-world data, which reflect actual decisions and outcomes in organizations and which create opportunities to examine whether theoretical predictions hold up across a multitude of contexts and circumstances.

In 2008, a wonderful review of the research published in JAP and Personnel Psychology by Cascio and Aguinis (Reference Cascio and Aguinis2008) revealed that (a) practitioners were not publishing in these journals (JAP had 4.67% of its articles by practitioners and (b) as would be expected from this figure, the articles were of not much relevance to the world of practice. Cascio and Aguinis concluded their review this way:

Early in our field, questions of import to individual and organizational performance drove applied research, but since then such important practice issues have given way to academic research driven by theory testing. The latter fits well with the open science model but the former fits better with the way research is done in practice settings. We are not alone in our views that theory is not the be all and end all for guiding research. Campbell and Wilmont (Reference Campbell, Wilmot, Ones, Anderson, Viswesvaran and Sinangil2018) argue that there are fundamental flaws in the theory-building process in the organizational sciences, including an overemphasis on theory for theory’s sake. Critiques of obsession with theory predate the rise of open science, such as that offered by Hambrick (Reference Hambrick2007) on the role of theory in management research:

Of course, we must make it clear that we are definitely in favor of deductive theory writing and research and have engaged in such efforts ourselves (Heilman & Guzzo, Reference Heilman and Guzzo1978; Nalbantian & Schotter, Reference Nalbantian and Schotter1997; Schneider, Reference Schneider1975; 1983). What we are not in favor of is making hypothetico-deductive theory writing and testing the predominant way to get articles published in the refereed journal space. The open science model, with its emphasis on hypothesis testing based on preexisting theories, can very easily become an impediment to knowledge advancement through problem-focused inductive and abductive work.

Incompatibility #3: The paradox of replication

The emphasis on theory-driven work along with disclosure and sharing create a serious paradox with respect to establishing robust relationships. Under pressures to replicate, what type of replication studies do we think researchers will attempt most frequently? It will be studies with simple hypotheses tested on small samples with few variables and not situated in applied settings. This type of study is most able to comply fully with open science principles designed to foster replication, and publication practices based on those principles will encourage more of the same. This is not the best way to improve the robustness of a field’s research findings. By comparison, complex questions studied on very large data sets can markedly improve the reliability of findings because such research can test relationships under a variety of circumstances and explicitly control for relevant variables (boundary conditions) that offer potentially competing hypotheses. In addition of course and as noted earlier, repeated randomly chosen data sets from very large databases can be used to do replications immediately, providing distributions of findings and relationships then testable via meta-analytic type models.

But small sample research prevails. Hernandez et al. (Reference Hernandez, Newman, Jeon, Tonidandel, King and Cortina2016) reported a median sample size of 73 for articles in Psychological Science, 90 for the Journal of Personality and Social Psychology, and 80 for regular articles (52 for brief articles) in Cognition. Marszalek et al. (Reference Marszalek, Barber, Kohlhart and Holmes2011) tracked four journals over time (Journal of Applied Psychology among them) and found median sample sizes of 48, 32, and 40, respectively, in 1977, 1995, and 2006. Shen et al. (Reference Shen, Kiger, Davies, Rasch, Simon and Ones2011) tracked JAP from 1995 to 2008 and reported median sample sizes of 173 for studies of individuals and 65 for studies of groups, and Kűhberger et al. (Reference Kűhberger, Fritz and Scherndl2014) randomly sampled 447 peer-reviewed journal articles in psychology and found that 85% had sample sizes less than 200. Small samples prevail in economics research as well. Ioannidis et al. (Reference Ionnidis, Stanley and Doucouliagos2017) quantitatively examined 159 meta-analyses covering 6,700 distinct empirical studies across a range of topic areas and found that in 50% of the areas studied, about 90% of the studies were underpowered. Further, more than 20% of the areas lacked even a single study with adequate statistical power. These results show that economic research, perhaps like organizational science research broadly, is extensively afflicted by residual bias that undermines the credibility—and likely replicability—of the statistically significant results reported.

Open science’s emphasis on replication to attempt to assure the certainty of published findings will encourage projects with small sample sizes that should necessarily raise concerns of adequate statistical power. But, when it comes to theory development, small-sample research has major implications beyond statistical power. It follows from small samples that (a) comparatively few variables can be analyzed in any one study, (b) few tests are able to be made of theoretical propositions of moderators or boundary conditions, and (c) there is limited opportunity to create new theoretical insights based on multivariate fact patterns. That is, the typical small to medium sample sizes in theory-testing research are actually incompatible with the need for strong tests of theory to achieve robust findings. Big-data efforts that are possible in practice permit, through induction and abduction, the testing and elaboration of complex theories based on robust findings. Prosperi et al. (Reference Prosperi, Bian, Buchan, Koopman, Sperrin and Wang2019) offer a new way of thinking about replication with big data. They suggest that it is usefully thought of as providing opportunities to reproduce (replicate) multivariate models of fact patterns rather than a very specific and narrow observation or inference.

Furthermore, there are blind spots in open-science-driven advocacy of replication. One is an overvaluing of exact replication of specific findings and an undervaluing of the role of meta-analysis of diverse studies as a means of establishing the robustness of relationships. Exact replication primarily is based on the degree to which a study finds or fails to find statistical significance. The problem is that we know, or should know, that statistical significance is highly dependent on sample size. It follows that with small sample sizes it is expected that there will be variability in the degree to which a given study will have its statistical significance replicated. It has been almost 45 years since Schmidt and Hunter (Reference Schmidt and Hunter1977) began showing that small sample sizes for studies masked the likely true relationships that were observed. In a summary of this work, Schmidt (Reference Schmidt2010) makes the following appropriate observation: “researchers appear to be virtually addicted to significance testing, and they hold many false beliefs about significance tests (p. 237).” The major false belief is that statistical significance indicates the probability of a study being replicated! Another blind spot concerns the association between sample size and effect size. Kűhberger et al. (Reference Kűhberger, Fritz and Scherndl2014) report a -.54 correlation between sample size and effect size. A literature of replicated small-sample studies thus runs the risk of overestimating the magnitude of “true” relationships. It is our personal experience that big-data-style research indeed yields smaller effect size estimates of X–Y relationships when compared with reports of the same X–Y relationship in published literature precisely because the big-data-style research controls for many other reasonable sources of variance in X and Y that the published research did not.

The paradox, then, is that attempts at replicating small-sample research will lead us away from robust theoretical insights. Although research conducted in practice settings may not be able to comply with the conditions that facilitate exact replication—full disclosure, sharing—and thus may be less likely to be replicated exactly, compared with its small-sample counterparts big-data-style research can yield more robust findings along with less risk of effect size overestimation.

To close the loop on the paradox of replication, we noted earlier that there is no logic that would require replication of qualitative case studies. As Pratt et al. (2019) noted in their Administrative Science Quarterly editorial essay, “replication misses the point of what the [case study] work is meant to accomplish” (p. 1). They note that the emphasis on replication in open science is an attempt to achieve trustworthiness but that “management journals need to tackle the core issues raised by the tumult over transparency by identifying solutions for enhanced trustworthiness that recognize the unique strengths and considerations of different methodological approaches in our field” (p. 1). Later we present some proposals for how to help resolve some of these replication issues with respect to publication.

Incompatibility #4: Evolving cultural and professional norms

We, like many readers, grew up in a professional milieu characterized by specific norms about how to conduct and publish research. Many of the old challenges, tactics, and rhythms of research that were part of that milieu have disappeared, thankfully. Mailing paper surveys, carrying boxes of IBM cards containing punched-in data to the computer center and waiting for printouts from behind the wall (and despairing at their thinness due to an error message), the snail’s pace of manuscript preparation, submission, and dissemination are now absent. Web-based surveys, ample computing power, and electronic management of manuscripts greatly improve the processes of conducting and publishing research. Important positive norms have persisted through these changes, such as the value of peer review in the publication process. However, we ask the following question: What happens when other norms that define “good” research change? Our fear is that practices aligned with open science principles will result in dramatic alterations in the norms that define “good” research in ways that may be bad for science–practice disciplines and what actually gets published in our journals and disseminated to the broadest audiences. We see several worrisome signs—preregistration sites that preclude use by business-based researchers, the privileging of confirmatory research, the presumption that applying open science principles in journals’ peer review processes is of interest only to academics and not to practitioners (Society for Industrial and Organizational Psychology, 2020)—that a cultural shift may be underway that devalues research that cannot fully conform to open-science-driven ideals. That cultural shift will unnecessarily (a) deter scholars from accessing the research opportunities presented by the explosion of useful data in organizations and (b) reduce the opportunities of practice-based researchers to contribute to the body of scientific literature. Another aspect of concern is the risk that open science practices will result in methodological sameness.

Consider the incompatibilities we have discussed and their potential effects on the kind of research likely to emerge. We have made our prediction about the type of replication research that will emerge. But what about other types of research? If disclosure and sharing become requirements for research to be deemed good and appropriate, where does that leave researchers who use proprietary measures, data, and samples that cannot be shared and/or disclosed (King & Persily, Reference King and Persily2019)? Will we then be a science of simple questions studied on small samples yielding unreliable and findings that are likely irrelevant to what happens in real organizations? If research is only deemed to be good when the hypotheses and the methods are stated in advance, what does that do to researchers who ask interesting and important practical questions and find new and interesting ways of developing measures (like via text analytics, for example) as the research process unfolds? What will happen to big, bold research endeavors that probe complex interactions of factors in and across organizational systems? Will our research become characterized by the laboratory studies now becoming so prominent in business schools that they hire more social psychologists so they can be “really” scientific? As Barry Staw (Reference Staw2016) has recently noted in his essay reflecting on trends he has observed in the field, “Although an increasing number of social psychologists are now housed in business schools, they are still doing social psychological rather than organizational research” (pp. 16–17). Staw’s Table 1 (p. 14) is very instructive wherein he notes that the social psychologist in business schools pursues lab and field experimentation to search for underlying processes rather than using multiple methods (including qualitative methods, observational and historical data) to search for upward, outward, and downward influences. Is small-sample, theory-first research the be all and end all of a science that has practical relevance? We see a real risk that professional norms overly aligned with open science principles will evolve the culture too far in this direction.

There is increasing decrying of the failure to use the newer sources of data available to both academics and practitioners. The paradox is that there has been a reversion to simpler studies with small samples, as Staw (Reference Staw2016) describes, limiting drastically (a) what appears in the peer-reviewed literature and (b) the relevance to organizational life. At least two factors have yielded this outcome, an outcome being pushed further by the open science movement. First, as King and Persily (Reference King and Persily2019) note in their prescient article on industry–academic partnerships in research, the existing culture for academic researchers has been characterized by the independent scholar who had “unfettered control over their research agenda, methodological choices, and publication options” (p. 1). They note that in the era of big data (they write as political scientists) that this traditional model no longer works if research is to be relevant to the real world. They suggest, as do we, that unless researchers have access to the vast big-data sources that exist, they will increasingly focus on smaller, societally irrelevant issues. The second factor paradoxically pushing the hypothetico-deductive model on smaller samples with fewer variables is the reward system that exists in the academic publishing world where publications in what have come to be called “A” outlets are the only ones that count for promotion. The perfect article is the one that generates the fewest methodological and conceptual issues that can be attacked by reviewers, so the surest way to get a publication is to conduct a study with clear hypotheses using standard, favored methodologies and having as much control as possible over the research process and outcomes. Aguinis, Cummings, et al. (Reference Aguinis, Cummings, Ramani and Cummings2020) thoroughly document the A-journal fetish, worldwide, noting that “there are mounting concerns about unintended negative effects of using A-journal lists to assess research value. Among these deleterious outcomes are questionable research practices, narrowing of research topics, theories, and methods, and lessening research care” (pp. 136–137). Sound familiar? There appears to be an implicit conspiracy fighting against research that is relevant to organizational life. Aguinis, Cummings, et al. reach the strange conclusion that this is not as bad as one might think with respect to the information business leaders have as a basis for making decisions. That is, they note that textbooks and articles in the press use the materials published by consulting firms and companies as sources for insights, not research published in academic journals. In other words, to have information that is relevant to the world of practice, textbook authors and journalists no longer rely on the research published in academic journals. Is this the world of evidence that open science initiatives wish to produce, one in which students and the public are being taught about important issues based on reports and results that have not met usual and typical peer-reviewed research standards?

Moreover, we believe that enthusiasm for open science runs the risk of creating overbearing methodological fashion. Such is evident in economics, where methodological conformity is fully on display in empirical research and that has led to greater concern with the wizardry of technique than with the value and practical relevance of insights generated from data. Labor/organizational economics has long been concerned with understanding phenomena like labor market participation, wage determination, the employment effects of minimum wage laws, human capital development, and the value of education and training, among other topics. Empirical analyses tend to draw on Census data and a relatively few other large national data bases such as the Panel Study of Income Dynamics and National Longitudinal Survey of Youth, surveys that track large cohorts of individuals over time. Scholars repeatedly going to the same well often find themselves relying on econometric dexterity to uncover something new. Another aspect of methodological uniformity in labor and organizational economics is an overreliance on a single type of model specification when making empirical tests, fixed effects models. The culture of conformity to this single “standard” foregoes the benefits of alternative model specifications that take better advantage of the value of big data for both theoretical and practical insights.

Encourage more collaboration between academic- and practice-based researchers

Grand, Rogelberg, Allen et al. (Reference Grand, Rogelberg, Allen, Landis, Reynolds, Scott, Tonidandel and Truxillo2018) offer this recommendation, and we repeat it here for its importance. In addition to the value inherent in combining the two perspectives when designing, analyzing, and interpreting studies, academic-based researchers are likely to see to it that practice-based researchers adopt open science practices wherever possible and are more likely to connect existing theories to research done with “real” databases. More and better publications of practice-based research will result. Such collaborations will foster a constructive culture of science–practice research. In the continued absence of such collaborations, the fault lines between “academic” and “applied” could easily widen because of one side’s advocacy of open science practices that the other side cannot satisfy.

This call for more collaboration is not just the echo of past exhortations. It is a renewal, a response to the center of gravity of data now found in organizations (Guzzo, Reference Guzzo2011). Organizations and their vendors have large volumes of relevant data within easy reach. Sources include HRIS systems (variables related to individual employees’ performance, jobs, experiences, leaders, coworkers, and workplace characteristics), learning management systems (providing records of training and development opportunities taken), communication (e.g., email, SMS records) databases documenting workplace interactions, applicant databases with such data as employment screening and work history variables, employee engagement databases, benefits elections and utilization databases that provide insights into such things as employees’ preferences and their well-being, and the records of routinely collected business outcomes (e.g., sales, customer retention). A great many of these variables take the very same form as they would had they been created specifically for research purposes (e.g., performance ratings in organizational databases often look just like performance ratings created by researchers). Such data, along with the presence of researchers in organizations, are good reasons for collaborating for the sake of better science. Organizational sciences risk being bypassed in relevance if they miss the opportunities being created by the proliferation of digital data on so many aspects of behavior, processes, and contexts.

Adapt the peer-review process to meet objectives of the open science transparency principle but without placing individuals or organizations at risk

Our focus here is on research reports from naturally occurring organizational settings where legal or other prohibitions prevent full disclosure to protect privacy or proprietary interests. The objectives of an adapted review process are to (a) pull back the veil so that a trusted outside party or parties can verify the authenticity of the details of the reported research effort and (b) remove one of the barriers (full disclosure) to publishing. One approach might be to provide a special reviewer (chosen from among acknowledged experts who would be uninvolved in decisions about the specific manuscript), after they sign a binding nondisclosure agreement (NDA), the right to see the original data and to discuss with the researchers details of methods of analysis and other relevant considerations. Checklists and journal policy standards could aid the process. This style of review would protect research integrity and provide a valuable imprimatur for the consumers of published research. Other approaches may exist. The process suggested here is not a review of the proposed article but a review of the methods/procedures/sample, and it does not preclude other forms of blind reviews of the same research report with respect to the possible actual publication value of the research effort itself.

As we prepared this article, and this recommendation, we became aware of a similar proposal for research in political science. King and Persily (Reference King and Persily2019) demonstrate how big data can be so facilitative of the research done in their field but lament the limitations to accessing such data, limitations like those in our field. As they note, “Today, big data collected by firms about individuals and human societies is more informative than ever, which means it has increasing scientific value but also more potential to violate individual privacy or help a firm’s competitors” (p. 2). Their proposed solution to the dilemma of access and privacy is to create two interacting but independent groups, one of academics who wish to do the research and a second of trusted third parties who sign NDAs regarding the details about a firm and its database. The third party “thus provides a public service by certifying to the academic community the legitimacy of any data provided.”

In short, there are tactics available to ensure that the research process and the research data meet the highest possible standards without giving rise to the risks that compliance with open science practices would create. We are also hopeful that journal reviewers are open to the alternatives to hypothetico-deductive models with big-data research reports. Needless to say, processes ensuring the privacy of data have the potential to make the most important data available to academic researchers, thus encouraging the best research on the most relevant possible databases for publication to other academics, to executives who require validated evidence, and to textbook authors and news reporters as well.

Revise how we teach theorizing

Training in the hypothetico-deductive approach is well rooted and closely integrated with training in inferential statistical methods. The advent of big data requires that we add mastery of inductive and abductive methods of theory development to that foundation. As a discipline, we have caught on to the data analysis innovations that come with big data. For example, see the “scenic tour” of statistical methods for big data offered by Oswald and Putka (Reference Oswald, Putka, Tonidandel, King and Cortina2016) and the more recent review by Oswald et al. (Reference Oswald, Behrend, Putka and Sinar2020). Although we may be quick at taking up these new methods, we see a need to increase our collective ability to advance theory from their application to our big-data research. This is where induction and abduction fill the void left by hypothetico-deductive inference. Both of these alternative approaches concern the process of testing and building theory from the integration of multiple findings and fact patterns illuminated by “big models” that facilitate replication and validation, making data analysis an opportunity to learn the unanticipated in addition to testing preexisting theory and, not so incidentally, allow for replication and meta-analysis of findings at the same time.

Do not specifically incentivize the publication of replications, especially exact (“direct”) replications

Journals can incentivize replications by inviting them, by creating special issues devoted to them, and by guaranteeing the publication of results of preregistered, peer-reviewed replication research plans. But if a replication crisis exists, such tactics are likely to magnify it. As we have discussed, attempts to replicate only a subset of extant research reports will predominate: the easy-to-do studies with smaller samples and fewer variables that are free of the constraints encountered in applied settings.Footnote ⁵ As the Center for Open Science recognizes, “journals and [professional] societies shape the incentives and reward structures driving researchers’ behavior” (Center for Open Science, 2020b). Incentivizing replications with the rewards of publication will shape behavior and the research literature in undesirable ways for science–practice disciplines, limit what we learn, limit the variety of methods we will use, and narrow the scope of what defines “good” research (Aguinis, Cummings et al., Reference Aguinis, Cummings, Ramani and Cummings2020).

Embrace conceptual replications and encourage them when big-data sets are available for exploring contingencies

Conceptual replications address previously tested theories and propositions but do so by being different from prior research “in the operationalizations of the phenomenon, the independent and dependent variables, the type and design of the study, and the participant populations” (Crandall & Sherman, Reference Crandall and Sherman2015, p. 93). Crandall and Sherman make the case that conceptual replications deliver far more scientific value than do direct replications. A primary reason for their superiority concerns what is learned when replication studies fail to recreate prior findings. When the usual exact replication fails, it is impossible to discern why: Was it because the theoretical concept is poorly specified or the operationalizations are faulty or because, as in the original study, the sample was too small? Exact replications—no matter how many times they are attempted—do not resolve this ambiguity. Conceptual replications can productively disperse sources of failure to replicate across operationalizations, settings, samples, and so on. Consequently, conceptual replications are far superior for establishing generalizability and identifying why and when theorized relationships hold. In other words, conceptual replications are the preferred way of creating robust research results and theory.

Conceptual replication opportunities abound in big-data research precisely because it enables testing of previously reported findings in settings different from prior studies, with different operationalizations of variables, alternative methods of data collection and analysis, with different participants, with more covariates and boundary conditions examined, and so on. The key is to get reports of such big-data research published. This way a body of robust findings—and replications—will grow quickly, without special inducements, in science–practice domains. Big-data research need not originate only in applied settings; it can occur in simulations and controlled experiments, too. However, we estimate that applied settings are the more likely source of higher volumes of such research.

Affirm the value of preregistration, where appropriate

Preregistration of research plans—including designs, measures, and analyses—is a valuable tool for combatting research ills such as p-hacking and for promoting open discourse early in a research project. However, it is unworkable for many research efforts, especially for research in applied settings and not only for inductive or abductive work. That is, although still rare, there is a growing literature of field experiments (Eden, Reference Eden2017) contributing to our research. On the surface, such field experiments would seem to fulfill open science principles given the focus of them on specific issues. However, companies may be averse to revealing who they are, who participated in the research, and which proprietary measures and procedures are used. Also, it is truly unrealistic to require researchers such as Illingworth et al. (Reference Illingworth, Lippstreu, Deprez-Sims, Tonidandel, King and Cortina2016) to fully preregister the over 125 variables used in their selection research or the 135 variables of Erland et al.’s (Reference Erland, Gross and Guzzo2017) analysis of turnover. Aguinis, Banks et al. (Reference Aguinis, Banks, Rogelberg and Cascio2020), publishing in a journal with a primarily academic rather than practitioner audience, state that it requires “30–60 min of authors’ time to preregister” (Table 1; p. 29). This is out of touch with the realities of multivariate, big-data research in applied settings. Beyond the mere burden of describing what could be well over a hundred variables, a task that constitutes only one small portion of what is called for in preregistration, research in applied settings such as business organizations only sometimes can be fully preplanned. By necessity, such research efforts, especially when oriented to finding solutions to pressing problems, often must respond to changing circumstances, take advantage of data or samples that become available midstream, or change tactics to respond to restatements of the problem driven by the results of initial analyses. These points are fairly straightforward and evident, but they can result in finished research projects that look quite different from initial plans. The concern we have is that in the advocacy of preregistration as a tactic for all there is too little detailed explication of when the tactic is ill-suited and when its enforcement will discourage very good practice-focused research from entering the research literature. Give credit for preregistering a study when it is useful and supportive of scientific advancement, but do not discredit research when it is appropriately not preregistered. These latter cases, of course, would benefit from our previous argument about having NDAs for reviewers and/or a panel of senior academics to review methods and analyses without the necessity of researchers and organizations bearing the risks of full disclosure.

Preregistration as a tactic impinges on the true nature of scientific advancement and the role of unexpected discoveries, especially when studying social systems such as organizations. This is nicely illustrated by the distinction in economics research between partial and general equilibrium contexts. Theory-driven research works best in what economists call a partial equilibrium context. In that context, narrowly defined relationships that can be easily be expressed mathematically and isolated in the data are tested. It is decidedly not good for addressing general equilibrium outcomes, those that reflect the simultaneous operation of complex interactions among system components. It is very hard to prespecify theoretical propositions about those interactions where often there are inflection points in specific system components and nonlinearities that can produce dramatic changes in the characteristics of the system as a whole and of the behavior of individuals and groups in those systems. Where those inflection points and nonlinearities come into play is an empirical question neither easily predicted from theory nor easily described in advance. From this perspective, then, current preregistration practices appear more to serve the advancement of marginal findings than of transformational findings.

Affirm the value of sharing, where appropriate

Sharing of data, code, methods of analysis, materials, and so forth, like preregistration, is a valuable tactic but as noted an often-impossible standard to meet for work done in and between organizations. As with preregistration, encourage and support it when possible but do nothing to devalue research that is constrained from sharing data or fully disclosing proprietary methods and procedures. It is easy to imagine a research community’s stealthy development of norms that “good” research is that which complies with tactics rooted in the principles of open science and, by implication, noncomplying research is less trustworthy. Vigilance against such a development is required because of the potential harm to scientific advancement in science-practice disciplines.