1. Robust Science Is Relevant

There is general agreement among economists and public policy experts that investment in basic and applied research is a significant contributor to and distinguisher of socioeconomic health and success across the global landscape (cf., Fagerberg, Reference Fagerberg1994). Although difficult to directly quantify the manner by which science leads to such improvements, the fundamental premise is simple: The knowledge, understanding, and expertise generated through the pursuit of science allow scholars and practitioners to address complex problems that benefit our lives in ways big and small (Pavitt, Reference Pavitt1996). For example, the science of I-O psychology involves generating knowledge and descriptions of the psychological experiences and social interactions that unfold within and between individuals and the organizations in which they work. The accuracy and usefulness of the knowledge generated by a discipline broadly characterizes its relevance—the degree to which science and practice improve understanding of the world, address important needs and problems of our time, and/or contribute to outcomes that benefit humanity and society (Allen, Reference Allen2015). Certainly not every scientific endeavor will (nor should) result in an immediately cohesive explanation, applicable technological advancement, policy reform, or cultural improvement. However, when these are the goals that spur science, both the producers and recipients of scientific knowledge are more likely to benefit. Consistent with these sentiments, we propose that a robust science is one in which activities throughout the entire scientific enterprise are conducted with the intention of producing positively impactful and relevant knowledge.

2. Robust Science Is Rigorous

The rigor of a science is reflected in the extent to which its core concepts and their relations are operationalized with precision, and the methodologies used to collect informative observations are accurate and appropriately aligned with the analytical techniques used to infer meaning from those observations. The technologies and methodological/analytical toolboxes available to I-O psychologists for accomplishing these tasks in both research and practice are more extensive and richer than ever before. Furthermore, there is growing appreciation of the unique insights that a wider breadth of approaches can bring to understanding the phenomena that we study (i.e., triangulating on a phenomenon from multiple perspectives, compensating for weaknesses among particular methodologies; Cortina, Aguinis, & DeShon, Reference Cortina, Aguinis and DeShon2017). More methodological freedom, of course, also places a greater onus on the producers, reviewers, and users of scientific knowledge to be informed of the strengths and inevitable weaknesses that choices among different philosophical (e.g., induction, deduction, abduction), methodological (e.g., self-report, behavioral, physiological), and analytical (e.g., quantitative, qualitative) perspectives hold for advancing scientific understanding. We therefore advance that a robust science is characterized by choices that favor rigor over expedience and values leveraging diverse investigative tools that align with the research questions being examined.

3. Robust Science Is Replicated

Recent discussions and headlines on the replicability/reproducibility crisis within science (Baker, Reference Baker2016) and the psychological sciences in particular (e.g., Carey, Reference Carey2015; Yong, Reference Yong2016) have raised many concerns and anxieties. Large-scale initiatives that have attempted to gather evidence on the replicability of research findings across a variety of topics have presented many sobering statistics on the extent to which the results of published research are reproducible (Ioannidis, Reference Ioannidis2005b; Klein et al., Reference Klein, Ratliff, Vianello, Adams, Bahnik, Bernstein and Nosek2014; Open Science Collaboration, 2015). These results have stimulated heated arguments about the validity of such metascience efforts and the conclusions they imply (see Anderson et al., Reference Anderson, Bahnik, Barnett-Cowan, Bosco, Chandler, Chartier and Zuni2016; Gilbert, King, Pettigrew, & Wilson, Reference Gilbert, King, Pettigrew and Wilson2016). However, similar to the inadequacy of only using p-values and statistical significance tests to determine whether scientific knowledge is “supported” versus “not supported” (Cohen, Reference Cohen1990; Schmidt, Reference Schmidt1996; Simmons et al., Reference Simmons, Nelson and Simonsohn2011), efforts to determine the replicability of findings can be similarly flawed when solely directed toward establishing a (false) dichotomy between scientific knowledge that “replicates” versus “fails to replicate” and debating the arbitrary criteria used to justify that distinction.Footnote ² The more critical significance of replication efforts is that they facilitate the collection of large bodies of evidence upon which scientific knowledge is based and refined. Consequently, we propose that a robust science is not one that seeks to establish the replicability of its findings per se but instead is one that values and pursues efforts to gather repeated (i.e., replicated) observations of the mechanisms and relationships among core concepts and processes of human behavior, and that these efforts are made accessible in the corpus of scientific evidence. Such practices are critical in order to establish the degree of confidence that should be placed in scientific claims and to support evidence-based practice.

4. Robust Science Is Accumulative and Cumulative

Despite the staggering number of scientific works published every year (National Science Board, 2016), many have maligned that much of this growth reflects that science is “fattening up” rather than “growing up.” That is, there is concern that the emphasis on exciting but untested theory that posits discontinuous change in current thinking is contributing to scientific bloat, whereas efforts to test and build upon existing explanations in ways that gradually refine current thinking languish in obscurity (e.g., Antonakis, Reference Antonakis2017; Colquitt & Zapata-Phelan, Reference Colquitt and Zapata-Phelan2007; Hambrick, Reference Hambrick2007; McKinley, Reference McKinley2010). There is clearly a delicate and difficult balance to strike between promoting scientific pursuits in ways that foster incremental progress (e.g., reliance on widely used and tightly controlled paradigms) and encouraging “disruptive” thinking that challenges the status quo with new ideas (Nosek et al., Reference Nosek, Spies and Motyl2012; Pfeffer, Reference Pfeffer1993). Indeed, it is difficult to envision how a field can remain impactful and credible when it favors one end of this continuum too strongly over the other. Science benefits when both incremental research and novel idea generation occur. Consequently, new ideas and ways of thinking should not be deterred, but we must also recognize that the strength of scientific understanding and inference is enhanced through careful vetting, deliberate calibration, and compounding multiple observations into an integrative whole. We thus propose that a robust science is one in which the pursuit of cumulative knowledge is reinforced by adopting an appropriate degree of intellectual skepticism toward novel propositions and appropriately adjusting those beliefs on the basis of accumulated evidence.

5. Robust Science Is Transparent and Open

Although we all contribute to and participate in multiple aspects of the broader scientific enterprise, a surprising amount of the system remains opaque. For example, authors are often the only ones who know whether time and effort was devoted to examining data supportive of their hypotheses, whereas less supportive data were relegated to the “file drawer” (Rosenthal, Reference Rosenthal1979). Editors and reviewers only evaluate a summary of the methodological and analytical decisions made in a research study rather than the full original source materials (Nosek et al., Reference Nosek, Spies and Motyl2012). End consumers of research typically never see nor have access to the exchanges, disagreements, and revisions that occur during the review process that ultimately shape a manuscript (Nosek & Bar-Anan, Reference Nosek and Bar-Anan2012). Practitioners or researchers at institutions with limited resources often lack access to research disseminated in scientific journals because of high publisher subscription costs (Houghton & Oppenheim, Reference Houghton and Oppenheim2010). As a result of such gatekeeping actions, very few (if any) individuals in a scientific community ever have full knowledge of “how the scientific sausage gets made.”

Many of the norms that guide the dissemination and communication of research products throughout the scientific enterprise emerged and persist for reasons that are valued, logical, and well-intentioned. Nevertheless, human history has taught us that increasing access to knowledge can be transformative on a grand scale, and organizational research has taught us that knowledge about the distributive, procedural, and interactional decisions made in a system is empowering and tends to promote positive behavior (Greenberg & Colquitt, Reference Greenberg and Colquitt2005). As such, we consider a robust science one in which transparency and openness are embraced throughout the research process and scientific system. Activities that embrace these principles include more complete disclosure of data, materials, analyses, and hypotheses to the scientific community; promoting publication practices in which important questions answered well have a place in the literature regardless of results; and creating accessibility to the research process at all stages of production. We recognize there are challenges with data transparency and openness from corporate and private settings (e.g., Gabriel & Wessel, Reference Gabriel and Wessel2013; Jones & Dages, Reference Jones and Dages2013). However, we wish to challenge I-O psychologists working in these settings to push for what can be shared with appropriate protections in place. Too often we fear the default is to not even engage in conversations about data sharing with clients or with I-O psychologists working internally. Ultimately, a robust science is one in which transparency across the scientific enterprise is the professional expectation and at the very least strived for in all that we do.

6. Robust Science Is Theory Oriented

In a manuscript discussing the limitations of quantum theory as the foundation for physics, Albert Einstein noted, “The whole of science is nothing more than a refinement of everyday thinking” (Reference Einstein1936, p. 349). In this, Einstein reflects the notion that a critical goal of science is to take observations from our environment and transform them into accurate and useful explanations (i.e., theories) of how and why phenomena in the natural world occur (see also Feynman, Reference Feynman1969). The development of accurate, useful, and evidence-based explanations is a distinguishing feature of all sciences. Consistent with many recent discussions on the role of theory in the organizational sciences (e.g., Antonakis, Reference Antonakis2017; Campbell & Wilmot, Reference Campbell, Wilmot, Anderson, Ones, Sinangil and Viswesvaranin press; Colquitt & Zapata-Phelan, Reference Colquitt and Zapata-Phelan2007; Hambrick, Reference Hambrick2007; McKinley, Reference McKinley2010; Ones, Kaiser, Chamorro-Premuzic, & Svensson, Reference Ones, Kaiser, Chamorro-Premuzic and Svensson2017; Spector, Rogelberg, Ryan, Schmitt, & Zedeck, Reference Spector, Rogelberg, Ryan, Schmitt and Zedeck2014), we advance that a robust science entails treating theory as a means to an end, not an end in and of itself. That is, theory for theory's sake is not rewarded, and resistances against or devaluing of applied, inductive, and exploratory research are removed. Instead, a robust science is simply one in which its scientific pursuits contribute to explanation and “refinement of everyday thinking” by replicating, bounding, revising, falsifying, and, when appropriate, advancing new claims. By this definition, we propose that robust science is “theory oriented” (not theory driven or theory dependent) and promotes this tenet by describing, evaluating, and refining explanations. Genuinely accomplishing this goal requires research that reflects quantitative and qualitative methodologies across the full range of inductive, deductive, and abductive approaches.

An important addendum to this principle is the recognition that theory in robust science is not an informed hunch, logical speculation, or collection of plausible relations—it is “a comprehensive explanation of some aspect of nature that is supported by a vast body of evidence” (National Academies of Sciences & Institute of Medicine, 2008, p. 11). Consequently, a theory-oriented robust science is one in which explanations and empirical research emphasize precision. As a general definition, scientific precision reflects the degree to which explanatory mechanisms are specified with sufficient detail such that predictions/explanations regarding the specific magnitude, form, processes, and conditions that account for a set of relations can be advanced and tested (Edwards & Berry, Reference Edwards and Berry2010; Kozlowski, Chao, Grand, Braun, & Kuljanin, Reference Kozlowski, Chao, Grand, Braun and Kuljanin2013). Note this definition does not require every aspect of a theory to be correct to be precise. The rightness of a theory is not determined by the clarity of its arguments or through formal logicFootnote ³ but by subjecting its claims to the gauntlet of empirical investigation. A science that strives for precise theories purposefully subjects its explanations to an increased “risk” of falsification to determine the level of confidence that should be placed in proposed relationships (Meehl, Reference Meehl1967). These higher evidentiary standards are commonplace in more well-developed areas of science, such as physics and chemistry, but are still lacking in the social sciences (Fanelli, Reference Fanelli2010a). We thus submit that a theory-oriented robust science is one in which its explanatory accounts strive for the highest degrees of precision.

In sum, the culture of the modern scientific enterprise in which I-O psychology resides presents unique challenges for enacting changes that foster healthy and credible science. However, we believe that making progress toward these ideals can be facilitated through establishing common ground among members of our scientific community about the meaning of robust science and its foundational aspirations. To this end, we advanced six defining principles of a robust science (summarized in Table 1); specifically, a robust science is (a) relevant, (b) rigorous, (c) replicated, (d) accumulative and cumulative, (e) transparent and open, and (f) theory oriented. Endorsing a shared vision of robust science represents a commitment to promoting and reinforcing activities that emphasize positive scientific behaviors. More important though, it can also serve as a means for guiding and coordinating collective action that incrementally contribute to these changes. It is to this point that we now turn our attention.

Table 1. Summary of the Defining Principles of Robust Science

Authors

Authors are the primary generators and communicators of the research products in a scientific field. As the first point of contact with the collection, creation, and interpretation of inductively driven ideas and deductively guided empirical data, authors give voice to the explanations that are advanced for phenomena observed in the natural world. Consequently, authors are arguably the most influential source of bottom-up change in the scientific ecosystem for fostering robust science. We believe that the core responsibilities authors hold for fostering a robust science can be categorized into two broad sets of activities: (a) accurately, honestly, and clearly reporting all activities used in the production of scientific research; and (b) prioritizing publication practices that embody the defining principles of robust science. These responsibilities span many specific actions encompassing both the research and publication processes, but below we highlight three activities perceived as particularly critical.

First, authors should accurately describe and represent the capabilities of their methodologies and analyses when proposing inferences from their research. Science is not—and can never be—perfect. To borrow from McGrath's (Reference McGrath, McGrath, Martin and Kulka1982) discussion of dilemmatics, research inevitably involves decisions that create tradeoffs among the precision, generalizability, and realism of results. In our field, it is common for authors to acknowledge these choices near the end of a manuscript when outlining the limitations of a research study. These sections have largely become placeholders for obligatory statements such as “the research should be tested using a different sample” or “the cross-sectional design does not allow for causal inferences,” which authors then feel motivated to defend to lessen the blow of their choices on conclusions they wish to draw. The role of an author in a healthy scientific ecosystem is not to convince the reader of how “exceptional” their study is, nor to generate a list of possible flaws and then systematically rebuke them. Instead, the author's responsibility is to openly acknowledge the extent to which the methods and analyses used in their research do and/or do not allow them to draw particular conclusions. This goal should be reflected throughout a manuscript rather than only after findings have been presented, including when hypotheses are developed (e.g., justifying hypotheses that do not propose the form, magnitude, or conditions for an effect; Edwards & Berry, Reference Edwards and Berry2010), methods and analyses are described (e.g., conveying extent to which a measure/technique permits one's predictions to be examined, degree to which results are robust to other plausible analytical choices; Simmons et al., Reference Simmons, Nelson and Simonsohn2011), and inferences are advanced (e.g. not overgeneralizing implications beyond the data; Hollenbeck & Wright, Reference Hollenbeck and Wright2017).

Second, authors should be precise and transparent when describing the development of research questions and the rationale for the methodological/analytical choices used in their research. Much has been written concerning the statistical and epistemological concerns of HARKing and similar detrimental research practices (Banks et al., Reference Banks, O'Boyle, Pollack, White, Batchelor, Whelpley and Adkins2016; Hitchcock & Sober, Reference Hitchcock and Sober2004; Hollenbeck & Wright, Reference Hollenbeck and Wright2017; Kerr, Reference Kerr1998). The specifics of those arguments are beyond the scope of this discussion, but their root cause is a lack of transparency from authors concerning conceptual rationale, methodological decisions, and analytical choices. Whether one identified a meaningful relationship after (faithfully) hypothesizing a finding a priori or it was identified post hoc does not change the fact that the observed relationship was present in the data. The natural world could not care less whether the scientist made an accurate prediction beforehand or not—but the manner by which scientific inference advances does. The attributions, interpretations, and implications that authors advance about their observations should change based on the process used to generate them. When a conceptual rationale is genuinely new, deviates from previous explanations, or is based on small sample sizes, it is important (and acceptable!) for authors to present interpretations from those studies with a healthy dose of skepticism. Similarly, it is important (and again, acceptable!) to openly describe the pros and cons of the methodological or analytical decisions made by the authors to pursue a scientific explanation. The goal of authors in a robust scientific ecosystem is to improve explanations and descriptions of how and why things in the natural world work the way they do. The ability to achieve this goal is significantly impaired when precision and transparency in conceptual rationale and methodology are lacking (Simmons et al., Reference Simmons, Nelson and Simonsohn2011).

Last, authors should commit to enacting the defining principles of robust science even when they are not rewarded for doing so. There are numerous avenues through which authors can attempt to meet the values of robust science. For example, authors can show a preference toward journals that actively reinforce robust science principles; make a point to have colleagues from both practice and academia collaborate and review manuscripts to engage multiple stakeholders in the research process; make materials, data, and/or analyses openly available when feasible; and place accepted prepublication manuscripts in open-access repositories. Whatever the ways in which these values could be fulfilled, they share the characteristic of being outside the current status quo and the route through which status and recognition are typically achieved in the current publication enterprise (Nosek et al., Reference Nosek, Spies and Motyl2012; Nosek & Bar-Anan, Reference Nosek and Bar-Anan2012). These activities feel like extra work with little direct payoff to authors—because that is precisely what they are at present. However, a vibrant ecosystem can only be sustained if authors collectively agree that “what's good for the science” is at least as important and helps to achieve “what's good for the researcher.” Ironically, this situation shares many of the same characteristics with classic social dilemma and game theoretic scenarios that organizational scientists have studied for decades (Byington & Felps, Reference Byington and Felps2017). The widely replicated inference from that line of research is that when individuals choose not to engage in socially beneficial behaviors, everybody loses in the end. If our field is to achieve the ideals of robust science, then all authors must contribute to the system they wish to create.

Editors

Editors-in-chief and associate editors of scientific journals (hereafter collectively referred to as editors) represent the most significant gatekeepers of published research in the current system of scientific communication (Nosek & Bar-Anan, Reference Nosek and Bar-Anan2012). Editors possess a number of avenues for influencing the quality and type of research published (Zedeck, Reference Zedeck, Baruch, Konrad, Aguinis and Starbuck2008, Reference Zedeck2017). Editors’ actions at a journal are generally perceived as communicating the requirements, criteria, and beliefs about what constitutes high quality and valued research. Furthermore, individuals also typically reach editorial positions because of their career successes and thus tend to be respected and widely connected figures in the broader science network. Editors thus possess multiple bases of power (e.g., reward, legitimate, referent, expert, and informational power; Raven, Reference Raven1992) that make them instrumental levers for stimulating change in the scientific enterprise. We highlight a number of recommended activities in which we believe editors in I-O psychology and the managerial sciences should engage to foster robust science.

First, editors should establish review and reviewer norms consistent with robust science. To have a robust science, compelling research questions answered in methodologically sound ways should be published regardless of the actual pattern of results. We believe editors should explicitly communicate to their editorial teams and editorial boards that publication recommendations be made based on two principal factors: (a) the theoretical, conceptual, and practical relevance of the research questions explored and (b) the rigor of the methodology/analyses used to answer those questions. Along with this, editors should strongly communicate continuously to reviewers the importance of being open to scientific advancement through both inductive and deductive research, and that a “perfect study” (i.e., all hypotheses fully supported, all a priori theoretical rationale completely consistent with observed data, etc.) is not a requirement for publication per se (Simmons et al., Reference Simmons, Nelson and Simonsohn2011). After all, null findings found in statistically and methodologically powerful ways are essential to the advancement and self-correction of science (Landis, James, Lance, Pierce, & Rogelberg, Reference Landis, James, Lance, Pierce and Rogelberg2014). This type of very frequent messaging facilitates alignment of the expectations and values expressed by reviewers.

Second, and very importantly, editors should provide regular and constructive feedback to reviewers. In many ways, editors represent the chief talent officers of their journals—they seek out reviewers they think will provide high quality services, wrestle with strategies for balancing reviewer workload, and are constantly looking for ways to diversify and maintain a productive crop of reviewers. However, one activity we believe could occur more frequently is provision of feedback and training opportunities to reviewers to help shape and/or correct behaviors in ways that promote robust science. For example, a reviewer who suggests dropping null results or creating new hypotheses after results are known should be informed by the editor of the inappropriateness of this recommendation and asked to revise the comment and/or signal in the decision letter that this action is inadvisable. Reviewer feedback and development by the editorial team not only plays an important role in facilitating more productive reviews but also signals that the practices of robust science are desirable and should be applied when reviewers adopt authorship roles with their own independent research. Editors must recognize their unique responsibilities in assuring that reviews and decision letters from their journal do not implicitly or explicitly encourage questionable research or work counter to robust science. If time for the above essential activities appears nearly impossible to find, editors must work strongly, in concert with their publishers, to expand the editorial team and/or streamline processes to recover time. A lack of time cannot be a reason to abdicate robust science responsibilities.

Third, editors should experiment with enacting nontraditional practices at their journal designed to promote robust science. Editors work hard to maintain fair and balanced practices that ensure the “rules of the game” implicitly and explicitly promote and result in dissemination of high quality research. Although the current system has been successful on a number of fronts, there are many new and emerging ideas about how these practices might be altered to better engender behaviors that facilitate robust science. Some visions for these practices represent novel and idealistic changes that would require substantial shifts in editorial infrastructure and journal management (e.g., Nosek & Bar-Anan, Reference Nosek and Bar-Anan2012). Others though are much more moderate in scope (e.g., Nosek et al., Reference Nosek, Alter, Banks, Borsboom, Bowman, Breckler and Yarkoni2015). For example, attempts by editors to introduce results-blind submission paths, explicit journal sections welcoming replications, and options for authors to preregister studies are relatively straightforward, and good models for these practices are emerging. Journal editors could encourage authors to host their data, materials, measures, additional analyses, previous manuscript versions, review letters, and other materials in accessible and online repositories (such as the Open Science Framework, http://osf.io) to promote increased transparency and accumulation of science. Additionally, editors could pursue and reward regular reviewer training to promote an improved review process. In essence, editors should not only strive to be a steward of their journal but to also be a steward of science and use their unique role to experiment with practices that can make meaningful improvements for the scientific enterprise.

Last, editors should evaluate and, where appropriate, revise journal management practices to ensure they are consistent with advancing the principles of robust science. Beyond the many tasks that editors perform related to promoting the rigor and quality of research published in their journals, editors are also the captains and caretakers of a journal's reputation while they are at the helm (Hollenbeck, Reference Hollenbeck, Baruch, Konrad, Aguinis and Starbuck2008). Consequently, engaging in thoughtful and diligent reflection on editorial leadership practices reflects an important piece of the puzzle. For example, much has been written about the importance of impact factors and “citability” as indicators of journal quality and editorial stewardship (e.g., Byington & Felps, Reference Byington and Felps2017; Ireland, Reference Ireland2008). Although impact factors convey useful heuristic information about journal quality, narrowly focusing on them can encourage “gaming” behaviors (e.g., coercive citations) and discourage experimentation with interventions that may be good for robust science but potentially poor for impact factors. Embracing a multifaceted perspective on journal quality that extends beyond impact factors to characteristics such as engagement in open science, humanity in the review process, responsiveness, timeliness, and stakeholder satisfaction should also be considered. In addition to managerial practices focused inwardly on one's own journal, editors also play a key role as a liaison with publishers and other journal editors within and outside our specific discipline. Advocating for publication models that provide greater access to science and engaging in cooperative cross-journal initiatives that collectively promote positive scientific behaviors should be normative editorial behaviors.Footnote ⁴

Professional Associations

There are typically multiple professional associations within any given scientific community. Some are small and regional; others are large and international. Some exist primarily to host conferences, and others provide certification, publishing, advocacy, and research support services. Irrespective of their differences though, all professional associations share the feature of promoting a common identity and network for members of a scientific community. As a result, professional associations provide a number of unique resources and opportunities for advancing initiatives to support robust scientific practices.

First, professional associations should publically endorse, openly advocate for, partner with, and reward initiatives intended to advance the defining principles of robust science. Whereas scientific journals and the publication review process implicitly communicate normative expectations about robust science, professional associations have the opportunity to communicate these values more directly and broadly. This activity should be seen as a core function of any scientific professional association, as it underlies the credibility of the field, the credibility of the academy, and the credibility of practice. Both the Society for Industrial and Organizational Psychology (SIOP; http://www.siop.org/mission.aspx) and the Academy of Management (AOM; http://aom.org/About-AOM/Vision,-Mission,-Objectives-Values.aspx), the two largest professional associations in our field, provide direct statements about the values, mission, and objectives held by their associations. We believe these mission statements could be updated to include declarative commitments to explicitly foster robust science. To be sure, both of these groups have engaged in some limited activities toward these ends. For example, in 2008 SIOP leadership entered into an Alliance for Organizational Psychology (AOP) with Division 1 of the International Association of Applied Psychology (IAAP) and the European Association of Work and Organizational Psychology (EAWOP). Eight years later, the first AOP summit meeting was held and a memorandum of understanding was established that identified several potential problems that may undermine the rigor, ethics, and relevance of our scientific enterprise and offered potential solutions for addressing these issues (Grote, Reference Grote2016). Although it is still too early to know what outcomes and initiatives may arise from these and other similar actions, momentum and commitment to achieving the goals of robust science can be improved through direct endorsements, declarations, and advocacy by professional associations. In other words, we need our key professional associations to make robust science a top priority that is strongly addressed year in and year out.

Second, professional associations should provide resources for developing and implementing tools, techniques, and systems that encourage robust scientific practices. Openly supporting the values that foster robust science is an important step to promoting a healthy culture of science. However, change will only happen with boots on the ground. Professional associations can make a difference by dedicating time and monetary resources to members for instituting change. For example, the task force on Robust and Reliable Science (to which the authors on this article belong) requested and received financial support from both the SIOP Executive Board and Scientific Affairs Committee to back its efforts. In collaboration with the Executive Board and the SIOP Administrative Office, the 2017 SIOP Program Committee was able to create a new submission track and infrastructure allowing SIOP conference attendees to share data, analysis code, and papers presented at a session or poster with other members (see also http://www.siop.org/rr/). The allocation of resources to robust science activities creates accountability on the part of advocates (i.e., members) and supporters (i.e., professional associations) for making positive changes. We believe there are many additional initiatives where the commitment of resources by professional associations would be beneficial, such as (a) efforts to develop competency models, education, and certificates of training for reviewers; (b) repositories for training and/or standardized applications for using open-source statistical programs that promote analytical transparency; (c) introductory training and incentives for engaging in open science practices (e.g., through Center for Open Science, https://cos.io/); (d) using conferences and workshops as platforms for teaching open science elements to members; (e) developing and managing online indexed open-access journals/reports that can become locations for replication research; and (f) providing small grants and seed money for initiatives designed to promote robust science (e.g., “study swap” and other similar initiatives that encourage data sharing/partnerships between academics and practitioners, creation of a “measures chest” or similar repository for sharing materials to facilitate replications). We recognize that the success and implementation of any such activities will be conditioned on the determined actions of individual members and collaborations within a scientific discipline. However, these bottom-up changes can be greatly accelerated when support and resources are provided by institutions and professional associations that align with and value these goals.

Practitioners

I-O psychology is based on a scientist–practitioner model in which the knowledge generated through science ideally informs practice, and the front-line needs faced in practice ideally inform science. Although debates over the extent to which our discipline has been successful at (or is even capable of) achieving this vision are common (e.g., Rupp & Beal, Reference Rupp and Beal2007), those who apply the evidentiary base of organizational science to address the concerns, problems, and goals of organizations and workers represent integral members of the scientific enterprise. Depending on their roles and activities, practitioners may assume many of the same responsibilities as authors, reviewers, editors, consumers, and participants for fostering robust science. In addition to the recommendations advanced for those groups, we posit two unique actions through which practitioners can further contribute to improving the scientific ecosystem.

First, practitioners should place greater value on practices whose evidentiary base exhibits characteristics of robust science. Just as one places greater confidence in empirical research knowledge generated by robust scientific practices, the application of knowledge that is supported by robust science to create tools, services, and solutions will be more generalizable and reliable. Practitioners who make use of the knowledge produced in I-O psychology to address issues faced by organizations are responsible for interpreting the appropriateness of research and the evidentiary base they use to support their practices. In some respects then, empirical and conceptual work that adheres to the defining principles of a robust science creates a competitive advantage for practitioners, as the degree of uncertainty around the efficacy of practices supported by that work should be reduced. Furthermore, practitioners supporting and more actively applying knowledge that is transparent, replicated, precise, and rigorous can also provide motivation for researchers to engage in these practices to improve the relevance of their primary research. Displaying preference for robust science by relying on sources of well-established understanding benefits practitioners and simultaneously helps advance the relevance and rigor of the field's scientific base.

Second, practitioners should collaborate with academic researchers (and vice versa) to broaden the range of settings, data sources, and interpretations present in the scientific knowledge base. The call to bridge the science–practice divide is not a new one (e.g., Latham, Reference Latham2007; Rynes, Reference Rynes2007; Rynes, Giluk, & Brown, Reference Rynes, Giluk and Brown2007). Many of these appeals cite the importance of shrinking this gap so that scientists and practitioners are aware of the latest developments/needs and to stimulate new directions for empirical study. Perhaps more fundamentally though, partnerships between scientists in academic settings and scientist–practitioners are among the most direct and straightforward methods available for strengthening the rigor, precision, and relevance of our evidentiary base—an outcome that benefits both the accumulation and application of science. Through sharing measurement practices, collaborating with researchers on field studies, and making the results of practice applications more accessible and available to the scientific community, practitioners represent a potent force for challenging the generalizability of research and theory. Academic–practitioner collaborations thus do more than keep the cutting edge sharp; they serve as an essential barometer and foundational bedrock for determining the degree to which scientific findings may be considered legitimately robust. Although such partnerships are likely to face additional hurdles with respect to transparency of methods and data due to concerns over intellectual property, confidentiality, and legal protections, we do not see these challenges as insurmountable. Practitioners frequently conduct research during the implementation and evaluation of organizational programs and should thus be similarly documenting their research methodology, sample characteristics, and statistical procedures. The steps taken to produce the results of their analyses (e.g., data exclusion criteria, scale construction, etc.) should also be documented to enable accurate interpretation of findings and limitations due to sample conditions. Whenever possible and appropriate, efforts to share data and methods associated with research application and extensions should be made available across the profession. This goal is facilitated through collaboration among scientists who work across various settings (e.g., academic or field-based practice).

Educators

Science is a continual learning process. Many in academic and practice settings facilitate this goal through the provision of both formal and informal instruction, guidance, and mentorship. The most familiar of these educational roles is found in universities where professors provide training to both undergraduate and graduate students on the principles of I-O psychology. However, educational roles span a variety of contexts, including professional development workshops, developing online tutorials, and working as project managers. Educators and mentors across the board thus play a pivotal role in shaping and transmitting norms for robust science through teaching and modeling positive scientific practices. We suggest two actions in which educators should engage to promote principles of robust science.

First, educators should incorporate and emphasize the significance of the defining principles of robust science into explicit educational objectives. Virtually all undergraduate and graduate education in I-O psychology and related sciences exposes students to fundamental issues related to the scientific method, research design, validity, and research ethics. In our experiences, the primary focus of instruction on these topics is most often directed toward discussing tradeoffs among methodological or analytical choices (in the case of research methods) and issues surrounding the protection of research participants (in the case of research ethics). Both of these content areas are unquestionably important, relevant, and should continue to be emphasized as part of formal training. However, we believe these educational efforts should be supplemented with instruction on the principles of robust science and the responsibility that scientists have for ensuring such criteria are met. For example, discussing the meaning and identification of research misconduct and detrimental research practices, and why they should be avoided; understanding the difference between HARKing versus alternative forms of inference and their implications for advancing science (e.g., Hitchcock & Sober, Reference Hitchcock and Sober2004; Hollenbeck & Wright, Reference Hollenbeck and Wright2017); understanding the difference between imprecise and precise theory (Edwards & Berry, Reference Edwards and Berry2010; Kozlowski et al., Reference Kozlowski, Chao, Grand, Braun and Kuljanin2013); and describing the significance and means for conveying transparency in one's methodological and analytical choices (e.g., Simmons et al., Reference Simmons, Nelson and Simonsohn2011) all reflect topic areas for which formal educational objectives for training would foster robust science. The same is true for educational initiatives that occur outside of student education as well (e.g., ensuring that learners in a workshop on research methodologies or analyses understand how to apply new techniques in ways that promote transparency, rigor, and precision). We recognize that these topics are often discussed idiosyncratically as part of the mentorship system in science education; however, we believe there is great benefit in more systematic and consistently delivered educational instruction on these topics.

Second, educators should actively demonstrate, model, and provide learners with opportunities to engage in robust scientific practices. Research has reliably demonstrated that the retention and transfer of learning is greatly facilitated when training and education provide clear models for how to enact desired behavior and provide learners with opportunities to practice and receive feedback on these skills (e.g., Bell & Kozlowski, Reference Bell and Kozlowski2008; Taylor, Russ-Eft, & Chan, Reference Taylor, Russ-Eft and Chan2005). In other words, learning is facilitated through showing and doing, not just telling. Because a great deal of education in I-O psychology occurs through individualized mentor–mentee or supervisor–supervisee relationships, it is particularly important that robust scientific practices be modeled and reinforced during collaborative work with junior learners. Additionally, opportunities to practice behaviors conducive to robust science should be made available to learners. There are many activities that could be used to satisfy these goals: engaging learners in the scientific review process (e.g., sharing reviews received and generated by the educator, “co-reviewing” a paper with the learner); exploring tools and emerging products that facilitate open science (e.g., using open source statistical programs such as R, using repositories such as the Open Science Framework for theses/dissertations); and promoting efforts to replicate existing research alongside new/ongoing research all represent avenues through which educators can enact and engage learners in robust science.

Publishers

The publishing industry is experiencing a shifting economic landscape. On the one hand, publishers reap tremendous profit from the current business model supporting scientific publishing (Houghton & Oppenheim, Reference Houghton and Oppenheim2010). On the other hand, business models have changed dramatically in recent years as the availability of electronic platforms continues to grow. One need only look at the impact of online reporting outlets on the newspaper and magazine industry to find examples of the rapid demise of once prominent publications and the dissolution of business enterprises that persevered within the traditional business model.Footnote ⁵ The scientific publishing industry is susceptible to similar patterns, though change has been slower. Revenue forecasts for scientific and technical journal publishers have shown flat growth in recent years (e.g., Ware & Mabe, Reference Ware and Mabe2015). Open-access vehicles are rapidly growing yet still capture only a small portion of the overall scientific and technical research output market (Neylon, Reference Neylon2013). With continued pressure on scientific publishers to compete with and adopt more open access practices, it seems likely the economics of publishing will favor the move toward digital formats. Nevertheless, so long as the current publication model remains advantageous to revenue generation from the sale of journal subscriptions (primarily to research libraries supported with public funding), the current publication model will remain steady. Consequently, we believe that publishers play a critical role in fostering robust science.

Most notably, publishers should recognize and be proactive in counteracting risks associated with the traditional publishing model that contribute to the prevalence of detrimental research practices. In theory, publishers should have a vested interest in the quality of the content they produce. If the status of published science erodes as a result of trends observed in the current publishing process (e.g., HARKing, dropping null findings to craft a better “narrative,” etc.), the value proposition offered by their most prestigious and profitable research outlets stands to be damaged. Furthermore, to the extent that authors and consumers within the scientific enterprise value the benefits to robust science that open-access journals provide (i.e., quicker time to print, greater data transparency, “crowd-sourced” peer reviews, more rapid dissemination of ideas throughout the scientific community, higher citation rates, etc.), shifts in submission rates and journal readership may change. Some have suggested that many of the publication practices observed today stem from constraints that emerged because of production costs associated with printing paper copies of journals (e.g., Nosek & Bar-Anan, Reference Nosek and Bar-Anan2012). Whatever the genesis, removing archaic page restrictions, ending restrictions on editorial team size, and providing latitude/encouragement to editors to engage in innovative initiatives opens the door for a variety of alternative journal publication and readership practices capable of fostering a more robust science. For example, new metrics for research quality that move beyond simple citation-based measures can be pursued. Research access models where impact is assessed by factors such as page views, the quantity and quality of active commentary, the number of studies that replicate and extend the original work, and the value conveyed by the provision of supporting materials and original data are more easily traceable in an open-access model and contribute to the accumulation of robust science. Increasing the variety and transparency of research outlets and publisher practices has potential to stimulate more and better scientific advances while maintaining the credibility of the scientific and technical publishing market. Like editors and reviewers, publishers should strive to be stewards of science as well.

Academic Hiring and Promotion Committees

The phrase “publish or perish” is a ubiquitous and anxiety-provoking maxim familiar to anyone within academia. Whereas the production and dissemination of scientific knowledge is a core activity of all researchers, publication numbers, citation counts, and other easily quantifiable metrics of research productivity remain the coin of the realm in higher education (De Rond & Miller, Reference De Rond and Miller2005). These norms have almost certainly been driven by increased competition over declining resources in the scientific landscape and the subsequent need to differentiate oneself and one's academic institution as respectable, distinguished, and desirable. Within academia, the proverbial rubber meets the road with the publish-or-perish adage in the form of hiring and promotion committees that determine who is hired, who is retained, and who is permitted to rise through the ranks. Much has been written and even limited empirical evidence exists linking the pressures to publish reinforced through hiring and promotion committees to detrimental research practices and other undesirable outcomes (e.g., Anderson, Reference Anderson2007; Fanelli, Reference Fanelli2010b; Rawat & Meena, Reference Rawat and Meena2014). Hiring and promotion committees thus represent an obvious and important contributor for shaping the way in which the scientific enterprise operates.

We recommend that academic hiring and promotion committees should adopt a more holistic approach to evaluating candidates that reduces singular emphasis on “very elite” research productivity per se and rewards engagement in practices that foster robust science. Advancing scientific knowledge that is relevant, rigorous, replicable, cumulative, transparent, and theory oriented—that is, research that adheres to the defining principles of robust science—should be among the most important criteria for determining the contributions of a researcher's work. These evaluative markers can be incorporated into hiring decisions by efforts to evaluate an applicant's record of robust scientific practices and the extent to which these individuals have taken proactive steps toward demonstrating these principles in their contributions to the field (i.e., engage in transparent data sharing, attempt to replicate their findings in addition to advancing new understanding, demonstrate the capability to produce precise theoretical predictions). Likewise, promotion and tenure decisions are often influenced by the sheer quantity of an individual's publications and the extent to which these products appear in “A” journals; at the extreme, these decisions may rely very little on appraisal of the quality of an individual's work. At the same time, they can create a very high-stakes publishing game where authors will perhaps do everything it takes (including engaging in detrimental research practices) to publish in the journals that will get them their tenure and/or promotion. The achievement of robust science simply cannot be achieved through such a reward structure. Only through valuation of the extent to which a researcher's body of work embraces robust science, regardless of where such work may be published, will the scientific enterprise advance in a positive direction. We recognize that such shifts in the evaluation of a researcher's record places different and significant demands upon peers and administrators to more fully understand the nature of robust scientific contributions. However, we believe such incentives are critical and liberating to the pursuit of accurate and useful scientific knowledge.

Consumers

How and why the scientific knowledge generated by a field is used vary widely depending on the needs and context of different research consumers. The insights gained through I-O psychology are used to advance future research, guide practice, and to support policies/decisions of critical importance to individuals, teams, businesses, and society. Consumers of scientific research span a diverse range of stakeholders, including authors, reviewers, editors, practitioners, funding agencies, organizational decision makers, and the general public (e.g., media). Although each of these constituencies shares varying degrees of responsibility for ensuring that scientific research is appropriately conducted, all share responsibility for ensuring that this knowledge is properly interpreted and correctly utilized. As a source of demand for the supply of scientific knowledge generated within our field, consumers of research also play an important role in fostering a healthy scientific enterprise.

We believe consumers should place greater value on research that embodies the defining principles of robust science. Novel, counterintuitive, and “catchy” theories/research findings have always sparked great interest across the spectrum of scientific consumers (Antonakis, Reference Antonakis2017), and likely will continue to do so. Why wouldn't they? Many who conduct research, and likely many more who do not, believe that the spirit of science lies in innovation, discovery, and intrepid exploration of the unknown (e.g., Astley, Reference Astley1985; Davis, Reference Davis1971; Pfeffer & Fong, Reference Pfeffer and Fong2002). Weick (Reference Weick1989) even famously equated the construction of theory in the managerial sciences as “disciplined” imagination. We acknowledge that these motives are central to the fabric of science and can be beneficial forces for scientific advancement. However, when consumers within the scientific enterprise too strongly favor “creative” science to the detriment of robust science, a positive feedback loop is created in which the number of ideas in need of testing/verification far surpasses the capacity to do so.Footnote ⁶ This state of affairs places the relevance, reliability, and credibility of a field's work at risk. Consumers of research can influence this trend by holding science accountable for producing robust science. The ability to distinguish robust science obviously requires understanding of the power and limitations of research methodologies as well as how to evaluate the degree of confidence that should be placed in research. However, the challenge in determining these criteria is attenuated when scientific evidence is supported by behaviors characterized by rigor, transparency, and precision. Consequently, by expressing preference toward journals and outlets that require and actively engage in robust scientific practices, consumers have potential to shape the scientific ecosystem.

Research Participants

Unlike many of the natural sciences, the foci of study for the social sciences do not uniformly adhere to well-known and clearly distinguishable principles of behavior. People and the psychological/social systems that they create are diverse, rich, and exhibit a free will of choice that makes efforts to understand how they function simultaneously fascinating and perplexing. As I-O psychologists, we rely on a variety of measurement and observational tools that attempt to capture the “hidden” experiences of people in laboratory and organizational contexts. The individuals who participate in our research thus carry significant weight in determining the quality and reliability of the evidentiary base. Although it can be difficult (and often undesirable) to directly influence the way in which research participants engage with the scientific process, they are integral elements of the scientific ecosystem whose actions clearly contribute to key aspects of a robust science.

Accordingly, we believe that research participants should be encouraged to recognize the importance of their contribution to science and participate in ways that reflect that status. Discussions about the use of college students (e.g., Greenberg, Reference Greenberg1987; Sears, Reference Sears1986) and more recently Mechanical Turk workers (e.g., Burhrmester, Kwang, & Gosling, Reference Buhrmester, Kwang and Gosling2011; Cheung, Burns, Sinclair, & Sliter, Reference Cheung, Burns, Sinclair and Sliter2017; Paolacci & Chandler, Reference Paolacci and Chandler2014) as research participants frequently cite concerns over the trustworthiness and credibility of these data sources. Though similar critiques can just as easily be levied against samples gathered from organizational workers (Landers & Behrend, Reference Landers and Behrend2015), the more fundamental concern is legitimate: the validity of scientific inference is dependent on the quality of the observations and the evidence upon which it is based. For example, recent research has identified the pervasiveness of careless responding on the part of research participants (Huang, Curran, Keeney, Poposki, & DeShon, Reference Huang, Curran, Keeney, Poposki and DeShon2012; Meade & Craig, Reference Meade and Craig2012). This lack of attention and effort can reduce the rigor, precision, and utility of our scientific conclusions (cf., Huang, Liu, & Bowling, Reference Huang, Liu and Bowling2015). Consequently, we believe it is important that research participants are cognizant that their actions can improve—and hinder—the quality of science. Beyond a role as reliable and honest contributors, research participants can also play an important role in reporting violations of participant rights. One emerging issue concerns the need to inform participants and for participants to be informed about what will happen with their data. Transparency is a central tenet of robust science, including data transparency. Researchers and practitioners have an important responsibility to ensure that appropriate identity and confidentiality safeguards are in place, but participants may be faced with a new normal where their data are likely to be available more broadly than they have been in the past, and thus participants should be encouraged to inquire about the protections and rights with which they are afforded.

Funders

Although funded research within I-O psychology and the managerial sciences remains much less common than in other fields, there are a number of government agencies and nonprofit foundations that support the advancement of scientific research. In the United States, the National Science Foundation, numerous agencies within the Department of Defense, the National Institutes of Health, the Alfred P. Sloan Foundation, and the Robert Wood Johnson Foundation, among others, all contain programs that provide financial resources to promote the accumulation of knowledge by extramural researchers in topic areas relevant to our discipline. Many industries and for-profit companies in the private sector also invest in their own internal research that can contribute to the accumulation of our science's evidentiary base. In addition to providing critical resources, the research agendas of many funders are crafted to promote research on topics of fundamental interest to society and humanity. Given that they represent a key source of support for establishing the relevance of research, funders can thus exert significant influence on what, how, and why research is conducted within the broader scientific enterprise.

Consequently, we believe that research funders should institute and enforce policies that promote greater transparency, rigor, and precision when evaluating and rewarding research proposals. In some respects, funders, particularly those within the federal sector, are already among the leading practitioners and endorsers of practices that support open and robust science. For example, a 2013 memorandum released by the United States Office of Science and Technology Policy directed all federal agencies whose budgets included at least $100 million to establish research and development plans to support increased public access to federally funded research (Holdren, Reference Holdren2013). As of 2016, 16 agencies established requirements for researchers to provide open access to any peer-reviewed publication resulting from funded research within at least 12 months of publication, and a number of these institutions have already created or been integrated into designated public repositories for housing this work. Building off that momentum, many federal funding agencies have also begun instituting infrastructure to support large-scale data storage and sharing agreements as well (Sheehan, Reference Sheehan2016). These initiatives represent an impressive and positive step in the direction of fostering robust science. Infrequent participation by I-O psychologists in seeking research funding has likely contributed to our field being less impacted by these changes. However, as pressure to secure external support continues to mount in higher education, we suspect these practices will become more commonly followed by researchers in our discipline as well.

Scientific Tool Providers

The process of science is supported by a large marketplace of tools and software to aid researchers in the production and generation of scientific knowledge. The tools most familiar to researchers include those used to perform statistical analyses and survey hosting platforms, but there are also a growing number of products available for data processing, project management, real-time collaboration, storage, and the dissemination of methods, measures, and data. As these and other technologies are developed and adopted into the scientific enterprise, they have the potential to significantly impact the way in which research is conducted and scientific knowledge is accumulated. Consequently, scientific tool builders can play an important role in shaping the behaviors and expectations related to robust science.

To this end, we believe that scientific tool builders should pursue product designs in which the “default” behavior of a tool facilitates robust scientific practice. The end users of scientific tools will ultimately determine the utility of a particular product and its value for facilitating research. However, some ways in which tools are perceived as useful is if they are easy to use “out of the box,” accomplish their desired tasks accurately and reliably, and allow users to perform tasks essential to high quality research with less effort. Similarly, the capability to store one's measures, data, analytic syntax, manuscript documents, and review letters in a secure repository that can be directly accessed across any journal submission portal can increase the probability of authors, reviewers, and editors promoting more transparent scientific reporting. The democratization of tools (such as the move to open access, statistical programs like R or Python) used in research will also benefit robust science. When more research is conducted on common platforms with identical or similar tools, transparency of methods and analytic techniques will be enhanced, and these conditions will contribute to the ease with which findings may be replicated. By no means should this recommendation be taken as a justification for tempering scientific innovation and the use of novel analytical techniques when and as appropriate. Rather, we see the use of common tools where possible as an aid (not a mandate) for promoting transparency, replicability, and rigor that should be encouraged.