Skip to main content

A Systems-Based Approach to Fostering Robust Science in Industrial-Organizational Psychology

  • James A. Grand (a1), Steven G. Rogelberg (a2), Tammy D. Allen (a3), Ronald S. Landis (a4), Douglas H. Reynolds (a5), John C. Scott (a6), Scott Tonidandel (a7) and Donald M. Truxillo (a8)...

Credibility and trustworthiness are the bedrock upon which any science is built. The strength of these foundations has been increasingly questioned across the sciences as instances of research misconduct and mounting concerns over the prevalence of detrimental research practices have been identified. Consequently, the purpose of this article is to encourage our scientific community to positively and proactively engage in efforts that foster a healthy and robust industrial and organizational (I-O) psychology. We begin by advancing six defining principles that we believe reflect the values of robust science and offer criteria for evaluating proposed efforts to change scientific practices. Recognizing that the contemporary scientific enterprise is a complex and diverse network of actors and institutions, we then conclude by identifying 12 stakeholders who play important roles in achieving a culture of robust science in I-O psychology and offer recommendations for actions we can take as members of these groups to strengthen our science.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the or variations. ‘’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      A Systems-Based Approach to Fostering Robust Science in Industrial-Organizational Psychology
      Available formats
      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      A Systems-Based Approach to Fostering Robust Science in Industrial-Organizational Psychology
      Available formats
      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      A Systems-Based Approach to Fostering Robust Science in Industrial-Organizational Psychology
      Available formats
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (, which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
Corresponding author
Correspondence concerning this article should be addressed to James A. Grand, Department of Psychology, University of Maryland, 3147A Biology-Psychology Building, 4094 Campus Drive, College Park, MD, 20742. E-mail:
Hide All

The authors of this article are members of a task force on Robust and Reliable Science created by the Society for Industrial and Organizational Psychology (SIOP). The task force was charged with evaluating the state of research practices in the field of industrial and organizational (I-O) psychology and with providing recommendations to SIOP's Executive Board for areas where targeted improvements could be directed. Summary material from this article was presented in a report to the Executive Board prior to SIOP's annual conference in April 2017. The intention of this focal article, which went through the journal peer review process, is to summarize the initial outcomes generated by the task force and elicit commentary from a wider range of stakeholders to critique, discuss, and expand upon these positions. The information and input received from these commentaries will be incorporated into a subsequent final report delivered to SIOP's Executive Board in February 2018. Our hope is that the collective discussion of these topics and the development and endorsement of a set of principles by our community will serve as a basis for future decisions and initiatives within SIOP and our broader profession that facilitate an even stronger and more robust science within I-O psychology.

Hide All
Allen, T. D. (2015). Connections past and present: Bringing our scientific influence into focus. The Industrial and Organizational Psychologist, 52 (3), 126133.
Anderson, C. J., Bahnik, S., Barnett-Cowan, M., Bosco, F. A., Chandler, J., Chartier, C. R., . . . Zuni, K. (2016). Response to comment on “Estimating the reproducibility of psychological science.” Science, 351, 1037–c.
Anderson, M. S. (2007). The perverse effects of competition on scientists’ work and relationships. Science and Engineering Ethics, 13, 437461.
Antonakis, J. (2017). On doing better science: From thrill of discovery to policy implications. The Leadership Quarterly, 28, 521.
Astley, W. G. (1985). Administrative science as socially constructed truth. Administrative Science Quarterly, 30, 497513.
Atwater, L. E., Mumford, M. D., Schriesheim, C. A., & Yammarino, F. J. (2014). Retraction of leadership articles: Causes and prevention. Leadership Quarterly, 25, 11741180.
Baker, M. (2016). Is there a reproducibility crisis? Nature, 533, 452454.
Banks, G. C., & O'Boyle, E. H. Jr. (2013). Why we need industrial-organizational psychology to fix industrial-organizational psychology. Industrial and Organizational Psychology: Perspectives on Science and Practice, 6, 284287.
Banks, G. C., O'Boyle, E. H. Jr., Pollack, J. M., White, C. D., Batchelor, J. H., Whelpley, C. E., . . . Adkins, C. L. (2016). Questions about questionable research practices in the field of management: A guest commentary. Journal of Management, 42, 520.
Banks, G. C., Rogelberg, S. G., Woznyj, H. M., Landis, R. S., & Rupp, D. E. (2016). Editorial: Evidence on questionable research practices: The good, the bad, and the ugly. Journal of Business and Psychology, 31, 323338.
Bedeian, A. G., Taylor, S. G., & Miller, A. N. (2010). Management science on the credibility bubble: Cardinal sins and various misdemeanors. Academy of Management Learning & Education, 9, 715725.
Bell, B. S., & Kozlowski, S. W. J. (2008). Active learning: Effects of core training design elements on self-regulatory processes, learning, and adaptability. Journal of Applied Psychology, 93, 296316.
Bosco, F. A., Aguinis, H., Field, J. G., Pierce, C. A., & Dalton, D. R. (2016). Harking's threat to organizational research: Evidence from primary and meta-analytic sources. Personnel Psychology, 69, 709750.
Boutelle, C. (2014, February 5). I-O is on top! Retrieved from
Bridges, J. H. (1897). The “Opus Majus” of Roger Bacon, Volume I. Oxford, UK: Clarendon Press.
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 35.
Byington, E., & Felps, W. (2017). Solutions to the credibility crisis in management science. Academy of Management Learning & Education, 16, 142162.
Campbell, J. P., & Wilmot, M. P. (in press). The functioning of theory in IWOP. In Anderson, N., Ones, D. S., Sinangil, H. K., & Viswesvaran, C. (Eds.), Handbook of industrial, work, and organizational psychology (IWOP): Volume 1, Personnel psychology (2nd ed). London, UK: Sage.
Carey, B. (2015). Many psychology findings not as strong as claimed, study says. New York Times. Retrieved from
Casadevall, A. & Fang, F. C. (2012). Reforming science: Methodological and cultural reforms. Infection and Immunity, 80, 891896.
Cheung, J. H., Burns, D. K., Sinclair, R. R., & Sliter, M. (2017). Amazon Mechanical Turk in organizational psychology: An evaluation and practical recommendations. Journal of Business and Psychology, 32 (4), 347361.
Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 13041312.
Colquitt, J. A., & Zapata-Phelan, C. P. (2007). Trends in theory building and theory testing: A five-decade study of the Academy of Management Journal . Academy of Management Journal, 50, 12811303.
Cortina, J. M., Aguinis, H., & DeShon, R. P. (2017). Twilight of dawn or evening? A century of research methods in the Journal of Applied Psychology . Journal of Applied Psychology, 102, 274290.
Davis, M. S. (1971). That's interesting! Towards a phenomenology of sociology and a sociology of phenomenology. Philosophy of Social Science, 1, 309344.
De Rond, M., & Miller, A. N. (2005). Publish or perish: Bane or boon of academic life? Journal of Management Inquiry, 14, 321329.
Edwards, J. R., & Berry, J. W. (2010). The presence of something or the absence of nothing: Increasing theoretical precision in management research. Organizational Research Methods, 13, 668689.
Einstein, A. (1936). Physics and reality. Journal of the Franklin Institute, 221, 349382.
Evans, J. A., & Reimer, J. (2009). Open access and global participation in science. Science, 323, 1025.
Fagerberg, J. (1994). Technology and international differences in growth rates. Journal of Economic Literature, 32, 11471175.
Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE, 4, e5738.
Fanelli, D. (2010a). “Positive” results increase down the hierarchy of the sciences. PLoS ONE, 5, e10068.
Fanelli, D. (2010b). Do pressures to publish increase scientists’ bias? An empirical support from US states data. PLoS ONE, 5, e10271.
Fanelli, D. (2011). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891904.
Feynman, R. P. (1969). What is science? The Physics Teacher, 7, 313320.
Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345, 15021505.
Gabriel, A. S., & Wessel, J. L. (2013). A step too far? Why publishing raw datasets may hinder data collection. Industrial and Organizational Psychology: Perspectives on Science and Practice, 6, 287290.
Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016). Comment on “Estimating the reproducibility of psychological science.” Science, 351, 1037-b.
Greenberg, J. (1987). The college sophomore as guinea pig: Setting the record straight. Academy of Management Review, 12, 157159.
Greenberg, J., & Colquitt, J. A. (2005). Handbook of organizational justice. Mahwah, NJ: Erlbaum.
Grote, G. (2016). There is hope for better science. European Journal of Work and Organizational Psychology, 26, 13.
Hambrick, D. C. (2007). The field of management's devotion to theory: Too much of a good thing? Academy of Management Journal, 50, 13461352.
Hitchcock, C., & Sober, E. (2004). Prediction versus accommodation and the risk of overfitting. British Journal for the Philosophy of Science, 55, 134.
Holdren, J. P. (2013). Increasing access to the results of federally funded scientific research [Memorandum]. Washington, DC: Office of Science and Technology Policy.
Hollenbeck, J. R. (2008). The role of editing in knowledge development: Consensus shifting and consensus creation. In Baruch, Y., Konrad, A. M., Aguinis, H., & Starbuck, W. H. (Eds.), Opening the black box of editorship (pp. 1626). New York: Macmillan.
Hollenbeck, J. R., & Wright, P. M. (2017). Harking, sharking, and tharking: Making the case for post hoc analysis of scientific data. Journal of Management, 43, 518.
Honig, B., Lampel, J., Siegel, D., & Drnevich, P. (2014). Ethics in the production and dissemination of management research: Institutional failure or individual fallibility? Journal of Management Studies, 51, 118142.
Houghton, J. W. & Oppenheim, C. (2010). The economic implications of alternative publishing models. Prometheus, 28, 4154.
Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27, 99114.
Huang, J. L., Liu, M., & Bowling, N. A. (2015). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100, 828845.
Institute of Medicine, National Academy of Sciences, & National Academy of Engineering. (1995). On being a scientist: Responsible conduct in research, 2nd ed. Washington, DC: The National Academies Press.
Ioannidis, J. P. A. (2005a). Why most published research findings are false. PLoS Medicine, 2, e124.
Ioannidis, J. P. A. (2005b). Contradicted and initially stronger effects in highly cited clinical research. Journal of the American Medical Association, 294, 218228.
Ireland, R. D. (2008). From the editors: Your manuscript's journey through the AMJ review process. Academy of Management Journal, 51, 409412.
Jha, A. (2012, September 13). False positives: Fraud and misconduct are threatening scientific research. The Guardian. Retrieved from
Jones, J. W, & Dages, K. D. (2013). A new era of large-scale data sharing: A test publisher's perspective. Industrial and Organizational Psychology: Perspectives on Science and Practice, 6, 309312.
Kepes, S., & McDaniel, M. A. (2013). How trustworthy is the scientific literature in industrial and organizational psychology? Industrial and Organizational Psychology, 6, 252268.
Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality & Social Psychology Review, 2, 196217.
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B. Jr., Bahnik, S., Bernstein, M. J., . . . Nosek, B. A. (2014). Investigating variation in replicability. Social Psychology, 45, 142152.
Kozlowski, S. W. J., Chao, G. T., Grand, J. A., Braun, M. T., & Kuljanin, G. (2013). Advancing multilevel research design: Capturing the dynamics of emergence. Organizational Research Methods, 16, 581615.
Kozlowski, S. W. J., Chen, G., & Salas, E. (2017). One hundred years of the Journal of Applied Psychology: Background, evolution, and scientific trends. Journal of Applied Psychology, 102, 237253.
Landers, R. N., & Behrend, T. S. (2015). An inconvenient truth: Arbitrary distinctions between organizational, Mechanical Turk, and other convenience samples. Industrial and Organizational Psychology: Perspectives on Science and Practice, 8, 142164.
Landis, R. S., James, L. R., Lance, C. E., Pierce, C. A., & Rogelberg, S. G. (2014). When is nothing something? Editorial for the null results special issue of Journal of Business and Psychology . Journal of Business and Psychology, 29, 163167.
Latham, G. P. (2007). A speculative perspective on the transfer of behavioral science findings to the workplace: “The times they are a-changin.” Academy of Management Journal, 50, 10271032.
Luscher, L. S., & Lewis, M. W. (2008). Organizational change and managerial sensemaking: Working through paradox. Academy of Management Journal, 51, 221240.
Marcus, E. (2014). Credibility and reproducibility. Cell, 159, 965966.
McDaniel, M. A., Kepes, S., Hartman, N. S., & List, S. K. (2017, April). Questionable research practices among researchers in top management programs. Paper presented at the 32nd Annual Conference of the Society for Industrial and Organizational Psychology. Orlando, FL.
McGrath, J. E. (1982). Dilemmatics: The study of research choices and dilemmas. In McGrath, J. E., Martin, J., & Kulka, R. A. (Eds.), Judgment calls in research (pp. 69102). Beverly Hills, CA: Sage.
McKinley, W. (2010). Organizational theory development: Displacement of ends? Organization Studies, 31, 4768.
Meade, A. W., & Craig, S. B. (2012). Identifying careless responding in survey data. Psychological Methods, 17, 437455. doi: 10.1037/a0028085
Meehl, P. E. (1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34, 103115.
Mitchell, G. (2012). Revisiting truth or triviality: The external validity of research in the psychological laboratory. Perspectives on Psychological Science, 7, 109117.
National Academies of Sciences & Institute of Medicine (2008). Science, evolution, and creationism. Washington, DC: The National Academies Press.
National Academies of Sciences, Engineering, & Medicine. (2017). Fostering integrity in research. Washington, DC: The National Academies Press
National Academy of Sciences, National Academy of Engineering, & Institute of Medicine. (1992). Responsible science, Volume I: Ensuring the integrity of the research process. Washington, DC: The National Academies Press.
National Science Board. (2016). Science and engineering indicators 2016. Arlington, VA: National Science Foundation.
Newcombe, R. G. (1987). Towards a reduction in publication bias. British Medical Journal, 295, 656659.
Neylon, C. (2013). Growth of open access - major publishers. Figshare. Retrieved February 26, 2017.
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., . . . Yarkoni, T. (2015). Promoting an open research culture: Author guidelines for journals could help to promote transparency, openness, and reproducibility. Science, 348, 14221425.
Nosek, B. A., & Bar-Anan, Y. (2012). Scientific utopia: I. Opening scientific communication. Psychological Inquiry, 23, 217243.
Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 76, 615631.
O'Boyle, E. H. Jr., Banks, G. C., & Gonzalez-Mule, E. (2017). The chrysalis effect: How ugly initial results metamorphosize into beautiful articles. Journal of Management, 43, 376399.
Ones, D. S., Kaiser, R. B., Chamorro-Premuzic, T., & Svensson, C. (2017). Has industrial-organizational psychology lost its way? The Industrial-Organizational Psychologist, 54. Retrieved from
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716.
Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184188.
Pavitt, K. (1996). National policies for technical change: Where are the increasing returns to economic research? Proceedings of the National Academy of Sciences, 93, 1269312700.
Pew Research Center. (2015). Public and scientists’ views on science and society. Retrieved from
Pew Research Center. (2016). State of the news media 2016. Retrieved from
Pfeffer, J. (1993). Barriers to the advance of organizational science: Paradigm development as a dependent variable. Academy of Management Review, 18, 599620.
Pfeffer, J., & Fong, C. T. (2002). The end of business schools? Less success than meets the eye. Academy of Management Learning and Education, 1, 7895.
Raven, B. H. (1992). A power/interaction model of interpersonal influence: French and Raven thirty years later. Journal of Social Behavior and Personality, 7, 217244.
Rawat, S., & Meena, S. (2014). Publish or perish: Where are we heading? Journal of Research in Medical Sciences, 19, 8789.
Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638641.
Rosnow, R. L., & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 12761284.
Rubin, M. B. (2011, July). Fraud in organic chemistry. Chemistry in New Zealand, 78, 128132.
Rudolph, J. W., & Repenning, N. P. (2002). Disaster dynamics: Understanding the role of quantity in organizational collapse. Administrative Science Quarterly, 47, 130.
Rupp, D. E., & Beal, D. (2007). Checking in with the scientist-practitioner model: How are we doing? The Industrial-Organizational Psychologist, 45, 3540.
Rynes, S. L. (2007). Let's create a tipping point: What academics and practitioners can do, alone and together. Academy of Management Journal, 50, 10451054.
Rynes, S. L., Giluk, T. L., & Brown, K. G. (2007). The very separate worlds of academic and practitioner periodicals in human resource management: Implications for evidence-based management. Academy of Management Journal, 50, 9871008.
Schmidt, F. L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1, 115129.
Schmidt, F. L., Hunter, J. E., Outerbridge, A. M., & Trattner, M. H. (1986). The economic impact of job selection methods on the size, productivity, and payroll costs of the federal workforce: An empirical demonstration. Personnel Psychology, 39, 129.
Sears, D. O. (1986). College sophomores in the laboratory: Influences of a narrow data base on social psychology's view of human nature. Journal of Personality and Social Psychology, 51, 515530.
Sheehan, J. (2016, February 22). Increasing access to the results of federally funded science. Retrieved from
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 13591366.
SIOP Membership Committee. (2017, April). Membership committee report. Retrieved from
Spector, P. E., Rogelberg, S. G., Ryan, A. M., Schmitt, N., & Zedeck, S. (2014). Moving the pendulum back to the middle: Reflections on and introduction to the inductive research special issue of Journal of Business and Psychology . Journal of Business and Psychology, 29, 499502.
Spier, R. (2002). The history of the peer-review process. Trends in Biotechnology, 20, 357358.
Taylor, P. J., Russ-Eft, D. F., & Chan, D. W. L. (2005). A meta-analytic review of behavior modeling training. Journal of Applied Psychology, 90, 692709.
Ware, M., & Mabe, M. (2015). The STM Report: An overview of scientific and scholarly journal publishing. International Association of Scientific, Technical and Medical Publishers, Netherlands. Retrieved from
Weick, K. (1989). Theory construction as disciplined imagination. Academy of Management Review, 14, 516531.
Yong, E. (2016). Psychology's replication crisis can't be wished away. The Atlantic. Retrieved from
Zedeck, S. (2008). Editing a top academic journal. In Baruch, Y., Konrad, A. M., Aguinis, H., & Starbuck, W. H. (Eds.), Opening the black box of editorship (pp. 145156). New York: Macmillan.
Zedeck, S. (2017). Reflections on the Journal of Applied Psychology for 2003 to 2008. Journal of Applied Psychology, 102, 574579.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Industrial and Organizational Psychology
  • ISSN: 1754-9426
  • EISSN: 1754-9434
  • URL: /core/journals/industrial-and-organizational-psychology
Please enter your name
Please enter a valid email address
Who would you like to send this to? *



Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed