Skip to main content

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
Cancel
×
Home
Article
    • Aa
    • Aa
  • Volume 5 , Issue 2 (Special Symposium Issue)
  • Get access
    Check if you have access via personal or institutional login
    Log in Register Recommend to librarian
  • Cited by 350
  • Cited by
    Crossref Citations
    Crossref logo
    This article has been cited by the following publications. This list is generated based on data provided by CrossRef.

    Aguinis, Herman O'Boyle, Ernest Gonzalez-Mulé, Erik and Joo, Harry 2016. Cumulative Advantage: Conductors and Insulators of Heavy-Tailed Productivity Distributions and Productivity Stars. Personnel Psychology, Vol. 69, Issue. 1, p. 3.

    Callaghan, Christian William 2016. Disaster management, crowdsourced R&D and probabilistic innovation theory: Toward real time disaster response capability. International Journal of Disaster Risk Reduction, Vol. 17, p. 238.

    Figdor, Carrie and Drabek, Matt L. 2016. A Companion to Experimental Philosophy.

    Hanitzsch, Thomas 2016. Impact und Normalwissenschaft. Publizistik, Vol. 61, Issue. 1, p. 41.

    Iantorno, Stephanie E. Andras, Lindsay M. and Skaggs, David L. 2016. Variability of Reviewers' Comments in the Peer Review Process for Orthopaedic Research. Spine Deformity, Vol. 4, Issue. 4, p. 268.

    Largent, Emily A. and Snodgrass, Richard T. 2016. Blinding as a Solution to Bias.

    Lau, David C.W. 2016. Tackling Peer Review: How to Improve Reviews and Minimize Abuse. Canadian Journal of Diabetes, Vol. 40, Issue. 2, p. 105.

    Louw, Quinette and Grimmer, Karen 2016. Peer Review: Professionally Important and an Opportunity to Contribute. Physiotherapy Research International, Vol. 21, Issue. 2, p. 67.

    Nobarany, Syavash and Booth, Kellogg S. 2016. Understanding and supporting anonymity policies in peer review. Journal of the Association for Information Science and Technology,

    Oler, Derek K. and Pasewark, William R. 2016. How to Review a Paper. Issues in Accounting Education, Vol. 31, Issue. 2, p. 219.

    Peres-Neto, Pedro R. 2016. Will technology trample peer review in ecology? Ongoing issues and potential solutions. Oikos, Vol. 125, Issue. 1, p. 3.

    Potuto, Josephine (Jo) R. 2016. Whose Article Is It Anyway?: Student Editors and Law Reviews. Indiana Law Review, Vol. 49, Issue. 3, p. 609.

    Resnik, David B. and Elmore, Susan A. 2016. Ensuring the Quality, Fairness, and Integrity of Journal Peer Review: A Possible Role of Editors. Science and Engineering Ethics, Vol. 22, Issue. 1, p. 169.

    Shadish, William R. Zelinsky, Nicole A. M. Vevea, Jack L. and Kratochwill, Thomas R. 2016. A survey of publication practices of single-case design researchers when treatments have small or large effects. Journal of Applied Behavior Analysis,

    Tsui, Anne 2016. Reflections on the so-called value-free ideal. Cross Cultural & Strategic Management, Vol. 23, Issue. 1, p. 4.

    Benatar, David 2015. The Gendered Conference Campaign: A Critique. Philosophia, Vol. 43, Issue. 1, p. 13.

    Blanes i Vidal, Jordi and Leaver, Clare 2015. Bias in Open Peer-Review: Evidence from the English Superior Courts. Journal of Law, Economics, and Organization, Vol. 31, Issue. 3, p. 431.

    Blasi, Brigida 2015. Severità di giudizio: dinamiche valutative nell'area della sociologia nella VQR 2004-2010. SOCIOLOGIA E POLITICHE SOCIALI, Issue. 2, p. 9.

    Chattoe-Brown, Edmund 2015. “Censorship”, early childhood research quarterly and qualitative research: Not so much aced out as an own goal?. Early Childhood Research Quarterly, Vol. 31, p. 163.

    Gantman, Ernesto R. 2015. Management research worldwide: extent and determinants (1996-2010). International Journal of Organizational Analysis, Vol. 23, Issue. 2, p. 250.

    Download full list
    ×

Peer-review practices of psychological journals: The fate of published articles, submitted again

  • Douglas P. Peters (a1) and Stephen J. Ceci (a2)
  • DOI: http://dx.doi.org/10.1017/S0140525X00011183
  • Published online: 01 February 2010
Abstract
Abstract

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.

The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we selected 12 already published research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.

With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated.

Copyright
COPYRIGHT: © Cambridge University Press 1982
Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

P. H. Abelson (1979) Problems of science faculties (editorial). Science 204:133. [CM]
R. K. Adair (1981) Anonymous refereeing. Physics Today 34:13–15. [RAG]
American Psychological Association (1973) Ethical principles in the conduct of research with human participants. American Psychologist 28:79–80. [JLF, BM]
American Psychological Association (1980) Summary report of journal operations for 1979. American Psychologist 35:575. [taDPP]
J. S. Armstrong (1979) Advocacy and objectivity in science. Management Science 25:423–28. [JSA]
J. S. Armstrong (1980a) Advocacy as a scientific strategy: The mitroff myth. Academy of Management Review 5:509–11. [JSA]
J. S. Armstrong (1980b) Unintelligible management research and academic prestige. Interfaces 10:80–86. [JSA]
J. J. Bartko (1966) The intraclass correlation coefficient as a measure of reliability. Psychological Reports 19:3–11. [GJW]
R. Benwell (1979) Authors anonymous? Physics Bulletin 30:288. [RAG]
J. Beyer (1978) Editorial policies and practices among leading journals in four scientific fields. Sociological Quarterly 19:68–88. [JMB]
D. D. Bowen ; R. Perloff & J. Jacoby (1972) Improving manuscript evaluation procedures. American Psychologist 27:221–25. [taDPP]
Y. Brackbill & F. Korton (1970) Journal reviewing practices: Authors' and APA members' suggestions for revision. American Psychologist 25:937–40. [taDPP]
W. J. Broad (1980a) Imbroglio at Yale. 1. Emergence of a fraud. Science 210:38–11. [DdB]
W. J. Broad (1980b) Would-be academician pirates papers. Science 208:1438–10. [DdB]
W. J. Broad (1981a) Congress told fraud issue “exaggerated.” Science 212:421. [CM]
W. J. Broad (1981b) Fraud and the structure of science. Science 212:137–41. [DdB, CM]
W. J. Broad (1981c) The publishing game: Getting more for less. Science 211:1137–39. [DdB]
U. Bronfenbrenner (1977) Toward an experimental ecology of human development. American Psychologist 32:513–31. [taDPP]
D. T. Campbell & D. W. Fiske (1959) Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin 56:81–105. [rDPP]
W. D. Carey (1975) Peer review revisited. Science 189:331. [MB]
D. E. Chubin (1980) Competence is not enough. Contemporary Sociology 9:204–7. [DEC, CM]
D. E. Chubin & T. Connolly (1982) Research trails and science policies: Local and extra-local negotiation of scientific work. In: Scientific establishments and hierarchies, ed. N. Elias , H. Martins , & H. Whitley . Sociology of the Sciences, vol. 6, pp. 293–311. [ALP]
D. V. Cicchetti (1980) Reliability of reviews for the American Psychologist: A biostatistical assessment of the data. American Psychologist 35:300–303. [DVC, tarDPP, GJW]
J. Cohen (1968) Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 70:213–20. [GJW]
J. R. Cole & S. Cole (1972) The Ortega hypothesis: Citation analysis suggests that only a few scientists contribute to scientific progress. Science 178:368–74. [CM, taDPP]
S. Cole ; J. R. Cole & G. A. Simon (1981) Chance and consensus in peer review. Science 214:881–86. [Ed., rDPP]
S. Cole ; L. Rubin & J. R. Cole (1977) Peer review and the support of science. Scientific American 237:34–11. [RTL, CM]
H. M. Collins (1981) Stages in the empirical programme of relativism. Social Studies of Science 11:3–10. [MDG]
W. M. Cox & V. Catt (1977) Productivity ratings of graduate programs in psychology based on publication in the journals of the American Psychological Association. American Psychologist 32:793–813. [taDPP]
R. Crandall (1977) How qualified are editors? American Psychologist 32:578–79. [RC]
R. Crandall (1978a) Interrater agreement on manuscripts is not so bad! American Psychologist 33:623–24. [RC, taDPP]
R. Crandall (1978b) The relationship between quantity and quality of publications. Personality and Social Psychology Bulletin 4:379–80. [RC]
J. M. Davidson & R. J. Davidson , eds. (1980) The psychobiology of consciousness. New York: Plenum. [MJM]
D. Eckberg & L. Hill (1979) The paradigm concept and sociology. American Sociological Review 44:925–37. [DLE]
N. S. Endler ; J. P. Rushton & H. L. Roediger (1978) Productivity and scholarly impact (citations) of British, Canadian, and U.S. departments of psychology. American Psychologist 33:1064–82. [taDPP]
W. D. Garvey & B. C. Griffith (1964) Scientific information exchange in psychology. Science 146:1655–59. [BCG]
W. D. Garvey & B. C. Griffith (1971) Scientific communication: Its role in the conduct of research and creation of knowledge. American Psychologist 26:349–62. [DdB, BCG, taDPP]
W. Garvey ; N. Lin & C. Nelson (1970) Communication in the physical and social sciences. Science 170:1166–73. [BCG]
J. C. Gibbs (1979) The meaning of ecologically oriented inquiry in contemporary psychology. American Psychologist 34:127–40. [taDPP]
L. D. Goodstein & K. L. Brazis (1970) Credibility of psychologists: An empirical study. Psychological Reports 27:835–38. [JSA, taDPP]
W. R. Gove (1979) The review process and its consequences in the major sociology journals. Contemporary Sociology 8:799–804. [taDPP]
W. O. Hagstrom (1974) Competition in science. American Sociological Review 39:1–18. [CM]
S. Harnad (1979) Creative disagreement. Sciences 19:18–20. [DVC, Ed., WMH, taDPP]
R. G. Hawkins ; L. S. Ritter & I. Walter (1973) What economists think of their journals. Journal of Political Economy 81:1017–32. [JSA]
C. Hendrick (1976) Editorial comment. Personality and Social Psychology Bulletin 2:207–8. [GJW]
C. Hendrick (1977) Editorial comment. Personality and Social Psychology Bulletin 3:1–2. [taDPP]
R. J. Herrnstein (1977) Doing what comes naturally: A reply to Professor Skinner. American Psychologist 32:1013–16. [taDPP]
D. F. Horrobin (1974) Referees and research administrators: Barriers to scientific research? British Medical Journal 2:216–18. [CM]
J. Hall (1979) Author review of reviewers. American Psychologist 34:798. [taDPP, GJW]
C. Holden (1979) Ethics and social science research. Science 206:357–340. [rDPP]
C. Holden (1980) Not what you know, but where you're from. Science 209:1097. [SP]
F. J. Ingelfinger (1974) Peer review in biomedical publication. American Journal of Medicine 56:686–92. [taDPP]
S. Kerr ; J. Tolliver & D. Petree (1977) Manuscript characteristics which influence acceptance for management and social science journals. Academy of Management Journal 20:132–41. [JSA]
D. Koulack & H. J. Keselman (1975) Ratings of psychology journals by members of the American Psychological Association. American Psychologist 30:1049–53. [taDPP]
K. Kumar (1979) Optional published refereeing. Physics Today 32:13–14. [RAG]
D. Lazarus (1980) Changes in “The Physical Review” and “Physical Review Letters.” Physical Review Letters 45:1605–6. [RAG]
A. C. Leopold (1978) The act of creation: Creative processes in science. BioScience 28:436–40. [CM]
H. Levenson ; B. Burford ; B. Bonno & L. Davis (1975) Are women still prejudiced against women? A replica and extension of Goldberg's study. Journal of Psychology 89:67–71. [RO]
L. S. Lewis (1972) Academic freedom cases and their disposition. Change 4:8, 77. [CM]
R. B. McCall (1977) Challenges to a science of developmental psychology. Child Development 48:333–44. [taDPP]
F. J. McGuigan (1968) Experimental psychology: A methodological approach. 2d ed.Englewood Cliffs, N.J.: Prentice-Hall. [BM]
W. J. McGuire (1973) The yin and yang of social psychology: Seven koan. Journal of Personality and Social Psychology 26:446–56. [taDPP]
P. McReynolds (1971) Reliability of ratings of research papers. American Psychologist 26:400–101. [DP, taDPP]
M. J. Mahoney (1977) Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive. Therapy and Research 1:161–75. [JSA, MJM, RO, taDPP]
M. J. Mahoney (1979) Psychology of the scientist: An evaluative review. Social Studies of Science 9:349–75. [JSA, DEC, CM]
M. J. Mahoney ; A. E. Kazdin & M. Kenigsberg (1978) Getting published: The effects of self-citation and institutional affiliation. Cognitive Therapy and Research 2:69–70. [JSA, MJM, RO, SP]
C. Manwell & C. M. A. Baker (1979) The double helix: Science and myth in the act of creation. BioScience 29:742–46. [CM]
R. K. Merton (1968a) The Matthew Effect in science. Science 159:56–63. [DdB, CM]
I. I. Mitroff & D. E. Chubin (1979) Peer review at NSF: A dialectical policy analysis. Social Studies of Science 9:199–232. [DEC, CM]
M. Moore (1978) Discrimination or favoritism? Sex bias in book reviews. American Psychologist 33:936–38. [RO]
R. E. Nisbett & T. D. Wilson (1977) The halo effect: Evidence for unconscious alterations of judgments. Journal of Personality and Social Psychology 35:250–56. [RO]
M. Oromaner (1977) Professional age and the reception of sociological publications: A test of the Zuckerman-Merton hypothesis. Social Studies of Science 7:381–88. [taDPP]
R. Over (1981) Representation of women on the editorial boards of psychology journals. American Psychologist 36:885–91. [RO]
E. H. Patterson (1969) Evaluation of manuscripts submitted for publication. American Psychologist 24:73. [DVC]
D. P. Peters , & S. J. Ceci (1980) A manuscript masquerade. How well does the review process work? Sciences 20:16–19. [JJB]
J. Pfeffer ; A. Leong & K. Strehl (1977) Paradigm development and particularism: Journal publication in three scientific disciplines. Social Forces 55:938–51. [JMB]
G. I. Pheterson ; S. B. Kiesler & P. A. Goldberg (1971) Evaluation of the performance of women as a function of their sex, achievement, and personal history. Journal of Personality and Social Psychology 19:114–18. [RO]
J. R. Ravetz (1981) Avoiding fraud (correspondence). Nature 291:7. [CM]
S. Revusky (1977) Interference with progress by the scientific establishment: Examples from flavor aversion learning. In: Food aversion learning, ed. N. W. Milgram , L. Knames & T. M. Alloway . London: Plenum. [JH, taDPP, CJT]
A. Rosenblatt & S. A. Kirk (1980) Recognition of authors in blind review of manuscripts. Journal of Social Service Research 3:383–94. [tarDPP]
J. A. Rowney & T. J. Zenisek (1980) Manuscript characteristics influencing reviewers' decisions. Canadian Psychology 21:17–21. [DP]
R. Roy (1981) An alternative funding mechanism. Science 211:1377. [DEC]
J. P. Rushton & H. L. Roediger (1978) An evaluation of 80 psychology journals based on the Science Citation Index. American Psychologist 33:520–23. [taDPP]
T. Saracevic (1975) Relevance: A review and framework for thinking on the notion in information science. Journal of the American Society for Information Science 26:321–13. [BCG]
S. Scarr & B. L. R. Weber (1978) The reliability of reviews for the American Psychologist. American Psychologist 33:935. [tarDPP, GJW, JCW]
D. L. Schaeffer (1970) Do APA journals play professional favorites? American Psychologist 25:362–65. [JSA]
W. A. Scott (1974) Interreferee agreement on some characteristics of manuscripts submitted to the Journal of Personality and Social Psychology. American Psychologist 29:698–702. [DVC, taDPP, WAS, GJW]
J. J. Siegfried (1970) A first lesson in econometrics. Jottrnal of Political Economy 78:1378–79. [JSA]
W. E. Stumpf (1980) “Peer” review. Science 207:822–23. [taDPP]
H. P. Tuckman & J. Leaky (1975) What is an article worth? Journal of Political Economy 83:951–67. [RC]
G. W. Walster & T. A. deary (1970) A proposal for a new editorial policy in the social sciences. American Statistician 24:16–19. [taDPP]
R. J. Weiss (1980). The use and abuse of deception. American Journal of Public Health 70:1097–98. [JLF, rDPP]
M. J. White & K. G. White (1977) Citation analysis of psychology journals. American Psychologist 32:301–5. [tarDPP, MJW]
W. M. Wolff (1973) Publication problems in psychology and an explicit evaluation schema for manuscripts. American Psychologist 28:257–61. [taDPP]
L. Wolin (1962) Responsibility for raw data. American Psychologist 17:657–58. [JSA]
W. Yoels (1974) The structure of scientific fields and the allocation of editorship on scientific journals: Some observations on the politics of knowledge. Sociological Quarterly 15:264–76. [JMB]
D. S. Zinberg (1976) Education through science: The early stages of career development in chemistry. Social Studies of Science 6:215–46. [CM]
H. Zuckerman (1970) Stratification in American science. Sociological Inquiry 40:235–57. [taDPP]
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Behavioral and Brain Sciences
  • ISSN: 0140-525X
  • EISSN: 1469-1825
  • URL: /core/journals/behavioral-and-brain-sciences
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
Request permission

Keywords:

Cancel
Confirm
×