Skip to main content
    • Aa
    • Aa
  • Get access
    Check if you have access via personal or institutional login
  • Cited by 12
  • Cited by
    This article has been cited by the following publications. This list is generated based on data provided by CrossRef.

    Bornmann, Lutz 2015. Interrater reliability and convergent validity of F1000Prime peer review. Journal of the Association for Information Science and Technology, Vol. 66, Issue. 12, p. 2415.

    Di Benedetto, Annalisa and Fasanella, Antonio 2015. La valutazione dei pari nelle scienze sociali e politiche. la lezione della VQR 2004-2010. SOCIOLOGIA E POLITICHE SOCIALI, Issue. 2, p. 44.

    Paolucci, Mario and Grimaldo, Francisco 2014. Mechanism change in a simulation of peer review: from junk support to elitism. Scientometrics, Vol. 99, Issue. 3, p. 663.

    Bornmann, Lutz 2011. Scientific peer review. Annual Review of Information Science and Technology, Vol. 45, Issue. 1, p. 197.

    Bornmann, Lutz and Daniel, Hans-Dieter 2008. The Effectiveness of the Peer Review Process: Inter-Referee Agreement and Predictive Validity of Manuscript Refereeing atAngewandte Chemie. Angewandte Chemie International Edition, Vol. 47, Issue. 38, p. 7173.

    Bornmann, Lutz and Daniel, Hans-Dieter 2008. Die Effektivität des Peer-Review-Verfahrens: Übereinstimmungsreliabilität und Vorhersagevalidität der Manuskriptbegutachtung bei derAngewandten Chemie. Angewandte Chemie, Vol. 120, Issue. 38, p. 7285.

    Hentschel, Uwe and Pokorny, Dan 2008. A Study of the Intersubjective Reviewer Agreement and Manuscript Improvement During the Evaluation Process of Journal Submissions. Behavioral & Social Sciences Librarian, Vol. 27, Issue. 2, p. 92.

    Turcotte, Claudine Drolet, Pierre and Girard, Michel 2004. Study design, originality and overall consistency influence acceptance or rejection of manuscripts submitted to the Journal. Canadian Journal of Anesthesia/Journal canadien d'anesthésie, Vol. 51, Issue. 6, p. 549.

    Cicchetti, Domenic V. 1993. The reliability of peer review for manuscript and grant submissions: “It's like déjà vu all over again!”. Behavioral and Brain Sciences, Vol. 16, Issue. 02, p. 401.

    Crothers, Charles 1993. Peer review reliability: The hierarchy of the sciences. Behavioral and Brain Sciences, Vol. 16, Issue. 02, p. 398.

    Luce, R. Duncan 1993. Reliability is neither to be expected nor desired in peer review. Behavioral and Brain Sciences, Vol. 16, Issue. 02, p. 399.

    Sinclair, J. D. 1993. Drop censorship in science. Behavioral and Brain Sciences, Vol. 16, Issue. 02, p. 400.


When nonreliability of reviews indicates solid science

  • Douglas Lee Eckberg (a1)
  • DOI:
  • Published online: 01 May 2011
Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

S. I. Abramowitz B. Gomes & C. V. Abrarnowitz (1975) Publish or politic: Referee bias in manuscript review. Journal of Applied Social Psychology 5:187200. [aDVC]

E. M. Allen (1960) Why are research grant applications disapproved? Science 132:1532–34. [aDVC]

J. S. Armstrong (1982b) The ombudsman: Is peer review by peers as fair as it appears? Interfaces 12:6274. [aDVC, JSA]

J. J. Bartko (1976) On various intraclass correlation reliability coefficients. Psychological Bulletin 83:762–65. [aDVC]

G. S. Bernstein (1984) Scientific rigor, scientific integrity: A comment on Sommer & Sommer. American Psychologist 39:1316. [aDVC]

J. M. Beyer (1978) Editorial policies and practices among leading journals in four scientific fields. The Sociological Quarterly 19:6888. [aDVC]

R. F. Bornstein (1990) Manuscript review in psychology: An alternative model. American Psychologist 45:672–73. [RFB]

L. D. Braida & N. T. Durlach (1972) Intensity perception. II. Resolution in one interval paradigms. Journal of the Acoustical Society of America 51:483502. [DL]

R. L. Brennan & R. J. Light (1974) Measuring agreement when two observers classify people into categories not defined in advance. British Journal of Mathematical and Statistical Psychology 27:154–63. [rDVC]

R. J. Brook & W. D. Stirling (1984) Agreement between observers when the categories are not specified in advance. British Journal of Mathematical and Statistical Psychology 37:271–82. [rDVC]

J. P. Campbell (1982) Some remarks from the outgoing editor. Journal of Applied Psychology 67:691700. [LLH]

S. J. Ceci & D. Peters (1984) How blind is blind review? American Psychologist 39:1491–94. [aDVC]

D. E. Chubin (1982) Reform of peer review. Science 215:40. [aDVC]

D. V. Cicchetti (1976) Assessing interrater reliability for rating scales: Resolving some basic issues. British Journal of Psychiatry 129:452–56. [aDVC].

D. V. Cicchetti & A. R. Feinstein (1990) High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology 43:551–68. [arDVC]

D. V. Cicchetti & J. L. Fleiss (1977) Comparison of the null distributions of weighted kappa and the C ordinal statistic. Applied Psychological Measurement 1:195201. [aDVC]

D. V. Cicchetti S. L. Aivano & J. Vitale (1976) A computer program for assessing the reliability and systematic bias of individual measurements. Educational and Psychological Measurement 36:761–64. [aDVC]

D. V. Cicchetti S. L. Aivano & J. Vitale (1977) Computer programs for assessing rater agreement and rater bias for qualitative data. Educational and Psychological Measurement 37:195201. [aDVC]

D. V. Cicchetti C. Lee A. F. Fontana & B. N. Dowds (1978) A computer program for assessing specific category-rater agreement for qualitative data. Educational and Psychological Measurement 38:805–13. [aDVC]

J. Cohen (1968) Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 70:213–20. [aDVC]

J. Cohen (1988) Statistical power analysis for the behavioral sciences, 2nd ed. Lawrence Erlbaum. [rDVC, RR]

S. Cole J. Cole & G. A. Simon (1981) Chance and consensus in peer review. Science 214:881–86. [aDVC, SC, LLH]

S. Cole G. Simon & J. Cole (1988) Do journal rejection rates index scientific consensus? American Sociological Review 53:152–56. [SC]

A. J. Conger (1985) Kappa reliabilities for continuous behaviors and events. Educational and Psychological Measurement 45:861–68. [rDVC]

J. Diamond (1985) Variations on a theme. Nature 314:222–23. [aDVC]

W. K. Estes (1975) Some targets for mathematical psychology. Journal of Mathematical Psychology 12:263–82. [PHS]

J. T. Evans H. I. Nadjari & S. A. Burchell (1990) Quotational and reference accuracy in surgical journals: A continuing peer-review problem. Journal of the American Medical Association 263:1353–54. [JSA].

A. R. Feinstein & D. V. Cicchetti (1990) High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology 43:543–49. [arDVC]

D. W. Fiske & L. Fogg (1990) But the reviewers are making different criticisms of my paper!: Diversity and uniqueness in reviewer comments. American Psychologist 45:591–98. [rDVC, JSA]

J. L. Fleiss & D. V. Cicchetti (1978) Inference about weighted kappa in the non-null case. Applied Psychological Measurement 2:113–17. [rDVC]

J. L. Fleiss & J. Cuzick (1979) The reliability of dichotomous judgments: Unequal numbers of judges per subject. Applied Psychological Measurement 3:537–52. [rDVC]

C. Freeman & P. Tyrer , eds. (1989) Research methodology in psychiatry: A beginners guide. Royal College of Psychiatrists/Gaskell Books. [PT]

L. D. Goodstein & K. L. Brazis (1970) Credibility of psychologists: An empirical study. Psychological Reports 27:835–38. [aDVC, JSA]

A. G. Greenwald (1975) Consequences of prejudice against the null hypothesis. Psychological Bulletin 82:120. [aDVC, PHS]

H. O. Gulliksen (1950) Theory of mental tests. Wiley. [DL, LJS]

G. H. Guyatt , M. Townsend & L. Berman (1987) A comparison of Likert and visual analogue scales for measuring change in function. Journal of Chronic Diseases 40:1129–33. [rDVC]

S. Harnad (1979) Creative disagreement. The Sciences 19:1820. [aDVC]

C. Hendrick (1976) Editorial comment. Personality and Social Psychology Bulletin 2:27–08. [aDVC]

K. Heskin (1984) The Milwaukee Project: A cautionary comment. American Psychologist 39:1316–17. [aDVC]

F. J. Ingelfinger (1975) Charity and peer review in publication. New England Journal of Medicine 293:1371–72. [aDVC]

J. R. Ison (1985) The granting system and healthy research. Science 230:376. [aDVC]

S. Kerr , J. Tolliver & D. Petree (1977) Manuscript characteristics which influence acceptance for management and social science journals. Academy of Management Journal 20:132–41. [aDVC]

J. Klayman & Y.-W. Ha (1987) Confirmation, discontinuation and information in hypothesis testing. Psychological Review 94:211–28. [MEG]

L. M. Koran (1975b) The reliability of clinical methods, data, and judgments. New England Journal of Medicine 293:695701. [rDVC]

K. Krippendorff (1970) Bivariate agreement coefficients for reliability of data. In: Sociological methodology, ed. E. G. Borgatta . Jossey-Bass. [aDVC]

D. Laming (1990) The reliability of a certain university examination compared with the precision of absolute judgments. Quarterly Journal of Experimental Psychology 42A:239–54. [DL]

G. F. Lawlis & E. Lu (1972) Judgment of counseling process: Reliability, agreement, and error. Psychological Bulletin 78:1720. [aDVC]

D. Lindsey (1988) Assessing precision in the manuscript review process: A little better than a dice roll. Scientometrics 14:7582. [LLH]

R. D. Luce , R. M. Nosofsky , D. M. Green & A. F. Smith (1982) The bow and sequential effects in absolute identification. Perception & Psychophysics 32:397408. [DL].

M. J. Mahoney (1977) Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy Research 1:161–75. [aDVC, LDN, PHS, SPL, JSA].

H. W. Marsh & S. Ball (1981) Interjudgmental reliability of reviews for the Journal of Educational Psychology. Journal of Educational Psychology 73:872–80. [aDVC], HWM

P. L. McCarthy , S. D. Sznajderman , K. Lustman-Findling , M. A. Baron , H. D. Fink , N. Czarkowski , H. Bauchner , B. C. Forsyth & D. V. Cicchetti (1990) Mothers' clinical judgment: A randomized trial of the acute illness observation scales. Journal of Pediatrics 116:200–06. [rDVC]

J. E. Mezzich , H. C. Kraemer , D. R. L. Worthington & G. A. Coffman (1981) Assessment of agreement among several raters formulating multiple diagnoses. Journal of Psychiatric Research 16:2939. [rDVC]

R. J. L. Murphy (1978) Reliability of marking in eight GCE examinations. British Journal of Educational Psychology 48:196200. [DL]

S. H. Newman (1966) Improving the evaluation of submitted manuscripts. American Psychologist 21:980–81. [aDVC]

I. Pollack (1952) The information of elementary auditory displays. Journal of the Acoustical Society of America 24:745–49. [DL]

W. S. Robinson (1957) The statistical measurement of agreement. American Sociological Review 22:1725. [aDVC]

N. S. Rosenfield , R. C. Ablow , R. I. Markowitz , M. DiPietro , J. H. Seashore , R. J. Touloukian & D. V. Cicchetti (1984) Hirschsprung Disease: Accuracy of the barium enema examination. Radiology 150:393400. [rDVC]

B. P. Rourke & L. Costa (1979) Editorial policy II. Journal of Clinical Neuropsychology 1:9396. [aDVC]

P. H. Schönemann (1989) New questions about old heritability estimates. Bulletin of the Psychonomic Society 27:175–78. [PHS]

H. E. A. Tinsley & D. J. Weiss (1975) Interrater reliability and agreement of subjective judgments. Journal of Counseling Psychology 22:358–76. [LLH]

P. Tyrer , D. V. Cicchetti , P. R. Casey , K. Fitzpatrick , R. Oliver , A. Baiter , E. Ciller & L. Harkness (1984) Cross-national reliability study of a schedule for assessing personality disorders. The Journal of Nervous and Mental Disease 172:718–21. [rDVC]

F. R. Volkmar , D. V. Cicchetti , E. Dykens , S. S. Sparrow , J. F. Leckman & D. J. Cohen (1988) An evaluation of the Autism Behavior Checklist. Journal of Autism and Developmental Disorders 18:8197. [rDVC]

W. M. Wolff (1973) Publication problems in psychology and an explicit evaluation schema for manuscripts. American Psychologist 28:257–61. [aDVC]

Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Behavioral and Brain Sciences
  • ISSN: 0140-525X
  • EISSN: 1469-1825
  • URL: /core/journals/behavioral-and-brain-sciences
Please enter your name
Please enter a valid email address
Who would you like to send this to? *