Skip to main content

The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation

  • Domenic V. Cicchetti (a1)

The reliability of peer review of scientific documents and the evaluative criteria scientists use to judge the work of their peers are critically reexamined with special attention to the consistently low levels of reliability that have been reported. Referees of grant proposals agree much more about what is unworthy of support than about what does have scientific value. In the case of manuscript submissions this seems to depend on whether a discipline (or subfield) is general and diffuse (e.g., cross-disciplinary physics, general fields of medicine, cultural anthropology, social psychology) or specific and focused (e.g., nuclear physics, medical specialty areas, physical anthropology, and behavioral neuroscience). In the former there is also much more agreement on rejection than acceptance, but in the latter both the wide differential in manuscript rejection rates and the high correlation between referee recommendations and editorial decisions suggests that reviewers and editors agree more on acceptance than on rejection. Several suggestions are made for improving the reliability and quality of peer review. Further research is needed, especially in the physical sciences.

Hide All
Abelson P. H. (1980) Scientific communication. Science 209:6062. [aDVC]
Abramowitz S. I.Gomes B. & Abrarnowitz C. V. (1975) Publish or politic: Referee bias in manuscript review. Journal of Applied Social Psychology 5:187200. [aDVC]
Abt H. A. (1988) What happens to rejected astronomical papers? Publications of the Astronomical Society of the Pacific 100:506–08. [aDVC]
Ad Hoc Working Group for Critical Appraisal of the Medical Literature (1987) A proposal for more informative abstracts of clinical articles. Annals of Internal Medicine 106:598604. [SPL]
Adair R. K. (1981) Anonymous refereeing. Physics Today 34:1315. [aDVC]
Adair R. K. (1982) A physics editor comments on Peters & Ceci's peer-review study. Behavioral and Brain Sciences 5:196. [aDVC]
Adair R. K. & Trigg G. L. (1979) Editorial: Should the character of Physical Review Letters be changed? Physical Review Letters 43:1969–74. [aDVC]
Allen E. M. (1960) Why are research grant applications disapproved? Science 132:1532–34. [aDVC]
Amabile T. M. (1983) Brilliant but cruel: Perceptions of negative evaluators. Journal of Experimental Social Psychology 19:146–56. [RC]
American Psychological Association (1983) Publication manual, 3rd ed. [aDVC]
American Psychological Association (1985) Standards for educational and psychological testing. [RFB]
American Psychologist (1989) Members of underrepresented groups: Reviewers for journal manuscripts wanted. American Psychologist 44:1555. [RC]
Anonymous (1987) The publication game: Beyond quality in the search for a lengthy vitae. Journal of Social Behavior and Personality 2:312 [RC]
Armstrong J. S. (1980) Unintelligible management research and academic prestige. Interfaces 10:8086. [aDVC]
Armstrong J. S. (1982a) Barriers to scientific contributions: The author's formula. Behavioral and Brain Sciences 5:197–99. [aDVC]
Armstrong J. S. (1982b) The ombudsman: Is peer review by peers as fair as it appears? Interfaces 12:6274. [aDVC, JSA]
Armstrong J. S. (1982c) Research on scientific journals: Implications for editors and authors. Journal of Forecasting 1:83104. [aDVC, JSA]
Bailar J. C. III & Patterson K. (1985) Journal peer review: The need for a research agenda. The New England Journal of Medicine 312:654–57. [aDVC]
Baird J. C.Green D. M. and Luce R. D. (1980) Variability and sequential effects in cross-modality matching of area and loudness. Journal of Experimental Psychology: Human Perception and Performance 6:277–89. [DL]
Bakanic V.McPhail C. & Simon R. J. (1987) The manuscript review and decision-making process. American Sociological Report 52:631–42. [aDVC, LJS]
Bartko J. J. (1966) The intraclass correlation coefficient as a measure of reliability. Psychological Reports 19:311. [aDVC]
Bartko J. J. (1974) Corrective note to: “The Intraclass Correlation Coefficient as a Measure of Reliability.” Psychological Reports 34:418. [aDVC]
Bartko J. J. (1976) On various intraclass correlation reliability coefficients. Psychological Bulletin 83:762–65. [aDVC]
Bartko J. J. & Carpenter W. T. (1976) On the methods and theory of reliability. Journal of Nervous and Mental Disease 163:307–17. [aDVC]
Beck A. T. (1976) Cognitive therapy and the emotional disorders. International Universities Press. [aDVC]
Benwell R. (1979) Authors anonymous? Physics Bulletin 30:288. [aDVC]
Berelson B. (1960) Graduate education in the United States. McGraw-Hill. [aDVC]
Bernstein G. S. (1984) Scientific rigor, scientific integrity: A comment on Sommer & Sommer. American Psychologist 39:1316. [aDVC]
Beyer J. M. (1978) Editorial policies and practices among leading journals in four scientific fields. The Sociological Quarterly 19:6888. [aDVC]
Blashfield R. K. (1976) Mixture model tests of cluster analysis: Accuracy of four agglomerative hierarchical methods. Psychological Bulletin 83:377–88. [rDVC]
Bloch D. A. & Kraemer H. C. (1989) 2X2 kappa coefficients: Measures of agreement or association. Biometrics 45:269–87. [HCK]
Boehme G. (1977) Models for the development of science. In: Science, technology, and society: A cross-disciplinary perspective, ed. Spiegel-Rosing I. & de Solla Price D.. Sage. [aDVC]
Boehme G.van der Daele W. & Krohn W. (1976) Finalization of science. Social Science Information 15:306–30. [aDVC]
Boor M. (1986) Suggestions to improve manuscripts submitted to professional journals. American Psychologist. 41:721–22. [RC]
Bornstein R. F. (1990) Manuscript review in psychology: An alternative model. American Psychologist 45:672–73. [RFB]
Bornstein R. F. (in press) Publication politics, experimenter bias, and the replication process in social science research. Journal of Social Behavior and Personality. [RFB].
Bowen D. D.Perloff R. & Jacoby J. (1972) Improving manuscript evaluation procedures. American Psychologist 25:221–25. [aDVC]
Bozarth H. D. & Roberts R. R. Jr. (1972) Signifying significant significance. American Psychologist 27:774–75. [aDVC]
Bradley J. V. (1981) Pernicious publication practices. Bulletin of the Psychonomic Society 18:3134. [aDVC]
Braida L. D. & Durlach N. T. (1972) Intensity perception. II. Resolution in one interval paradigms. Journal of the Acoustical Society of America 51:483502. [DL]
Brennan R. L. & Light R. J. (1974) Measuring agreement when two observers classify people into categories not defined in advance. British Journal of Mathematical and Statistical Psychology 27:154–63. [rDVC]
Broad W. J. (1988) Science can't keep up with the flood of new journals. The New York Times, Feb. 16:C1, Cll. [JF]
Brook R. J. & Stirling W. D. (1984) Agreement between observers when the categories are not specified in advance. British Journal of Mathematical and Statistical Psychology 37:271–82. [rDVC]
Byrne C. (1980) Tutor marked assessments at the Open University: A question of reliability. Assessment in Higher Education 5:104–18. [DL]
Campbell J. P. (1982) Some remarks from the outgoing editor. Journal of Applied Psychology 67:691700. [LLH]
Carsrud K. B. (1984) Out of the frying pan: A reply to Sommer & Sommer. American Psychologist 31:1317–18. [aDVC]
Ceci S. J. & Peters D. (1984) How blind is blind review? American Psychologist 39:1491–94. [aDVC]
Chalmers I. (1990) Underreporting research is scientific misconduct. Journal of the American Medical Association 263:1405–08. [SPL]
Chalmers T. C.Frank C. S. & Reitman D. (1990) Minimizing the three stages of publication bias. Journal of the American Medical Association 263:1392–95. [SPL]
Chase J. M. (1970) Normative criteria for scientific publication. American Sociologist 5:262–65. [aDVC]
Chubin D. E. (1982) Reform of peer review. Science 215:40. [aDVC]
Cicchetti D. V. (1976) Assessing interrater reliability for rating scales: Resolving some basic issues. British Journal of Psychiatry 129:452–56. [aDVC].
Cicchetti D. V. (1980)Reliability of reviews for the American Psychologist: A biostatistical assessment of the data. American Psychologist 35:300–3. [aDVC, LJS]
Cicchetti D. V. (1980)Testing the normal approximation and minimal sample size requirements of weighted kappa when the number of categories is large. Applied Psychological Measurement 5:101–04. [arDVC]
Cicchetti D. V. (1980) On peer review: “We have met the enemy and he is us.” Behavioral and Brain Sciences 5:205. [arDVC]
Cicchetti D. V. (1985) A critique of Whitehurst's “Interrater agreement for journal manuscript reviews:” De omnibus, disputandum est. American Psychologist 40:563–68. [aDVC, MED]
Cicchetti D. V. (1988) When diagnostic agreement is high, but reliability is low: Some paradoxes occurring in independent neuropsychological assessments. Journal of Clinical and Experimental Neuropsychology 10:605–22. [aDVC]
Cicchetti D. V. & Conn H. O. (1976) A statistical analysis of reviewer agreement and bias in evaluating medical abstracts. Yale Journal of Biology and Medicine 45:373–83. [aDVC]
Cicchetti D. V. & Conn H. O. (1978) Reviewer evaluation of manuscripts submitted to medical journals. Paper presented to the American Statistical Association Meetings, san Diego, CA. (also abstracted in Biometrics [1978] 34:728) [aDVC]
Cicchetti D. V. & Eron L. D. (1979) The reliability of manuscript reviewing for the Journal of Abnormal Psychology. Proceedings of the American Statistical Association (Social Statistics Section) 22:596600. [aDVC]
Cicchetti D. V. & Feinstein A. R. (1990) High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology 43:551–68. [arDVC]
Cicchetti D. V. & Fleiss J. L. (1977) Comparison of the null distributions of weighted kappa and the C ordinal statistic. Applied Psychological Measurement 1:195201. [aDVC]
Cicchetti D. V. & Heavens R. (1979) RATCAT (Rater Agreement/Categorical Data). American Statistician 33:91. [aDVC]
Cicchetti D. V. & Showalter D. (1988) A computer program for determining the reliability of dimensionally scaled data when the numbers and specific sets of examiners may vary at each assessment. Educational and Psychological Measurement 48:717–20. [aDVC]
Cicchetti D. V. & Sparrow S. S. (1981) Developing criteria for establishing interrater reliability of specific items: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency 86:127–37. [aDVC]
Cicchetti D. V. & Tyrer P. (1988) Reliability and validity of personality assessment. In: Personality disorders: Diagnosis, management and course, ed. Tyrer P. L.. Butterworth Scientific Ltd. [rDVC]
Cicchetti D. V.Aivano S. L. & Vitale J. (1976) A computer program for assessing the reliability and systematic bias of individual measurements. Educational and Psychological Measurement 36:761–64. [aDVC]
Cicchetti D. V.Aivano S. L. & Vitale J. (1977) Computer programs for assessing rater agreement and rater bias for qualitative data. Educational and Psychological Measurement 37:195201. [aDVC]
Cicchetti D. V.Lee C.Fontana A. F. & Dowds B. N. (1978) A computer program for assessing specific category-rater agreement for qualitative data. Educational and Psychological Measurement 38:805–13. [aDVC]
Cicchetti D. V.Sharma Y. & Cotlier E. (1982) Assessment of observer variability in the classification of human cataracts. Yale Journal of Biology and Medicine 55:8188. [rDVC]
Cicchetti D. V.Showalter D. & Tyrer P. (1985) The effect of number of rating-scale categories upon levels of interrater reliability: A Monte Carlo investigation. Applied Psychological Measurement 9:3136. [aDVC]
Cleary F. R. & Edwards D. J. (1960) The origins of the contributors to the A.E.R. during the fifties. American Economic Review 50:1011–14. [aDVC]
Cohen J. (1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20:3746. [aDVC, RR]
Cohen J. (1968) Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 70:213–20. [aDVC]
Cohen J. (1988) Statistical power analysis for the behavioral sciences, 2nd ed. Lawrence Erlbaum. [rDVC, RR]
Cole J. & Cole S. (1973) Social stratification in science. University of Chicago Press. [aDVC]
Cole J. & Cole S. (1981) Peer review in the National Science Foundation: Phase U of a study. National Academy of Sciences. [aDVC]
Cole J. & Cole S. (1985) Experts' “consensus” and decision-making at the National Science Foundation. In: Selectivity in information systems: Survival of the fittest, ed. Warren K. S.. Praeger. [aDVC]
Cole S. (1978) Scientific reward systems: A comparative analysis. In: Research in sociology of knowledge, sciences, and art, ed. Jones R. A.. JAI Press. [aDVC]
Cole S. (1983) The hierarchy of the sciences. American Journal of Sociology 89:111–39. [aDVC], SC
Cole S.Cole J. & Simon G. A. (1981) Chance and consensus in peer review. Science 214:881–86. [aDVC, SC, LLH]
Cole S.Cole J. & Dietrich L. (1978) Measuring the cognitive state of scientific disciplines. In: Toward a metric of science: The advent of science indicators, ed. Elkana Y.Lederberg J.Merton R. K.Thackray A. & Zuckerman J.. Wiley. [SC]
Cole S.Rubin L. & Cole J. (1978) Peer review in the National Science Foundation. National Academy of Sciences. [aDVC]
Cole S.Simon G. & Cole J. (1988) Do journal rejection rates index scientific consensus? American Sociological Review 53:152–56. [SC]
Colman A. M. (1982a) Game theory and experimental games: The study of strategic interaction. Pergamon Press. [AMC]
Colman A. M. (1982b) Manuscript evaluation by journal referees and editors: Randomness or bias? Behavioral and Brain Sciences 5:205–06. [AMC]
Conger A. J. (1980) Integration and generalization of Kappa for multiple raters. Psychological Bulletin 88:322–28. [rDVC]
Conger A. J. (1985) Kappa reliabilities for continuous behaviors and events. Educational and Psychological Measurement 45:861–68. [rDVC]
Conn H. O. (1974) An experiment in blind program selection. Clinical Research 22:128–34. [aDVC]
Cotlier E.Fagadau W. & Cicchetti D. V. (1982) Methods for evaluation of medical therapy of senile and diabetic cataracts. Transactions of the Opthalmologic Societies of the United Kingdom 102:416–22. [rDVC]
Cox R. (1967) Examinations and higher education: A survey of the literature. Universities Quarterly 21:292340. [DL]
Grandall R. (1986) Peer review: Improving editorial procedures. Bio Science 36:607–09. [RC]
Grandall R. (1987a) Gauntlet thrown: Publication procedures are challenged. Dialogue (APA Division 8) 1:5. [RC]
Grandall R. (1987b) We need research on what constitutes good journal papers - and good editing - not guesswork on how to improve manuscripts! American Psychologist 42:407–08. [RC]
Grandall R. (1990) Improving editorial procedures. American Psychologist 45:665–66. [RC]
Crane D. (1967) The gatekeepers of science: Some factors affecting the selection of articles for scientific journals. American Sociologist 32:195201. [aDVC]
Crane D. (1972) Invisible colleges. University of Chicago Press. [DLE]
Cronbach L. J. (1981) Comment on “Chance and consensus in peer review.” Science 214:1293. [LLH]
Culliton B. J. (1984) Fine-tuning peer review. Science 226:1401–02. [aDVC, RG]
Darley J. M. & Latane B. (1968) Bystander intervention in emergencies: Diffusion of responsibility. Journal of Personality and Social Psychology 8:337–83. [AMC]
Darlington R. (1980) Another peek in the file drawers (unpublished manuscript). [PHS]
Davies M. & Fleiss J. L. (1982) Measuring agreement for multinomial data. Biometrics 38:1047–51. [rDVC]
DeBakey L. & DeBakey S. (1976) Impartial, signed reviews. New England Journal of Medicine 294:564. [aDVC]
Delucchi K. L. (1983) The use and misuse of chi-square: Lewis and Burke revisited. Psychological Bulletin 94:166–76. [rDVC]
Diamond J. (1985) Variations on a theme. Nature 314:222–23. [aDVC]
Dickersin K. (1990) The existence of publication bias and risk factors for its occurrence. Journal of the American Medical Association 263:1385–89. [SPL]
Doherty M. E. & Tweney R. D. (1988) The role of data and feedback error in inference and prediction. Final report for ARI Contract MDA903-85-K-0193. [MEG]
Eckberg D. (1982) Theoretical implications of failure to detect prepublished submissions. Behavioral and Brain Sciences 5:2526. [DLE]
Eells W. C. (1930) Reliability of reported grading of essay type examinations. Journal of Educational Psychology 21:4852. [DL]
Eichorn D. H. & VandenBos G. R. (1985) Dissemination of scientific and professional knowledge. American Psychologist 40:1301–16. [RFB]
Eight APA journals initiate controversial blind reviewing (1972) APA Monitor, pp. 1, 5. [aDVC]
Epstein W. M. (1990) Confirmatory response bias among social work journals. Science, Technology and Human Values 15:938. [rDVC, MJM]
Estes W. K. (1975) Some targets for mathematical psychology. Journal of Mathematical Psychology 12:263–82. [PHS]
Evans J. T.Nadjari H. I. & Burchell S. A. (1990) Quotational and reference accuracy in surgical journals: A continuing peer-review problem. Journal of the American Medical Association 263:1353–54. [JSA].
Feinstein A. R. (1987) Clinimetrics. Yale University Press. [rDVC]
Feinstein A. R. & Cicchetti D. V. (1990) High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology 43:543–49. [arDVC]
Feynman R. P. (1985) Surely you are joking, Mr. Feynman. Bantam. [PHS]
Finn R. H. (1970) A note on estimating the reliability of categorical data. Educational and Psychological Measurement 30:7176. [aDVC]
Fisher A. (1989) Seeing atoms. Popular Science: 102–07. [JSA]
Fiske D. W. & Fogg L. (1990) But the reviewers are making different criticisms of my paper!: Diversity and uniqueness in reviewer comments. American Psychologist 45:591–98. [rDVC, JSA]
Fleiss J. L. (1971) Measuring nominal scale agreement among many raters. Psychological Bulletin 76:378–82. [rDVC]
Fleiss J. L. (1975) Measuring agreement between two judges on the presence or absence of a trait. Biometrics 31:651–59. [aDVC]
Fleiss J. L. (1981) Statistical methods for rates and proportions, 2nd ed. Wiley. [aDVC, RR]
Fleiss J. L. & Cicchetti D. V. (1978) Inference about weighted kappa in the non-null case. Applied Psychological Measurement 2:113–17. [rDVC]
Fleiss J. L. & Cohen J. (1973) The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement 33:613–19. [aDVC]
Fleiss J. L. & Cuzick J. (1979) The reliability of dichotomous judgments: Unequal numbers of judges per subject. Applied Psychological Measurement 3:537–52. [rDVC]
Fleiss J. L.Cohen J. & Everitt B. S. (1969) Large sample standard errors of kappa and weighted kappa. Psychological Bulletin 72:323–37. [rDVC]
Fleiss J. L.Nee J. C. M. & Landis J. R. (1979) Large sample variance of kappa in the case of different sets of raters. Psychological Bulletin 86:974–77. [rDVC]
Freeman C. & Tyrer P., eds. (1989) Research methodology in psychiatry: A beginners guide. Royal College of Psychiatrists/Gaskell Books. [PT]
Fuller S. (1989). Philosophy of science and its discontents. Westview Press. [MEG]
Furchtgott E. (1984) Replicate, again and again. American Psychologist 39:1315–16. [aDVC]
Garber H. L. (1984) On Sommer & Sommer. American Psychologist 31:1315. [aDVC]
Garcia C.Rosenfield N. S.Markowitz R. K.Seashore J. H.Touloukian R. J. & Cicchetti D. V. (1987) Appendicitis in children: Accuracy of the barium enema. American Journal of Diseases of Children 141:1309–12. [rDVC]
Gardner M. J.Snee M. P.Hall A. J.Powell C. A.Downes S. & Terrell J. D. (1990) Results of case-control study of leukaemia and lymphoma among young people near Sellafield nuclear plant in West Cumbria. British Medical Journal 300:423–29. [SPL]
Garfield E. (1972) Citation analysis as a tool in journal evaluation. Science 178:471–79. [RFB]
Garfunkel J. M.Ulshen R. H.Hamrick H. J. & Lawson E. E. (1990) Problems identified by secondary review of accepted manuscripts. Journal of the American Medical Association 263:1369–71. [rDVC, SPL]
Garner W. R. (1962) Uncertainty and structure as psychological concepts. Wiley. [DL]
Garner W. R. & McGill W. J. (1956) The relation between information and variance analyses. Psychometrika 21:219–28. [arDVC], JBG
Garvey W. D.Lin N. & Nelson C. E. (1970) Some comparisons of communication activities in the physical and social sciences. In: Communication among scientists and engineers, ed. Nelson C. E. & Pollock D. K.. Health. [SC]
Garvey W. D.Lin N. & Nelson C. E. (1979) Communication in the physical and social sciences. In: Communication: The essence of science, ed. Garvey W. D.. Pergamon Press. [aDVC]
Gholson B. & Barker B. (1985) Kuhn, Lakatos, and Laudan: Applications in the history of physics and psychology. American Psychologist 40:755–69. [aDVC]
Giere R. N. (1988). Explaining science: A cognitive approach. University of Chicago Press. [MEG]
Gillett R. (1985) Nominal scale response agreement and rater uncertainty. British Journal of Mathematical and Statistical Psychology 38:5866. [rDVC]
Gilmore J. B. (1979) Illusory reliability in journal reviewing. Canadian Psychological Review 20:157–58. [arDVC, JBG]
Glenn N. D. (1976) The journal article review process: Some proposals for change. American Sociologist 11:179–85. [aDVC]
Goodman L. A. & Kruskal W. H. (1954) Measures of association for cross classifications. Journal of the American Statistical Association 49:732–64. [aDVC]
Goodrich D. W. (1945) An analysis of manuscripts received by the editors of the American Sociological Review from May 1, 1944, to September 1, 1945. American Sociological Review1 10:716–25. [aDVC]
Goodstein L. D. (1982) When will the editors start to edit? Behavioral and Brain Sciences 5:212–13. [LJS]
Goodstein L. D. & Brazis K. L. (1970) Credibility of psychologists: An empirical study. Psychological Reports 27:835–38. [aDVC, JSA]
Gordon M. D. (1977) Evaluating the evaluators. New Scientist 73:342–43. [aDVC]
Gordon M. D. (1978) A study of the evaluation of papers by primary journals in the U.K. University of Leicester. [LLH]
Gorman M. E. (1986) How the possibility of error affects falsification on a task that models scientific problem-solving. British Journal of Psychology 77:6579. [MEG]
Gorman M. E. (1989) Error, falsification and scientific inference: An experimental investigation. Quarterly Journal of Experimental Psychology, 41A, 385412. [MEG]
Gorman Michael E. & Gorman Margaret E. (1984) A comparison of disconfirmatory, confirmatory and a control strategy on Wason–s 2, 4, 6 task. Quarterly Journal of Experimental Psychology 12:129–40. [MEG]
Gottfredson S. D. (1978) Evaluating psychology research reports: Dimensions, reliability, and correlates of quality judgments. American Psychologist 33:920–34. [aDVC, RFB, JBG]
Green D. M.Luce R. D. & Duncan J. E. (1977) Variability and sequential effects in magnitude production and estimation of auditory intensity. Perception & Psychophysics 22:450–56. [DL]
Green D. M.Luce R. D. & Smith A. F. (1980) Individual magnitude estimates for various distributions of signal intensity. Perception & Psychophysics 27:483–88. [DL]
Greenwald A. G. (1975) Consequences of prejudice against the null hypothesis. Psychological Bulletin 82:120. [aDVC, PHS]
Greenwald A. G. (1976) An editorial. Journal of Personality and Social Psychology 33:17. [aDVC]
Greenwald A. G., Pratkanis A. R., Leippe M. R. & Baumgardner M. H. (1986) Under what conditions does theory obstruct research progress? Psychological Review 93:216–29. [aDVC]
Gross S. T. (1986) The kappa coefficient of agreement for multiple observers when the number of subjects is small. Biometrics 42:883–93. [rDVC].
Grove W. M., Andreasen N. C., McDonald-Scott P., Keller M. B. & Shapiro R. W. (1981) Reliability studies of psychiatric diagnosis: Theory and practice. Archives of General Psychiatry 38:408–13. [rDVC]
Guilford J. P. (1954) Psychometric methods, 2nd ed. McGraw-Hill. [RR]
Gulliksen H. O. (1950) Theory of mental tests. Wiley. [DL, LJS]
Guyatt G. H., Townsend M. & Berman L. (1987) A comparison of Likert and visual analogue scales for measuring change in function. Journal of Chronic Diseases 40:1129–33. [rDVC]
Hall J. A. (1979) Author review of reviewers. American Psychologist 34:798. [aDVC]
Hargens L. L. (1988) Scholarly consensus and journal rejection rates. American Sociological Review 53:139–51. [aDVC, SC]
Hargens L. L. (1990) Variation in journal peer-review systems: Possible causes and consequences. Journal of the American Medical Association 263:1348–52. [arDVC, LLH]
Hargens L. L. & Herting J. R. (1990a) A new approach to referees' assessments of manuscripts. Social Science Research 19:116. [arDVC, LLH]
Hargens L. L. & Herting J. R. (1990b) Neglected considerations in the analysis of agreement among journal referees. Scientometrics 19:91106. [aDVC, LLH]
Harnad S. (1979) Creative disagreement. The Sciences 19:1820. [aDVC]
Hamad S. ed. (1983) Peer commentary on peer review: A case study in scientific quality control. Cambridge University Press (reprinted from Behavioral and Brain Sciences, vol. 5). [aDVC].
Harnad S. (1985)Rational disagreement in peer review. Science, Technology &; Human Values 10(3):5562. [aDVC, LJS].
Harnad S. (1986)Policing the paper chase. Nature 322:2425. [aDVC, JBG].
Hartog P., Rhodes E. C., and Burt C. (1936) The marks of examiners. Macmillan. [DL]
Heavens R. H. Jr. & Cicchetti D. V. (1978) A computer program for calculating rater agreement and bias statistics using contingency table input. Proceedings of the American Statistical Association (Statistical Computing Section) 21:366–70. [aDVC]
Hendrick C. (1976) Editorial comment. Personality and Social Psychology Bulletin 2:27–08. [aDVC]
Hendrick C. (1977) Editorial comment. Personality and Social Psychology Bulletin 3:12. [aDVC].
Hensler D. (1976) Perceptions of the National Science Foundation peer-review process: A report on a survey of NSF reviewers and applicants. NSF publication 77–33. [aDVC]
Heskin K. (1984) The Milwaukee Project: A cautionary comment. American Psychologist 39:1316–17. [aDVC]
Holt V. E. (1985) Research briefings: Peer-review appeals system established. American Psychological Association (APA) Monitor 16:18. [aDVC]
Horrobin D. F. (1990) The philosophical basis of peer review and the suppression of innovation. Journal of the American Medical Association 263:1438–41. [JSA]
Howe M. J. A. (1982) Peer reviewing: Improve or be rejected. Behavioral and Brain Sciences 5:218–19. [aDVC]
Hubbard R. & Armstrong J. S. (1990) Replication and the development of marketing science. Marketing Department working paper, The Wharton School, University of Pennsylvania. [JSA]
Hughes H. M. (1976) Letter to the editor. American Sociologist 11:178–79. [aDVC].
Hull D. L. (1988) Science as a process. University of Chicago Press. [LLH]
Hunt E. (1971) Psychological publications. American Psychologist 26:311. [aDVC]
Hunt K. (1975) Do we really need more replications? Psychological Reports 36:587–93. [aDVC]
Ingelfinger F. J. (1974) Peer review in biomedical publication. American Journal of Medicine 56:686–92. [aDVC]
Ingelfinger F. J. (1975) Charity and peer review in publication. New England Journal of Medicine 293:1371–72. [aDVC]
Ison J. R. (1985) The granting system and healthy research. Science 230:376. [aDVC]
Iyengar S. & Greenhouse J. B. (1988) Selection model and the file drawer hypothesis. Statistical Science 33:109–35. [PHS]
Jesteadt W., Luce R. D. & Green D. M. (1977) Sequential effects in judgment of loudness. Journal of Experimental Psychology: Human Perception and Performance 3:92104. [DL]
Jesteadt W., Wier C. C. & Green D. M. (1977) Intensity discrimination as a function of frequency and sensation level. Journal of the Acoustical Society of America 61:169–77. [DL]
Jonckheere A. R. (1970) Techniques for ordered contingency tables. In: Proceedings of the NUFFIC International Summer Session in Science, Het Oude Hof ed. Riemersma J. B. & van der Meer H. C.. The Hague. [aDVC].
Jones R. (1974) Rights, wrongs, and referees. New Scientist 61:758–59. [aDVC]
Kahneman D., Slovic P. & Tversky A., eds. (1982) Judgment under uncertainty: Heuristics and biases. Cambridge University Press. [HLR]
Kamin L. J. (1981) The intelligence controversy, ed. Eysenck H. J.. Wiley. [PHS]
Kazdin A. E. (1982) Single-case research designs: Methods for clinical and applied settings. Oxford University Press. [aDVC]
Kerr S., Tolliver J. & Petree D. (1977) Manuscript characteristics which influence acceptance for management and social science journals. Academy of Management Journal 20:132–41. [aDVC]
Klayman J. & Ha Y.-W. (1987) Confirmation, discontinuation and information in hypothesis testing. Psychological Review 94:211–28. [MEG]
Koran L. M. (1975a) The reliability of clinical methods, data, and judgments. New England Journal of Medicine 293:642–46. [rDVC]
Koran L. M. (1975b) The reliability of clinical methods, data, and judgments. New England Journal of Medicine 293:695701. [rDVC]
Koshland D. E. Jr. (1985) Peer review of peer review. Science 228:1387. [aDVC]
Kraemer H. C. (1980) Extension of the kappa coefficient. Biometrics 36:207–16. [rDVC]
Kraemer H. C. (1982) Estimating false alarms and missed events from interobserver agreement: Comment on Kaye. Psychological Bulletin 92:749–54. [rDVC]
Kraemer H. C. (1988) Assessment of 2x2 associations: Generalization or signal-detection methodology. The American Statistician 42:3749. [rDVC]
Kraus C. A. (1950) The present state of academic research. Chemical and Engineering News 28:3203–04. [aDVC]
Krippendorff K. (1970) Bivariate agreement coefficients for reliability of data. In: Sociological methodology, ed. Borgatta E. G.. Jossey-Bass. [aDVC]
Krystal J., Giller E. & Cicchetti D. V. (1986) Assessment of alexithymia in post-traumatic stress disorder and psychosomatic illness: Introduction of a reliable measure. Psychosomatic Medicine 48:8494. [rDVC]
Kuhn T. (1962) The structure of scientific revolutions. University of Chicago Press. [aDVC, LDN]
Lakatos I. (1972) Falsification and the methodology of scientific research programmes. In: Criticism and the growth of knowledge, ed. Lakatos I. & Musgrave A.. Cambridge University Press. [aDVC]
Laming D. (1984) The relativity of “absolute” judgments. British Journal of Mathematical and Statistical Psychology 37:152–83. [DL]
Laming D. (1990) The reliability of a certain university examination compared with the precision of absolute judgments. Quarterly Journal of Experimental Psychology 42A:239–54. [DL]
Laming D. (in press) Reconciling Fechner and Stevens? Behavioral and Brain Sciences. [DL]
Landis J. R. & Koch G. G. (1977) The measurement of observer agreement for categorical data. Biometrics 33:1599–74. [rDVC]
Latane B., Williams K. & Harkins S. (1979) Many hands make light work: The causes and consequences of social loafing. Journal of Personality and Social Psychology 37:822–32. [AMC]
Laudan L. (1984) Science and values: The aims of science and their role in scientific debate. University of California Press. [aDVC]
Lawlis G. F. & Lu E. (1972) Judgment of counseling process: Reliability, agreement, and error. Psychological Bulletin 78:1720. [aDVC]
Lazarus D. (1982) Interreferee agreement and acceptance rates in physics. Behavioral and Brain Sciences 5:219. [aDVC]
LeLewis D. & Burke C. J. (1949) The use and misuse of the chi square test. Psychological Bulletin 46:433–89. [rDVC]
Leach C. (1979) Introduction to statistics: A nonparametric approach for the social sciences. John Wiley. [aDVC]
Lindsey D. (1977) Participation and influence in publication review proceedings. American Psychologist 32:379–86. [RFB]
Lindsey D. (1978) The scientific publication system in social science. Jossey-Bass. [aDVC, RFB]
Lindsey D. (1988) Assessing precision in the manuscript review process: A little better than a dice roll. Scientometrics 14:7582. [LLH]
Lock S. (1985) A difficult balance: Editorial peer review in medicine. ISI Press. [aDVC, JF]
Lodahl J. B. (1970) Paradigm development as a source of consensus in scientific fields (unpublished master's thesis). [aDVC]
Lord F. N. & Novick M. R. (1968) Statistical theories of mental test scores. Addison-Wesley. [LLH]
Luce R. D. (1989) A history of psychology in autobiography, vol. 8, ed. Lindzey G.. Stanford University Press. [PHS]
Luce R. D. & Green D. M. (1978) Two tests of a neural attention hypothesis for auditory psychophysics. Perception & Psychophysics 23:363–71. [DL].
Luce R. D., Nosofsky R. M., Green D. M. & Smith A. F. (1982) The bow and sequential effects in absolute identification. Perception & Psychophysics 32:397408. [DL].
Machol R. (1981) Letter to the editor. The Sciences 21:xxi. [aDVC]
Maher B. A. (1978) A reader's, writer's, and reviewer's guide to assessing research reports in clinical psychology. Journal of Consulting and Clinical Psychology 46:835–38. [aDVC]
Mahoney M. J. (1976) Scientist as subject: The psychological imperative. Ballinger. [LLH].
Mahoney M. J. (1977) Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy Research 1:161–75. [aDVC, LDN, PHS, SPL, JSA].
Mahoney M. J. (1978) Publish and perish. Human Behavior 7:3841. [aDVC].
Mahoney M. J. (1982) Publication, politics, and scientific progress. Behavioral and Brain Sciences 5:220–21. [AMC]
Mahoney M. J. (1985) Open exchange and epistemic progress. American Psychologist 40:2939. [ADVC, JBG, RFB, JF]
Mahoney M. J. (1987) Scientific publication and knowledge politics. Journal of Social Behavior and Personality 2:165–76. [RFB]
Mahoney M. J. (1990) Bias, controversy, and abuse in the study of the scientific publication system. Science, Technology, & Human Values 15:5055. [MJM]
Mahoney M. J., Kazdin A. E. & Kenigsberg M. (1978) Getting published. Cognitive Therapy and Research 2:6970. [aDVC]
Margulis L. (1977) Letter to the editor: Peer review attacked. The Sciences 17:5, 31. [aDVC]
Marsh H. W. & Ball S. (1981) Interjudgmental reliability of reviews for the Journal of Educational Psychology. Journal of Educational Psychology 73:872–80. [aDVC], HWM
Marsh H. W. & Ball S. (1989) The peer review process used to evaluate manuscripts submitted to academic journals: Interjudgmental reliability. Journal of Experimental Education 57:151–69. [HWM]
McCarthy P., Sharpe M. R., Spiesel S. Z., Dolan T. F., Forsyth B. W., DeWitt T. G., Fink H. D., Baron M. A. & Cicchetti D. V. (1982) Observation scales to identify serious illness in febrile children. Pediatrics 70:802–09. [rDVC]
McCarthy P. L., Sznajderman S. D., Lustman-Findling K., Baron M. A., Fink H. D., Czarkowski N., Bauchner H., Forsyth B. C. & Cicchetti D. V. (1990) Mothers' clinical judgment: A randomized trial of the acute illness observation scales. Journal of Pediatrics 116:200–06. [rDVC]
McCartney J. L. (1978) Making sense of reviewers' comments. Paper presented to the Southern Sociological Association Meetings, New Orleans, LA. [aDVC]
McNemar Q. (1947) Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12:153–57. [rDVC].
McNemar Q. (1955) Psychological statistics. Wiley. [GSW]
McNutt R. A., Evans A. T., Fletcher R. H. & Fletcher S. W. (1990) The effects of blinding on the quality of peer review. Journal of the American Medical Association 263:137–76. [rDVC, JSA, SPL]
Merton R. K. (1973) The sociology of science: Theoretical and empirical investigations. University of Chicago Press. [aDVC, DLE]
Meyer G. S. (1979) Academic labor and the development of science (unpublished doctoral dissertation). State University of New York at Stony Brook. [SC]
Mezzich J. E., Kraemer H. C., Worthington D. R. L. & Coffman G. A. (1981) Assessment of agreement among several raters formulating multiple diagnoses. Journal of Psychiatric Research 16:2939. [rDVC]
Mitroff I. I. & Chubin D. E. (1979) Peer review at the NSF: A dialectical policy and analysis. Social Studies of Science 9:199232. Sage [aDVC]
Mulkay M. (1977) Sociology of the scientific research community. In: Science, technology, and society, ed. Spiegel-Rosing I. & de Solla Price D.. Sage. [aDVC]
Mulkay M. & William A. T. (1971) A sociological study of a physics department. British Journal of Sociology 22:6880. [SC]
Murphy R. J. L. (1978) Reliability of marking in eight GCE examinations. British Journal of Educational Psychology 48:196200. [DL]
Murphy R. J. L. (1982) A further report of investigations into the reliability of marking of GCE examinations. British Journal of Educational Psychology 52:5863. [DL]
National Research Council (1988) The behavioral and social sciences: Achievements and opportunities. National Academy Press. [MJM]
Nelson L., Satz P., Mitrushiea M., Van Gorp W., Cicchetti D., Lewis R. & Van Lancker D. (1989) Development and validation of the Neuropsychology Behavior and Affect Profile. Psychological Assessmen: A Journal of Consulting and Clinical Psychology 1:266–72. [rDVC]
Newman H., Freeman F. & Holzinger K. (1937) Twins. A study of heredity and environment. University of Chicago Press. [PHS].
Newman S. H. (1966) Improving the evaluation of submitted manuscripts. American Psychologist 21:980–81. [aDVC]
Nisbett R., & Ross L. (1980) Human inference: Strategies and shortcomingsin human judgments. Prentice-Hall. [HLR]
Nisbett R. E. & Wilson T. D. (1977) The halo effect: Evidence for unconscious alteration of judgments. Journal of Personality and socialPsychology 35:250–56.[RFB]
Noble J. H. (1974) Peer review: Quality control of applied social research. Science 185:916–21. [aDVC]
Nunnally J. C. (1978) Psychometric theory, 2nd ed. McGraw-Hill. [aDVC]
Orlinsky D. & Howard K. (1978) The relation of process to outcome in psychotherapy. In: Handbook of psychotherapy and behavior change, ed. Garfield S. & Bergin A.. John Wiley & Sons. [LDN]
Orr R. H. & Kassab J. (1965) Peer group judgments on scientific merit: Editorial refereeing. Paper presented to the Congress of the International Federation of Documentation, Washington, D.C. [aDVC]
Over R. (1982) What is the source of bias in peer review?. Behavioral and Brain Sciences 5:229–30. [aDVC]
Oxman A. D., Guyatt G. H., Singer J., Goldsmith C. H., Hutchison B. G., Milner R. A. & Streiner D. L. (1991) Agreement among reviewers of review articles. Journal of Clinical Epidemiology 44:9198. [rDVC]
Patterson E. H. (1969) Evaluation of manuscripts submitted for publication. American Psychologist 24:73. [aDVC]
Patterson K. & Bailar J. C. III (1985) A review of journal peer review. In: Selectivity in information systems: Survival of the fittest, ed. Warren K. S.. Praeger Scientific. [aDVC]
Peters C. (1976) Multiple submissions: Why not?. American Sociologist 11:165–79. [aDVC]
Peters D. P. & Ceci S. J. (1982) Peer-review practices of psychological journals: The fate of published articles submitted again. Behavioral and Brain Sciences 5:187255. [aDVC, AMC, DLE]
Pfeffer J., Leong A. & Strehl K. (1977) Paradigm development and particularism: Journal publication in three scientific disciplines. Social Forces 55:938–51. [aDVC]
Physical Review & Physical Review Letters (1987) Annual report 1986. [rDVC]
Pollack I. (1952) The information of elementary auditory displays. Journal of the Acoustical Society of America 24:745–49. [DL]
Pollack I. (1953) The information of elementary auditory displays. II Journal of the Acoustical Society of America 25:765–69. [DL]
Price D. de Solla (1963) Little science, big science. Columbia University Press. [aDVC]
Reid L. N., Soley L. C. & Wimmer R. D. (1981) Replications in advertising research: 1977, 1978, 1979. Journal of Advertising 10:313. [aDVC]
Relman A. S. (1978) Are journals really quality filters? Rockefeller Foundation working papers (conference, May 22–23). Coping with the biomedical literature explosion: A qualitative approach. [aDVC]
Remington M., Tyrer P. J., Newson-Smith J. & Cicchetti D. V. (1979) Comparative reliability of categorical and analogue rating scales in the assessment of psychiatric symptomatology. Psychological Medicine 9:765–70. [aDVC]
Rennie D. (1986) Guarding the guardians: A conference on editorial peer review. Journal of the American Medical Association 256:2391–92. [MJM]
Roberts W. A. (1976) Failure to replicate visual discrimination learning with a 1-min delay of reward. Learning and Motivation, 7, 313–25. [TRZ]
Robertson P. (1976) Towards open refereeing. New Scientist 71:410. [aDVC]
Robinson W. S. (1957) The statistical measurement of agreement. American Sociological Review 22:1725. [aDVC]
Rodman H. (1970) The moral responsibility of journal editors and referees. American Sociologist 5:351–57. [RC]
Roediger H. L. III (1987) The role of journal editors in the scientific process. In: Scientific excellence: Origins and assessment, ed. Jackson D. N. and Rushton J. P.. Sage. [LLH, HLR]
Rogot E. & Goldberg I. D. (1966) A proposed index for measuring agreement in test-retest studies. Journal of Chronic Diseases 19:9911006. [rDVC]
Romanczyk R. G., Kent R. N., Diament C. & O'Leary K. D. (1973) Measuring the reliability of observational data: A reactive process. Journal of Applied Analysis 6:175–84. [JDC]
Rosenfield N. S., Ablow R. C., Markowitz R. I., DiPietro M., Seashore J. H., Touloukian R. J. & Cicchetti D. V. (1984) Hirschsprung Disease: Accuracy of the barium enema examination. Radiology 150:393400. [rDVC]
Rosenthal R. (1979) The “file drawer problem” and tolerance for null results Psychological Bulletin 86:638–41. [PHS]
Rosenthal R. (1984) Meta-analytic procedures for social research. Sage. [RR]
Rosenthal R. (1987) Judgment studies: Design, analysis, and meta-analysis. Cambridge University Press. [RR]
Rosenthal R. & Rosnow R. L. (1984) Essentials of behavioral research: Methods and data analysis. McGraw-Hill. [RR]
Rosenthal R. & Rosnow R. L. (1985) Contrast analysis: Focused comparisons in the analysis of variance. Cambridge University Press. [RR]
Rosenthal R. & Rubin D. B. (1978) Interpersonal expectancy effects: The first 345 studies. Behavioral and Brain Sciences 3:377415. [PHS]
Rosenthal R. & Rubin D. B. (1982) A simple, general purpose display of magnitude of experimental effect. Journal of Educational Psychology 74:166–69. [RR]
Rourke B. P. & Costa L. (1979) Editorial policy II. Journal of Clinical Neuropsychology 1:9396. [aDVC]
Rowney J. A. & Zenisek T. J. (1980) Manuscript characteristics influencing reviewers' decisions. Canadian Psychology 21:1721. [aDVC]
Roy R. (1985) Funding science: The real defects of peer review and an alternative to it. Science, Technology, and Human Values 10:7381. [aDVC]
Rubin D. B. (1982) Rejection, rebuttal, revision: Some flexible features of peer review. Behavioral and Brain Sciences 2:236–37. [PHS]
Scarr S. (1982) Anosmic peer review: A rose by another name is evidently not a rose. Behavioral and Brain Sciences 5:237–38. [aDVC]
Scarr S. & Weber B. L. R. (1978) The reliability of reviews for the American Psychologist. American Psychologist 33:935. [aDVC, LJS]
Schönemann P. H. (1971) The minimum average correlation between equivalent sets of uncorrelated factors. Psychometrika 36:2130. [PHS]
Schönemann P. H. (1989) New questions about old heritability estimates. Bulletin of the Psychonomic Society 27:175–78. [PHS]
Schönemann P. H. & Wang M. M. (1972) Some new results on factor indeterminancy. Psychometrika 37:6191. [PHS]
Scott W. A. (1974) Interreferee agreement on some characteristics of manuscripts submitted to the Journal of Personality and Social Psychology. American Psychologist 29:698702. [aDVC, CAK]
Sharp D. W. (1990) What can and should be done to reduce publication bias?. Journal of the American Medical Association 263:1390–91. [SPL]
Shrout P. E., Spitzer R. L. & Fleiss J. L. (1987) Quantification of agreement in psychiatric diagnosis revisited. Archives of General Psychiatry 44:172–77. [rDVC]
Smart R. (1964) The importance of negative results in psychological research. Canadian Psychologist 5a:225–32. [aDVC]
Smigel E. O. & Ross H. L. (1970) Factors in the editorial decision. American Sociologist 5:1921. [aDVC]
Smith K. (1977) Letter to the editor: Peer review defended. The Sciences 17:5. [aDVC]
Snedecor G. W. & Cochran W. G. (1967) Statistical methods, 6th ed. Iowa State University Press. [RR]
Snedecor G. W. & Cochran W. G. (1980) Statistical methods, 7th ed. Iowa State University Press. [RR]
Solomon D. L. (1989) Editorial communication. Biometrics, June 21. [PHS]
Sommer R. & Sommer B. A. (1984) Reply from Sommer & Sommer. American Psychologist 39:1318–19. [aDVC]
Soper H. V., Cicchetti D. V., Satz P., Light R. & Orsini D. L. (1988) Null hypothesis disrespect in neuropsychology: Dangers of alpha and beta errors. Journal of Clinical and Experimental Neuropsychology 10:255–70. [aDVC]
Sparrow S. S., Balla D. A. & Cicchetti D. V. (1984a) The Vineland Adaptive Behavior Scales: A revision of the Vineland Social Maturity Scale by E. A. Doll. I. Survey form. American Guidance Service. [rDVC]
Sparrow S. S., Balla D. A. & Cicchetti D. V. (1984b) The Vineland Adaptive Behavior Scales: A revision of the Vineland Social Maturity Scale by E. A. Doll. II. Expanded form. American Guidance Service. [rDVC]
Sparrow S. S., Balla D. A. & Cicchetti D. V. (1985) The Vineland Adaptive Behavior Scales: A revision of the Vineland Social Maturity Scale by E. A. Doll. III. Classroom edition. American Guidance Service. [rDVC]
Spearman K. (1927) The abilities of man. MacMillan. [PHS]
Spitzer R. L. & Fleiss J. L. (1974) A reanalysis of the reliability of psychiatric diagnosis. British Journal of Psychiatry 125:341–47. [aDVC]
Steiger J. J. & Schonemann P. H. (1976) A history of factor indeterminancy. In: Theory construction and data analysis in the behavioral sciences, ed. Shye S.. Jossey-Bass. [PHS]
Steinberg M., Rounsaville B. & Cicchetti D. V. (1990) Interview for DSM- III-R dissociative disorders: Preliminary report on a new diagnostic instrument. American Journal of Psychiatry 147:7682. [rDVC]
Sterling T. D. (1959) Publication decisions and their possible effects on inferences drawn from tests of significance - or vice versa. Journal of the American Statistical Association 54:3034. [aDVC]
Stevens J. C. & Tulving E. (1957) Estimations of loudness by a group of untrained observers. American Journal of Psychology 70:600–05. [DL]
Stevens S. S. (1971) Issues in psychophysical measurement. Psychological Review 78:426–50. [DL]
Stinchcombe A. L. & Ofshe R. (1969) On journal editing as a probabilistic process. American Sociologist 4:116–17. [rDVC, SC]
Stumpf W. E. (1980) Letters: “Peer” review. Science 207:822–23. [aDVC]
Summary Report of Journal Operations (1989) American Psychologist 44:1070. [aDVC]
Surwillo W. W. (1986) Anonymous reviewing and the peer review process. American Psychologist 41:218. [aDVC]
Thomas G. J. (1982) Perhaps it was right to reject the resubmitted manuscripts. Behavioral and Brain Sciences 5:240. [aDVC]
Thomas H. (1985) On the “file drawer” problem (unpublished manuscript). [PHS]
Tinsley H. E. A. & Weiss D. J. (1975) Interrater reliability and agreement of subjective judgments. Journal of Counseling Psychology 22:358–76. [LLH]
Torgerson W. S. (1958) Theory and methods of scaling. Wiley. [DL]
Tyrer P., Cicchetti D. V., Casey P. R., Fitzpatrick K., Oliver R., Baiter A., Ciller E. & Harkness L. (1984) Cross-national reliability study of a schedule for assessing personality disorders. The Journal of Nervous and Mental Disease 172:718–21. [rDVC]
Tyrer P., Owen R. & Cicchetti D. V. (1984) The Brief Scale for Anxiety: A subdivision of the Comprehensive Psychopathological Rating Scale. Journal of Neurology, Neurosurgery and Psychiatry 47:970–75. [rDVC]
Tyrer P., Strauss J. & Cicchetti D. V. (1983) Temporal reliability of personality in psychiatric patients. Psychological Medicine 13:393–98. [rDVC]
Uebersax J. S. (1981) GKAPPA: Generalized kappa coefficient. Applied Psychological Measurement 5:28. [rDVC]
Uebersax J. S. (1982) A generalized kappa coefficient. Educational and Psychological Measurement 42:181–83. [rDVC]
Uebersax J. S. (1989) Latent structure modeling of ordered category rating agreement. Paper presented at the annual meeting of the Psychometric Society, UCLA, Los Angeles (A Rand Rand Corp. Note). [rDVC]
Ubersax J. & Grove W. (1989) Latent structure agreement analysis. Rand Corp. (A Rand Note). [rDVC]
Volkmar F. R., Cicchetti D. V., Dykens E., Sparrow S. S., Leckman J. F. & Cohen D. J. (1988) An evaluation of the Autism Behavior Checklist. Journal of Autism and Developmental Disorders 18:8197. [rDVC]
Wason P. C. (1960) On the failure to eliminate hypotheses in a conceptual task. Quarterly Journal of Experimental Psychology, 12, 129–40. [MEG]
Watkins M. W. (1979) Chance and interrater agreement on manuscripts. American Psychologist 34:796–97. [aDVC]
Whitehurst G. J. (1983) Interrater agreement for reviews for Developmental Review. Developmental Review 3:7378. [aDVC]
Whitehurst G. J. (1984) Interrater agreement for journal manuscript reviews. American Psychologist 39:2228. [aDVC, MED]
Wiener S. L., Urivetsky M., Bregman D., Cohen J., Eich R., Gootman N., Gulotta S., Taylor B., Tuttle R., Webb W. & Wright J. (1977) Peer review: Inter-reviewer agreement during evaluation of research grant applications. Clinical Research 25:306–11. [aDVC]
Wilson E. B. (1928) Review of “The Abilities of Man, Their Nature and Measurement,” by C. Spearman. Science 67:244–48. [PHS]
Wilson J. D. (1978) Peer review and publication. Journal of Clinical Investigation 61:16971701. [PT]
Wolff W. M. (1970) A study of criteria for journal manuscripts. American Psychologist 25:3639. [aDVC]
Wolff W. M. (1973) Publication problems in psychology and an explicit evaluation schema for manuscripts. American Psychologist 28:257–61. [aDVC]
Wright R. D. (1970) Truth and its keeper. New Scientist 45:402–04. [aDVC]
Wyer R. S., Greenwald A. G.Bernard H. R., Crandall R. & Anon. (1987) Comments on “The publication game.” Journal of Social Behavior and Personality 2:1322. [RC]
Yotopoulos P. A. (1961) Institutional affiliation of the contributors to three professional journals. American Economic Review 5:665–70. [aDVC]
Ziman J. (1968) Public knowledge: The social dimension of science. Cambridge University Press. [AMC]
Ziman J. (1976) The force of knowledge: The scientific dimension of society. Cambridge University Press. [AMC]
Zuckerman H. & Merton R. K. (1971) Patterns of evaluation in science: Institutionalization, structure, and functions of the referee system. Minerva 9:66100. [aDVC, LLH]
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Behavioral and Brain Sciences
  • ISSN: 0140-525X
  • EISSN: 1469-1825
  • URL: /core/journals/behavioral-and-brain-sciences
Please enter your name
Please enter a valid email address
Who would you like to send this to? *



Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 37 *
Loading metrics...

Abstract views

Total abstract views: 1194 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 18th November 2017. This data will be updated every 24 hours.