, Volume 91, Issue 3, pp 857–862 | Cite as

The Hawthorne effect in journal peer review

  • Lutz Bornmann


Purpose—this paper aims to look at the Hawthorne effect in editorial peer review. Design/methodology/approach—discusses the quality evaluation of refereed scholarly journals. Findings—a key finding of this research was that in the peer review process of one and the same manuscript, reviewers or editors, respectively, arrive at different judgments. This phenomenon is named as “Hawthorne effect” because the different judgements are dependent on the specific conditions under which the peer review process at the individual journals takes place. Originality/value—provides a discussion on the quality evaluation of scholarly journals.


Editorial peer review Manuscript rejection Hawthorne effect 


  1. Abelson, P. H. (1980). Scientific communication. Science, 209(4452), 60–62.CrossRefGoogle Scholar
  2. Bakanic, V., McPhail, C., & Simon, R. J. (1987). The manuscript review and decision-making process. American Sociological Review, 52(5), 631–642.CrossRefGoogle Scholar
  3. Benda, W. G. G., & Engels, T. C. E. (2011). The predictive validity of peer review: a selective review of the judgmental forecasting qualities of peers, and implications for innovation in science. International Journal of Forecasting, 27(1), 166–182. doi: 10.1016/j.ijforecast.2010.03.003.CrossRefGoogle Scholar
  4. Bornmann, L. (2008). Scientific peer review. An analysis of the peer review process from the perspective of sociology of science theories. Human Architecture—Journal of the Sociology of Self-Knowledge, 6(2), 23–38.Google Scholar
  5. Bornmann, L. (2011). Scientific peer review. Annual Review of Information Science and Technology, 45, 199–245.Google Scholar
  6. Bornmann, L., & Daniel, H.-D. (2008a). The effectiveness of the peer review process: inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angewandte Chemie International Edition, 47(38), 7173–7178. doi: 10.1002/anie.200800513.CrossRefGoogle Scholar
  7. Bornmann, L., & Daniel, H.-D. (2008b). Selecting manuscripts for a high impact journal through peer review: a citation analysis of Communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere. Journal of the American Society for Information Science and Technology, 59(11), 1841–1852. doi: 10.1002/asi.20901.CrossRefGoogle Scholar
  8. Bunting, C. (2005, 25 February). Early careers spent grinding teeth, not cutting them. Times Higher Education Supplement, 18.Google Scholar
  9. Callaham, M., & McCulloch, C. (2011). Longitudinal trends in the performance of scientific peer reviewers. [Proceedings Paper]. Annals of Emergency Medicine, 57(2), 141–148. doi: 10.1016/j.annemergmed.2010.07.027.CrossRefGoogle Scholar
  10. Chew, F. S. (1991). Fate of manuscripts rejected for publication in the AJR. American Journal of Roentgenology, 156(3), 627–632.Google Scholar
  11. Chiesa, M., & Hobbs, S. (2008). Making sense of social research: how useful is the Hawthorne Effect? [Article]. European Journal of Social Psychology, 38(1), 67–74. doi: 10.1002/ejsp.401.CrossRefGoogle Scholar
  12. Cole, S. (1992). Making science, between nature and society. Cambridge: Harvard University Press.Google Scholar
  13. Cronin, B., & McKenzie, G. (1992). The trajectory of rejection. Journal of Documentation, 48(3), 310–317.CrossRefGoogle Scholar
  14. Daniel, H.-D. (1993). Guardians of science, fairness and reliability of peer review. Weinheim: Wiley-VCH.Google Scholar
  15. de Vries, D. R., Marschall, E. A., & Stein, R. A. (2009). Exploring the peer review process: What is it, does it work, and can it be improved? Fisheries, 34(6), 270–279.CrossRefGoogle Scholar
  16. Eisenhart, M. (2002). The paradox of peer review: Admitting too much or allowing too little? Research in Science Education, 32(2), 241–255.CrossRefGoogle Scholar
  17. Fogg, L., & Fiske, D. W. (1993). Foretelling the judgments of reviewers and editors. American Psychologist, 48(3), 293–294.CrossRefGoogle Scholar
  18. French, J. R. P. (1953). Experiments in field settings. In L. Festinger & D. Katz (Eds.), Research methods in the behavioral sciences (pp. 98–135). New York: Holt, Rinehart and Winston.Google Scholar
  19. Gans, J. S., & Shepherd, G. B. (1994). How are the mighty fallen—rejected classic articles by leading economists. Journal of Economic Perspectives, 8(1), 165–179.CrossRefGoogle Scholar
  20. Gordon, M. D. (1984). How authors select journals—a test of the reward maximization model of submission behavior. Social Studies of Science, 14(1), 27–43.CrossRefGoogle Scholar
  21. Gorman, G. E. (2007). The Oppenheim effect in scholarly journal publishing. Online Information Review, 31(4), 417–419. doi: 10.1108/14684520710780386.CrossRefGoogle Scholar
  22. Hojat, M., Gonnella, J. S., & Caelleigh, A. S. (2003). Impartial judgment by the “gatekeepers” of science: fallibility and accountability in the peer review process. Advances in Health Sciences Education, 8(1), 75–96.CrossRefGoogle Scholar
  23. Jayasinghe, U. W., Marsh, H. W., & Bond, N. (2001). Peer review in the funding of research in higher education: the Australian experience. Educational Evaluation and Policy Analysis, 23(4), 343–346.CrossRefGoogle Scholar
  24. Khosla, A., McDonald, R. J., Bornmann, L., & Kallmes, D. F. (2011). Getting to yes: the fate of neuroradiology manuscripts rejected by Radiology over a 2-year period. Radiology, 260(1), 3–5. doi: 10.1148/radiol.11110490.CrossRefGoogle Scholar
  25. Lipworth, W. L., Kerridge, I. H., Carter, S. M., & Little, M. (2011). Journal peer review in context: a qualitative study of the social and subjective dimensions of manuscript review in biomedical publishing. [Review]. Social Science and Medicine, 72(7), 1056–1063. doi: 10.1016/j.socscimed.2011.02.002.CrossRefGoogle Scholar
  26. Lock, S. (1985). A difficult balance: editorial peer review in medicine. Philadelphia: ISI Press.Google Scholar
  27. Marsh, H. W., Jayasinghe, U. W., & Bond, N. W. (2008). Improving the peer-review process for grant applications—reliability, validity, bias, and generalizability. American Psychologist, 63(3), 160–168. doi: 10.1037/0003-066x.63.3.160.CrossRefGoogle Scholar
  28. Martin, B. (2000). Research grants: problems and options. Australian Universities’ Review, 43(2), 17–22.Google Scholar
  29. McCook, A. (2006). Is peer review broken? The Scientist, 20(2), 26.Google Scholar
  30. McDonald, R. J., Cloft, H. J., & Kallmes, D. F. (2007). Fate of submitted manuscripts rejected from the American Journal of Neuroradiology: outcomes and commentary. American Journal of Neuroradiology, 28(8), 1430–1434. doi: 10.3174/Ajnr.A0766.CrossRefGoogle Scholar
  31. Opthof, T., Furstner, F., van Geer, M., & Coronel, R. (2000). Regrets or no regrets? No regrets! The fate of rejected manuscripts. Cardiovascular Research, 45(1), 255–258.CrossRefGoogle Scholar
  32. Owen, R. (1982). Reader bias. Journal of the American Medical Association, 247(18), 2533–2534.CrossRefGoogle Scholar
  33. Petty, R. E., & Fleming, M. A. (1999). The review process at PSPB: correlates of interreviewer agreement and manuscript acceptance. Personality and Social Psychology Bulletin, 25(2), 188–203.CrossRefGoogle Scholar
  34. Pruthi, S., Jain, A., Wahid, A., Mehra, K., & Nabi, S. A. (1997). Scientific community and peer review system—a case study of a central government funding scheme in India. Journal of Scientific & Industrial Research, 56(7), 398–407.Google Scholar
  35. Pulverer, B. (2010). Transparency showcases strength of peer review. [10.1038/468029a]. Nature, 468(7320), 29–31.CrossRefGoogle Scholar
  36. Ray, J., Berkwits, M., & Davidoff, F. (2000). The fate of manuscripts rejected by a general medical journal. American Journal of Medicine, 109(2), 131–135.CrossRefGoogle Scholar
  37. Ross, P. F. (1980). The sciences’ self-management: manuscript refereeing, peer review, and goals in science. Lincoln: The Ross Company.Google Scholar
  38. Sharp, D. W. (1990). What can and should be done to reduce publication bias—the perspective of an editor. Journal of the American Medical Association, 263(10), 1390–1391.MathSciNetCrossRefGoogle Scholar
  39. Shatz, D. (2004). Peer review: a critical inquiry. Lanham: Rowman & Littlefield.Google Scholar
  40. Sternberg, R. J., Hojjat, M., Brigockas, M. G., & Grigorenko, E. L. (1997). Getting in: criteria for acceptance of manuscripts in Psychological Bulletin, 1993–1996. Psychological Bulletin, 121(2), 321–323.CrossRefGoogle Scholar
  41. van Rooyen, S., Godlee, F., Evans, S., Smith, R., & Black, N. (1998). Effect of blinding and unmasking on the quality of peer review—a randomized trial. Journal of the American Medical Association, 280(3), 234–237.CrossRefGoogle Scholar
  42. Weller, A. C. (1996). Editorial peer review: a comparison of authors publishing in two groups of US medical journals. Bulletin of the Medical Library Association, 84(3), 359–366.MathSciNetGoogle Scholar
  43. Weller, A. C. (2002). Editorial peer review: its strengths and weaknesses. Medford: Information Today, Inc.Google Scholar
  44. Whitman, N., & Eyre, S. (1985). The pattern of publishing previously rejected articles in selected journals. Family Medicine, 17(1), 26–28.Google Scholar
  45. Wood, F. Q., & Wessely, S. (2003). Peer review of grant applications: a systematic review. In F. Godlee & T. Jefferson (Eds.), Peer review in health sciences (2nd ed., pp. 14–44). London: BMJ Books.Google Scholar
  46. Ziman, J. (1968). Public knowledge: an essay concerning the social dimension of science. Cambridge: Cambridge University Press.Google Scholar
  47. Ziman, J. (2000). Real science, what it is, and what it means. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  48. Zuckerman, H., & Merton, R. K. (1971). Patterns of evaluation in science: institutionalisation, structure and functions of the referee system. Minerva, 9(1), 66–100. doi: 10.1007/bf01553188.CrossRefGoogle Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2011

Authors and Affiliations

  1. 1.Max Planck Society, Administrative HeadquartersMunichGermany

Personalised recommendations