, 77:415 | Cite as

Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication

  • Lutz BornmannEmail author
  • Irina Nast
  • Hans-Dieter Daniel


The case of Dr. Hwang Woo Suk, the South Korean stem-cell researcher, is arguably the highest profile case in the history of research misconduct. The discovery of Dr. Hwang’s fraud led to fierce criticism of the peer review process (at Science). To find answers to the question of why the journal peer review system did not detect scientific misconduct (falsification or fabrication of data) not only in the Hwang case but also in many other cases, an overview is needed of the criteria that editors and referees normally consider when reviewing a manuscript. Do they at all look for signs of scientific misconduct when reviewing a manuscript? We conducted a quantitative content analysis of 46 research studies that examined editors’ and referees’ criteria for the assessment of manuscripts and their grounds for accepting or rejecting manuscripts. The total of 572 criteria and reasons from the 46 studies could be assigned to nine main areas: (1) ‘relevance of contribution,’ (2) ‘writing / presentation,’ (3) ‘design / conception,’ (4) ‘method / statistics,’ (5) ‘discussion of results,’ (6) ‘reference to the literature and documentation,’ (7) ‘theory,’ (8) ‘author’s reputation / institutional affiliation,’ and (9) ‘ethics.’ None of the criteria or reasons that were assigned to the nine main areas refers to or is related to possible falsification or fabrication of data. In a second step, the study examined what main areas take on high and low significance for editors and referees in manuscript assessment. The main areas that are clearly related to the quality of the research underlying a manuscript emerged in the analysis frequently as important: ‘theory,’ ‘design / conception’ and ‘discussion of results.’


Main Area Peer Review Process Ranking List Research Misconduct Institutional Affiliation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. Anon (2006a), Ethics and fraud. Nature, 439(7073): 117–118.Google Scholar
  2. Anon (2006b), Three cheers for peers. Nature, 439(7073): 118.Google Scholar
  3. Armstrong, J. S. (1982), Research on scientific journals: implications for editors and authors. Journal of Forecasting, 1(1): 83–104.CrossRefGoogle Scholar
  4. Bauch, H. (2006), Fraud: anonymous ’stars’ would not dazzle reviewers. Nature, 440(7083): 408.CrossRefGoogle Scholar
  5. Bornmann, L., Daniel, H.-D. (2007), Multiple publication on a single research study: does it pay? The influence of number of research articles on total citation counts in biomedicine. Journal of the American Society for Information Science and Technology, 58(8): 1100–1107.CrossRefGoogle Scholar
  6. BRAD WRAY, K. (2006), Scientific authorship in the age of collaborative research. Studies in History and Philosophy of Science Part A, 37(3): 505–514.CrossRefGoogle Scholar
  7. Byrne, D. W. (1998), Publishing Your Medical Research paper. What They Don’t Teach in Medical School, London, UK, Williams & Wilkins.Google Scholar
  8. Campanario, J. M. (1998), Peer review for journals as it stands today — part 1. Science Communication, 19(3): 181–211.CrossRefGoogle Scholar
  9. Cho, M. K., Mcgee, G., Magnus, D. (2006), Lessons of the stem cell scandal. Science, 311(5761): 614–615.CrossRefGoogle Scholar
  10. Cohen, J. (1988), Statistical Power Analysis for the Behavioral Sciences, Hillsdale, NJ, USA, Lawrence Erlbaum Associates, Publishers.zbMATHGoogle Scholar
  11. Couzin, J. (2006), … And how the problems eluded peer reviewers and editors. Science, 311(5757): 23–24.CrossRefGoogle Scholar
  12. Cyranoski, D. (2006), Verdict: Hwang’s human stem cells were all fakes. Nature, 439(7073): 122–123.CrossRefGoogle Scholar
  13. Fletcher, R. H., Fletcher, S. W. (2003), The effectiveness of journal peer review. In: F. Godlee, T. Jefferson (Eds), Peer Review in Health Sciences. London, UK, BMJ Books, pp. 62–75.Google Scholar
  14. Fox, M. F. (1994), Scientific misconduct and editorial and peer review processes. Journal of Higher Education, 65(3): 298–309.CrossRefGoogle Scholar
  15. Hirschauer, S. (2004), Peer Review Verfahren auf dem Prüfstand. Zum Soziologiedefizit der Wissenschaftsevaluation. Zeitschrift für Soziologie, 33(1): 62–83.Google Scholar
  16. Howard, L., Wilkinson, G. (1998), Peer review and editorial decision-making. British Journal of Psychiatry, 173: 110–113.CrossRefGoogle Scholar
  17. Huth, E. J. (2000), Repetitive and divided publication. In: A. H. Jones, F. Mclellan (Eds), Ethical Issues in Biomedical Publication. Baltimore, MA, USA, Johns Hopkins University Press, pp. 112–136.Google Scholar
  18. Lee, K., Bero, L. (2006), Ethics: increasing accountability. What authors, editors and reviewers should do to improve peer review. Retrieved June 17, 2006, from
  19. Martin, T. J. (2006), Reactions to the Hwang scandal. Science, 311(5761): 607.CrossRefGoogle Scholar
  20. Martinson, B. C., Anderson, M. S., De Vries, R. (2005), Scientists behaving badly. Nature, 435(7043):737–738.CrossRefGoogle Scholar
  21. Meadows, A. J. (1998), Communicating Research, London, UK, Academic Press.Google Scholar
  22. Merton, R. K. (1973), The Sociology of Science: Theoretical and Empirical Investigations. Chicago, IL, USA: University of Chicago Press.Google Scholar
  23. Normile, D., Vogel, G., Couzin, J. (2006), South Korean team’s remaining human stem cell claim demolished. Science, 311(5758): 156–157.CrossRefGoogle Scholar
  24. Odling-Smee, L., Giles, J., Fuyuno, I., Cyranoski, D., Marris, E. (2007), Where are they now? Nature, 445(7125): 244–245.CrossRefGoogle Scholar
  25. Office of Management and Budget (2004), Revised Information Quality Bulletin for Peer Review. Washington, DC, USA: Office of Management and Budget.Google Scholar
  26. Rennie, D. (2003), Misconduct and journal peer review. In: F. Godlee, T. Jefferson (Eds), Peer Review in Health Sciences. London, UK, BMJ Books, pp. 118–129.Google Scholar
  27. Sense About Science (2005), “I Don’t Know What to Believe …” Making Sense of Science Stories. London, UK: Sense about Science.Google Scholar
  28. Shapin, S. (1994), A Social History of Truth: Civility and Science in Seventeenth-Century England, Chicago, IL, USA, The University of Chicago Press.Google Scholar
  29. Smith, R. (2006), Peer review: a flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99(4): 178–182.CrossRefGoogle Scholar
  30. Weller, A. C. (2002), Editorial Peer Review: Its Strengths and Weaknesses, Medford, NJ, USA, Information Today, Inc.Google Scholar
  31. White, H. D. (2005), On extending informetrics: an opinion paper. In: P. Ingwersen, B. Larsen (Eds), Proceedings of the 10th International Conference of the International Society for Scientometrics and Informetrics. Stockholm, Sweden, Karolinska University Press, pp. 442–449.Google Scholar
  32. Ziman, J. (2000), Real Science. What It Is, and What It Means, Cambridge, UK, Cambridge University Press.Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2008

Authors and Affiliations

  • Lutz Bornmann
    • 1
    • 2
    Email author
  • Irina Nast
    • 1
  • Hans-Dieter Daniel
    • 1
  1. 1.ETH ZurichZurichSwitzerland
  2. 2.ETH ZurichZurichSwitzerland

Personalised recommendations