Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication

Abstract

The case of Dr. Hwang Woo Suk, the South Korean stem-cell researcher, is arguably the highest profile case in the history of research misconduct. The discovery of Dr. Hwang’s fraud led to fierce criticism of the peer review process (at Science). To find answers to the question of why the journal peer review system did not detect scientific misconduct (falsification or fabrication of data) not only in the Hwang case but also in many other cases, an overview is needed of the criteria that editors and referees normally consider when reviewing a manuscript. Do they at all look for signs of scientific misconduct when reviewing a manuscript? We conducted a quantitative content analysis of 46 research studies that examined editors’ and referees’ criteria for the assessment of manuscripts and their grounds for accepting or rejecting manuscripts. The total of 572 criteria and reasons from the 46 studies could be assigned to nine main areas: (1) ‘relevance of contribution,’ (2) ‘writing / presentation,’ (3) ‘design / conception,’ (4) ‘method / statistics,’ (5) ‘discussion of results,’ (6) ‘reference to the literature and documentation,’ (7) ‘theory,’ (8) ‘author’s reputation / institutional affiliation,’ and (9) ‘ethics.’ None of the criteria or reasons that were assigned to the nine main areas refers to or is related to possible falsification or fabrication of data. In a second step, the study examined what main areas take on high and low significance for editors and referees in manuscript assessment. The main areas that are clearly related to the quality of the research underlying a manuscript emerged in the analysis frequently as important: ‘theory,’ ‘design / conception’ and ‘discussion of results.’

This is a preview of subscription content, log in to check access.

References

  1. Anon (2006a), Ethics and fraud. Nature, 439(7073): 117–118.

  2. Anon (2006b), Three cheers for peers. Nature, 439(7073): 118.

    Google Scholar 

  3. Armstrong, J. S. (1982), Research on scientific journals: implications for editors and authors. Journal of Forecasting, 1(1): 83–104.

    Article  Google Scholar 

  4. Bauch, H. (2006), Fraud: anonymous ’stars’ would not dazzle reviewers. Nature, 440(7083): 408.

    Article  Google Scholar 

  5. Bornmann, L., Daniel, H.-D. (2007), Multiple publication on a single research study: does it pay? The influence of number of research articles on total citation counts in biomedicine. Journal of the American Society for Information Science and Technology, 58(8): 1100–1107.

    Article  Google Scholar 

  6. BRAD WRAY, K. (2006), Scientific authorship in the age of collaborative research. Studies in History and Philosophy of Science Part A, 37(3): 505–514.

    Article  Google Scholar 

  7. Byrne, D. W. (1998), Publishing Your Medical Research paper. What They Don’t Teach in Medical School, London, UK, Williams & Wilkins.

    Google Scholar 

  8. Campanario, J. M. (1998), Peer review for journals as it stands today — part 1. Science Communication, 19(3): 181–211.

    Article  Google Scholar 

  9. Cho, M. K., Mcgee, G., Magnus, D. (2006), Lessons of the stem cell scandal. Science, 311(5761): 614–615.

    Article  Google Scholar 

  10. Cohen, J. (1988), Statistical Power Analysis for the Behavioral Sciences, Hillsdale, NJ, USA, Lawrence Erlbaum Associates, Publishers.

    Google Scholar 

  11. Couzin, J. (2006), … And how the problems eluded peer reviewers and editors. Science, 311(5757): 23–24.

    Article  Google Scholar 

  12. Cyranoski, D. (2006), Verdict: Hwang’s human stem cells were all fakes. Nature, 439(7073): 122–123.

    Article  Google Scholar 

  13. Fletcher, R. H., Fletcher, S. W. (2003), The effectiveness of journal peer review. In: F. Godlee, T. Jefferson (Eds), Peer Review in Health Sciences. London, UK, BMJ Books, pp. 62–75.

    Google Scholar 

  14. Fox, M. F. (1994), Scientific misconduct and editorial and peer review processes. Journal of Higher Education, 65(3): 298–309.

    Article  Google Scholar 

  15. Hirschauer, S. (2004), Peer Review Verfahren auf dem Prüfstand. Zum Soziologiedefizit der Wissenschaftsevaluation. Zeitschrift für Soziologie, 33(1): 62–83.

    Google Scholar 

  16. Howard, L., Wilkinson, G. (1998), Peer review and editorial decision-making. British Journal of Psychiatry, 173: 110–113.

    Article  Google Scholar 

  17. Huth, E. J. (2000), Repetitive and divided publication. In: A. H. Jones, F. Mclellan (Eds), Ethical Issues in Biomedical Publication. Baltimore, MA, USA, Johns Hopkins University Press, pp. 112–136.

    Google Scholar 

  18. Lee, K., Bero, L. (2006), Ethics: increasing accountability. What authors, editors and reviewers should do to improve peer review. Retrieved June 17, 2006, from http://www.nature.com/nature/peerreview/debate/op3.html.

  19. Martin, T. J. (2006), Reactions to the Hwang scandal. Science, 311(5761): 607.

    Article  Google Scholar 

  20. Martinson, B. C., Anderson, M. S., De Vries, R. (2005), Scientists behaving badly. Nature, 435(7043):737–738.

    Article  Google Scholar 

  21. Meadows, A. J. (1998), Communicating Research, London, UK, Academic Press.

    Google Scholar 

  22. Merton, R. K. (1973), The Sociology of Science: Theoretical and Empirical Investigations. Chicago, IL, USA: University of Chicago Press.

    Google Scholar 

  23. Normile, D., Vogel, G., Couzin, J. (2006), South Korean team’s remaining human stem cell claim demolished. Science, 311(5758): 156–157.

    Article  Google Scholar 

  24. Odling-Smee, L., Giles, J., Fuyuno, I., Cyranoski, D., Marris, E. (2007), Where are they now? Nature, 445(7125): 244–245.

    Article  Google Scholar 

  25. Office of Management and Budget (2004), Revised Information Quality Bulletin for Peer Review. Washington, DC, USA: Office of Management and Budget.

    Google Scholar 

  26. Rennie, D. (2003), Misconduct and journal peer review. In: F. Godlee, T. Jefferson (Eds), Peer Review in Health Sciences. London, UK, BMJ Books, pp. 118–129.

    Google Scholar 

  27. Sense About Science (2005), “I Don’t Know What to Believe …” Making Sense of Science Stories. London, UK: Sense about Science.

    Google Scholar 

  28. Shapin, S. (1994), A Social History of Truth: Civility and Science in Seventeenth-Century England, Chicago, IL, USA, The University of Chicago Press.

    Google Scholar 

  29. Smith, R. (2006), Peer review: a flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99(4): 178–182.

    Article  Google Scholar 

  30. Weller, A. C. (2002), Editorial Peer Review: Its Strengths and Weaknesses, Medford, NJ, USA, Information Today, Inc.

    Google Scholar 

  31. White, H. D. (2005), On extending informetrics: an opinion paper. In: P. Ingwersen, B. Larsen (Eds), Proceedings of the 10th International Conference of the International Society for Scientometrics and Informetrics. Stockholm, Sweden, Karolinska University Press, pp. 442–449.

    Google Scholar 

  32. Ziman, J. (2000), Real Science. What It Is, and What It Means, Cambridge, UK, Cambridge University Press.

    Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Lutz Bornmann.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Bornmann, L., Nast, I. & Daniel, H. Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication. Scientometrics 77, 415 (2008). https://doi.org/10.1007/s11192-007-1950-2

Download citation

Keywords

  • Main Area
  • Peer Review Process
  • Ranking List
  • Research Misconduct
  • Institutional Affiliation