Abstract
The case of Dr. Hwang Woo Suk, the South Korean stem-cell researcher, is arguably the highest profile case in the history of research misconduct. The discovery of Dr. Hwang’s fraud led to fierce criticism of the peer review process (at Science). To find answers to the question of why the journal peer review system did not detect scientific misconduct (falsification or fabrication of data) not only in the Hwang case but also in many other cases, an overview is needed of the criteria that editors and referees normally consider when reviewing a manuscript. Do they at all look for signs of scientific misconduct when reviewing a manuscript? We conducted a quantitative content analysis of 46 research studies that examined editors’ and referees’ criteria for the assessment of manuscripts and their grounds for accepting or rejecting manuscripts. The total of 572 criteria and reasons from the 46 studies could be assigned to nine main areas: (1) ‘relevance of contribution,’ (2) ‘writing / presentation,’ (3) ‘design / conception,’ (4) ‘method / statistics,’ (5) ‘discussion of results,’ (6) ‘reference to the literature and documentation,’ (7) ‘theory,’ (8) ‘author’s reputation / institutional affiliation,’ and (9) ‘ethics.’ None of the criteria or reasons that were assigned to the nine main areas refers to or is related to possible falsification or fabrication of data. In a second step, the study examined what main areas take on high and low significance for editors and referees in manuscript assessment. The main areas that are clearly related to the quality of the research underlying a manuscript emerged in the analysis frequently as important: ‘theory,’ ‘design / conception’ and ‘discussion of results.’
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Anon (2006a), Ethics and fraud. Nature, 439(7073): 117–118.
Anon (2006b), Three cheers for peers. Nature, 439(7073): 118.
Armstrong, J. S. (1982), Research on scientific journals: implications for editors and authors. Journal of Forecasting, 1(1): 83–104.
Bauch, H. (2006), Fraud: anonymous ’stars’ would not dazzle reviewers. Nature, 440(7083): 408.
Bornmann, L., Daniel, H.-D. (2007), Multiple publication on a single research study: does it pay? The influence of number of research articles on total citation counts in biomedicine. Journal of the American Society for Information Science and Technology, 58(8): 1100–1107.
BRAD WRAY, K. (2006), Scientific authorship in the age of collaborative research. Studies in History and Philosophy of Science Part A, 37(3): 505–514.
Byrne, D. W. (1998), Publishing Your Medical Research paper. What They Don’t Teach in Medical School, London, UK, Williams & Wilkins.
Campanario, J. M. (1998), Peer review for journals as it stands today — part 1. Science Communication, 19(3): 181–211.
Cho, M. K., Mcgee, G., Magnus, D. (2006), Lessons of the stem cell scandal. Science, 311(5761): 614–615.
Cohen, J. (1988), Statistical Power Analysis for the Behavioral Sciences, Hillsdale, NJ, USA, Lawrence Erlbaum Associates, Publishers.
Couzin, J. (2006), … And how the problems eluded peer reviewers and editors. Science, 311(5757): 23–24.
Cyranoski, D. (2006), Verdict: Hwang’s human stem cells were all fakes. Nature, 439(7073): 122–123.
Fletcher, R. H., Fletcher, S. W. (2003), The effectiveness of journal peer review. In: F. Godlee, T. Jefferson (Eds), Peer Review in Health Sciences. London, UK, BMJ Books, pp. 62–75.
Fox, M. F. (1994), Scientific misconduct and editorial and peer review processes. Journal of Higher Education, 65(3): 298–309.
Hirschauer, S. (2004), Peer Review Verfahren auf dem Prüfstand. Zum Soziologiedefizit der Wissenschaftsevaluation. Zeitschrift für Soziologie, 33(1): 62–83.
Howard, L., Wilkinson, G. (1998), Peer review and editorial decision-making. British Journal of Psychiatry, 173: 110–113.
Huth, E. J. (2000), Repetitive and divided publication. In: A. H. Jones, F. Mclellan (Eds), Ethical Issues in Biomedical Publication. Baltimore, MA, USA, Johns Hopkins University Press, pp. 112–136.
Lee, K., Bero, L. (2006), Ethics: increasing accountability. What authors, editors and reviewers should do to improve peer review. Retrieved June 17, 2006, from http://www.nature.com/nature/peerreview/debate/op3.html.
Martin, T. J. (2006), Reactions to the Hwang scandal. Science, 311(5761): 607.
Martinson, B. C., Anderson, M. S., De Vries, R. (2005), Scientists behaving badly. Nature, 435(7043):737–738.
Meadows, A. J. (1998), Communicating Research, London, UK, Academic Press.
Merton, R. K. (1973), The Sociology of Science: Theoretical and Empirical Investigations. Chicago, IL, USA: University of Chicago Press.
Normile, D., Vogel, G., Couzin, J. (2006), South Korean team’s remaining human stem cell claim demolished. Science, 311(5758): 156–157.
Odling-Smee, L., Giles, J., Fuyuno, I., Cyranoski, D., Marris, E. (2007), Where are they now? Nature, 445(7125): 244–245.
Office of Management and Budget (2004), Revised Information Quality Bulletin for Peer Review. Washington, DC, USA: Office of Management and Budget.
Rennie, D. (2003), Misconduct and journal peer review. In: F. Godlee, T. Jefferson (Eds), Peer Review in Health Sciences. London, UK, BMJ Books, pp. 118–129.
Sense About Science (2005), “I Don’t Know What to Believe …” Making Sense of Science Stories. London, UK: Sense about Science.
Shapin, S. (1994), A Social History of Truth: Civility and Science in Seventeenth-Century England, Chicago, IL, USA, The University of Chicago Press.
Smith, R. (2006), Peer review: a flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99(4): 178–182.
Weller, A. C. (2002), Editorial Peer Review: Its Strengths and Weaknesses, Medford, NJ, USA, Information Today, Inc.
White, H. D. (2005), On extending informetrics: an opinion paper. In: P. Ingwersen, B. Larsen (Eds), Proceedings of the 10th International Conference of the International Society for Scientometrics and Informetrics. Stockholm, Sweden, Karolinska University Press, pp. 442–449.
Ziman, J. (2000), Real Science. What It Is, and What It Means, Cambridge, UK, Cambridge University Press.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Bornmann, L., Nast, I. & Daniel, HD. Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication. Scientometrics 77, 415–432 (2008). https://doi.org/10.1007/s11192-007-1950-2
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-007-1950-2