Advertisement

Scientometrics

, Volume 107, Issue 1, pp 15–26 | Cite as

Priority criteria in peer review of scientific articles

  • Olgica NedićEmail author
  • Aleksandar Dekanski
Article

Abstract

Number of researchers, journals and articles has significantly increased in the last few years and peer review is still the most reliable instrument to sort out innovative, valuable, scientifically sound information from the pool of submitted results. Editors and publishers join their efforts to improve peer review process and to be able to do so properly, they need “field information” from contributors. Editorial board of the Journal of the Serbian Chemical Society performed its own survey to find out what reviewers recognize as priority criteria in manuscript evaluation and whether the professional background (title, number of years in research or experience in reviewing) influences these criteria. Most reviewers declared that they consider peer review as an essential component of the scientific professionalism. Scientific contribution and originality were the most important criteria in the evaluation of papers. Most reviewers preferred to see conclusions completely supported by experimental data, without additional speculations. Although there were no large differences between early stage and experienced researchers, early stage researchers and less experienced reviewers used grade 5 (indicating the highest priority) much more often in their evaluation of priority criteria than experienced researchers and/or reviewers, suggesting possible evolution of tolerance with experience.

Keywords

Scientific review Criteria Reviewer background 

Mathematics Subject Classification

62-07 

JEL Classification

Z000 

Notes

Acknowledgments

This research work was inspired by and is part of scientific activities in the COST Action TD1306 “New Frontiers of Peer Review (PEERE)”.

References

  1. Aleksic, J., Alexa, A., Attwood, T. K., Chue Hong, N., Dahlö, M., Davey, R., et al. (2014). An open science peer review oath. F1000Research, 3, 271. doi: 10.12688/f1000research.5686.2.Google Scholar
  2. Ashfold, M., & Appleyard, S. (2004). Standards for the quality and originality of articles in PCCP? A notice to authors and referees. Physical Chemistry Chemical Physics, 6(15), E9. doi: 10.1039/b409821h.CrossRefGoogle Scholar
  3. Azar, O. H. (2005). The review process in economics: Is it too fast? Southern Economic Journal, 72(2), 482–491. doi: 10.2307/20062123.CrossRefGoogle Scholar
  4. Bailey, C. D., Hermanson, D. R., & Louwers, T. J. (2008). An examination of the peer review process in accounting journals. Journal of Accounting Education, 26(2), 55–72. doi: 10.1016/j.jaccedu.2008.04.001.CrossRefGoogle Scholar
  5. Benos, D. J., Bashari, E., Chaves, J. M., Gaggar, A., Kapoor, N., LaFrance, M., et al. (2007). The ups and downs of peer review. Advances in Physiology Education, 31(2), 145–152. doi: 10.1152/advan.00104.2006.CrossRefGoogle Scholar
  6. Birukou, A., Wakeling, J. R., Bartolini, C., Casati, F., Marchese, M., Mirylenka, K., et al. (2011). Alternatives to peer review: Novel approaches for research evaluation. Frontiers in Computational Neuroscience, 5, 56. http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citation&list_uids=22174702
  7. Bornmann, L., Nast, I., & Daniel, H.-D. (2008). Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication. Scientometrics, 77(3), 415–432. doi: 10.1007/s11192-007-1950-2.CrossRefGoogle Scholar
  8. Bosch, X., Hernández, C., Pericas, J. M., Doti, P., & Marušić, A. (2012). Misconduct policies in high-impact biomedical journals. PLoS One, 7(12), e51928. doi: 10.1371/journal.pone.0051928.CrossRefGoogle Scholar
  9. Clark, R. K. F. (2012). Peer review: A view based on recent experience as an author and reviewer. British Dental Journal, 213(4), 153–154. doi: 10.1038/sj.bdj.2012.721.CrossRefGoogle Scholar
  10. Day, T. E. (2015). The big consequences of small biases: A simulation of peer review. Research Policy, 44(6), 1266–1270. doi: 10.1016/j.respol.2015.01.006.CrossRefGoogle Scholar
  11. DeMaria, A. N. (2008). Looking back. Journal of the American College of Cardiology, 52(25), 2211–2212. doi: 10.1016/j.jacc.2008.12.001.CrossRefGoogle Scholar
  12. Ernst, E., & Resch, K. L. (1994). Reviewer bias: A blinded experimental study. Journal of Laboratory and Clinical Medicine, 124(2), 178–182. http://www.ncbi.nlm.nih.gov/pubmed/8051481
  13. Fein, C. (2013). Multidimensional Journal Evaluation of PLOS ONE. Libri, 63(4), 259–271. doi: 10.1515/libri-2013-0021.CrossRefGoogle Scholar
  14. García Puig, J., Gaspar Alonso-Vega, G., & Ríos Blanco, J. J. (2012). Report of the editors, 2011. Revista Clínica Española, 212(1), 31–39. doi: 10.1016/j.rce.2011.11.004.CrossRefGoogle Scholar
  15. Ghosh, S. S., Klein, A., Avants, B., & Millman, K. J. (2012). Learning from open source software projects to improve scientific review. Frontiers in Computational Neuroscience, 6, 18. doi: 10.3389/fncom.2012.00018.CrossRefGoogle Scholar
  16. Godlee, F., Gale, C. R., & Martyn, C. N. (1998). Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: A randomized controlled trial. JAMA: The Journal of the American Medical Association, 280(3), 237–240. doi: 10.1001/jama.280.3.237.CrossRefGoogle Scholar
  17. Golden, M., & Schultz, D. M. (2012). Quantifying the volunteer effort of scientific peer reviewing. Bulletin of the American Meteorological Society, 93(3), 337–345. doi: 10.1175/BAMS-D-11-00129.1.CrossRefGoogle Scholar
  18. Guraya, S. Y. (2014). Accuracy of references in scholarly journals: An analysis of 450 references in ten biomedical journals. European Science Editing, 40(4), 88–90. http://www.scopus.com/inward/record.url?eid=2-s2.0-84920670315&partnerID=tZOtx3y1
  19. Hesman Saey, T. (2015). Is redoing scientific research the best way to find truth? Science News 187(2), January 13. https://www.sciencenews.org/article/redoing-scientific-research-best-way-find-truth
  20. Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature, 520(7548), 429–431. doi: 10.1038/520429a.CrossRefGoogle Scholar
  21. Ho, R. C.-M., Mak, K.-K., Tao, R., Lu, Y., Day, J. R., & Pan, F. (2013). Views on the peer review system of biomedical journals: an online survey of academics from high-ranking universities. BMC Medical Research Methodology, 13, 74. doi: 10.1186/1471-2288-13-74.CrossRefGoogle Scholar
  22. Hoffmann, R. (2008). A wiki for the life sciences where authorship matters. Nature Genetics, 40(9), 1047–1051. doi: 10.1038/ng.f.217.CrossRefGoogle Scholar
  23. Holden, C. (2005). Stem cell research. Korean cloner admits lying about oocyte donations. Science, 310(5753), 1402–1403. doi: 10.1126/science.310.5753.1402.CrossRefGoogle Scholar
  24. Houry, D., Green, S., & Callaham, M. (2012). Does mentoring new peer reviewers improve review quality? A randomized trial. BMC Medical Education, 12, 83. doi: 10.1186/1472-6920-12-83.CrossRefGoogle Scholar
  25. Kachewar, S. G., & Sankaye, S. B. (2013). Reviewer index: A new proposal of rewarding the reviewer. Mens Sana Monographs, 11(1), 274–284. doi: 10.4103/0973-1229.109347.CrossRefGoogle Scholar
  26. Kearney, M. H., Baggs, J. G., Broome, M. E., Dougherty, M. C., & Freda, M. C. (2008). Experience, time investment, and motivators of nursing journal peer reviewers. Journal of Nursing Scholarship, 40(4), 395–400. doi: 10.1111/j.1547-5069.2008.00255.x.CrossRefGoogle Scholar
  27. Kelty, C. M., Burrus, S., & Baranuik, R. G. (2008). Peer review anew: Three principles and a case study in publication quality assurance. Proceedings of the IEEE, 96(6), 1000–1011. doi: 10.1109/JPROC.2008.921613.CrossRefGoogle Scholar
  28. Kronick, D. A. (1990). Peer review in 18th-century scientific journalism. JAMA: The Journal of the American Medical Association, 263(10), 1321–1322. doi: 10.1001/jama.1990.03440100021002.CrossRefGoogle Scholar
  29. Laakso, M., & Björk, B.-C. (2012). Anatomy of open access publishing: A study of longitudinal development and internal structure. BMC Medicine, 10(1), 124. doi: 10.1186/1741-7015-10-124.CrossRefGoogle Scholar
  30. Marchionini, G. (2008). Editorial: Reviewer merits and review control in an age of electronic manuscript management system. ACM Transactions on Information Systems, 26(4), 25. doi: 10.1145/1402256.1402264.Google Scholar
  31. Moizer, P. (2009). Publishing in accounting journals: A fair game? Accounting, Organizations and Society, 34(2), 285–304. doi: 10.1016/j.aos.2008.08.003.CrossRefGoogle Scholar
  32. Moos, D. D., & Hawkins, P. (2009). Barriers and strategies to the revision process from an editor’s perspective. Nursing Forum, 44(2), 79–92. doi: 10.1111/j.1744-6198.2009.00131.x.CrossRefGoogle Scholar
  33. Nedić, O., & Dekanski, A. (2015). A survey on the Journal of the Serbian Chemical Society publishing policies—On the occasion of the 80th volume. Journal of the Serbian Chemical Society, 80(7), 959–969. doi: 10.2298/JSC150306036N.CrossRefGoogle Scholar
  34. Oosterhaven, J. (2015). Too many journals? Towards a theory of repeated rejections and ultimate acceptance. Scientometrics, 103(1), 261–265. doi: 10.1007/s11192-015-1527-4.CrossRefGoogle Scholar
  35. Reuters, T. (2015). IP & Science. http://ip-science.thomsonreuters.com/
  36. Siler, K., Lee, K., & Bero, L. (2014). Measuring the effectiveness of scientific gatekeeping. Proceedings of the National Academy of Sciences, 112(2), 360–365. doi: 10.1073/pnas.1418218112.CrossRefGoogle Scholar
  37. Souder, L. (2011). The ethics of scholarly peer review: A review of the literature. Learned Publishing, 24(1), 55–74. doi: 10.1087/20110109.CrossRefGoogle Scholar
  38. Squazzoni, F., Bravo, G., & Takács, K. (2013). Does incentive provision increase the quality of peer review? An experimental study. Research Policy, 42(1), 287–294. doi: 10.1016/j.respol.2012.04.014.CrossRefGoogle Scholar
  39. Stahel, P. F., & Moore, E. E. (2014). Peer review for biomedical publications: We can improve the system. BMC Medicine, 12, 179. doi: 10.1186/s12916-014-0179-1.CrossRefGoogle Scholar
  40. Steinhauser, G., Adlassnig, W., Risch, J. A., Anderlini, S., Arguriou, P., Armendariz, A. Z., et al. (2012). Peer review versus editorial review and their role in innovative science. Theoretical Medicine and Bioethics, 33(5), 359–376. doi: 10.1007/s11017-012-9233-1.CrossRefGoogle Scholar
  41. Vintzileos, A. M., Ananth, C. V., Odibo, A. O., Chauhan, S. P., Smulian, J. C., & Oyelese, Y. (2014). The relationship between a reviewer’s recommendation and editorial decision of manuscripts submitted for publication in obstetrics. American Journal of Obstetrics and Gynecology, 211(6), 703.e1–e5.CrossRefGoogle Scholar
  42. Yalow, R. S. (1977). Radioimmunoassay: A probe for fine structure of biologic systems. Nobel Lecture. http://www.nobelprize.org/nobel_prizes/medicine/laureates/1977/yalow-lecture.pdf

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2016

Authors and Affiliations

  1. 1.Institute for the Application of Nuclear Energy (INEP)University of BelgradeBelgradeSerbia
  2. 2.Department of Electrochemistry, Institute of Chemistry, Technology and MetallurgyUniversity of BelgradeBelgradeSerbia

Personalised recommendations