Scientometric indicators: peer-review, bibliometric methods and conflict of interests
- 656 Downloads
The paper discusses the role of scientometric indicators in peer-review selection of research project proposals. An ex post facto evaluation was made of three calls for research project proposals in Slovenia: 2003 with a peer review system designed in a way that conflict of interest was not avoided effectively, 2005 with a sound international peer-review system with minimized conflict of interest influence but a limited number of reviewers, and 2008 with a combination of scientometric indicators and a sound international peer review with minimized conflict of interest influence. The hypothesis was that the three different peer review systems would have different correlations with the same set of scientometric indicators. In the last two decision-making systems (2005 and 2008) where conflict of interest was effectively avoided, we have a high percentage (65%) of projects that would have been selected in the call irrespective of the method (peer review or bibliometrics solely). In contrast, in the 2003 call there is a significantly smaller percentage (49%) of projects that would have been selected in the call irrespective of the method (peer review or bibliometrics solely). It was shown that while scientometric indicators can hardly replace the peer-review system as the ultimate decision-making and support system, they can reveal its weaknesses on one hand and on the other can verify peer-review scores and minimize conflict of interest if necessary.
KeywordsScientometric indicators Research project proposals Ex post evaluation Peer review systems Conflict of interests
- Reinhart, M. (2009). Peer review of grant applications in biology and medicine. Reliability, fairness, and validity. Scientometrics, 1–21. Accessed November 15, 2009, from http://www.springerlink.com.nukweb.nuk.uni-lj.si/content/h697768254544588/fulltext.pdf.
- Seglen, P. O. (1997). Why the impact factors of journals should not be used for evaluating research. British Medical Journal, 314(7079), 498–502.Google Scholar
- Slovenian Current Research Information System (SICRIS). (2009). Accessed October 25, 2009, from http://sicris.izum.si/about/cris.aspx?lang=eng.
- Sorčan, S., Demsar, F., & Valenci, T. (2008). Znanstveno raziskovanje v Sloveniji. Ljubljana: Javna Agencija za raziskovalno dejavnost Republike Slovenije.Google Scholar
- van Raan, A. F. J. (2006). Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups. Scientometrics, 67(3), 491–502.Google Scholar