National peer-review research assessment exercises for the hard sciences can be a complete waste of money: the Italian case
- 758 Downloads
There has been ample demonstration that bibliometrics is superior to peer-review for national research assessment exercises in the hard sciences. In this paper we examine the Italian case, taking the 2001–2003 university performance rankings list based on bibliometrics as benchmark. We compare the accuracy of the first national evaluation exercise, conducted entirely by peer-review, to other rankings lists prepared at zero cost, based on indicators indirectly linked to performance or available on the Internet. The results show that, for the hard sciences, the costs of conducting the Italian evaluation of research institutions could have been completely avoided.
KeywordsResearch evaluation Bibliometrics VTR Ranking Productivity Universities
- ERA (2010). The excellence in research for Australia initiative. http://www.arc.gov.au/era/. Accessed 5 Sept 2012.
- Guisan, M. C. (2005). Universities and research expenditure in Europe and the USA, 1993–2003: an analysis of countries and regions. Regional and Sectoral Economic Studies, AEEADE, 5(2), 35–46.Google Scholar
- Moxham, H., & Anderson, J. (1992). Peer-review. A view from the inside. Science and Technology Policy, 5, 7–15.Google Scholar
- Oppenheim, C., & Norris, M. (2003). Citation counts and the research assessment exercise V: archaeology and the 2001 RAE. Journal of Documentation, 56(6), 709–730.Google Scholar
- Pendlebury, D. A. (2009). The use and misuse of journal metrics and other citation indicators. Scientometrics, 57(1), 1–11.Google Scholar
- RAE (2008). Research assessment exercise. http://www.rae.ac.uk/aboutus/. Accessed 5 Sept. 2012.
- Van Raan, A. F. J. (2006). Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups. Scientometrics, 67(3), 491–502.Google Scholar
- VTR (2006). Three-year research evaluation (2001–2003). Results separated by scientific area. http://vtr2006.cineca.it/index_EN.html. Accessed on 5 Sept 2012.