Abstract
One of the critical issues in national research assessment exercises concerns the choice of whether to evaluate the entire scientific portfolio of the institutions or a subset composed of the best products. Under the second option, the capacities of the institutions to select the appropriate researchers and their best products (the UK case) or simply the best products of every researcher (the Italian case) becomes critical, both for purposes of correct assessment of the real quality of research in the institutions evaluated, and for the selective funding that follows. In this work, through case studies of three Italian universities, we analyze the efficiency of the product selection that is intended to maximize the universities’ scores in the current national research assessment exercise, the results of which will be the basis for assigning an important share of public financing over the coming years.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
The VQR was preceded by the VTR (Triennial Research Evaluation), conducted for the period 2001–2003: for information see http://vtr2006.cineca.it/index_EN.html (last accessed 30 April 2013).
This problem occurred especially in a number of research institutions where recent reorganization had provoked union-management conflict.
Other public and private organizations engaged in research could participate in the evaluation by request, subject to fees.
Acronym for Evaluation Groups.
Lists of panel members are available at http://www.anvur.org/sites/anvur-miur/files/gev_elenco_0.pdf, last accessed on 30 April 2013.
The deadline was later extended to 31 May and finally to 15 June 2012.
In several cases the results from the internal workings of these committees appeared on the Internet, in what the authors consider an improper manner.
The complete list is accessible at http://cercauniversita.cineca.it/php5/settori/index.php, last accessed 30 April 2013.
http://cercauniversita.cineca.it/php5/docenti/cerca.php, last accessed 30 April 30 2013.
For simplicity in language we consistently indicate that the institutions were expected to submit “three” products for each researcher, although the precise number depended on the academic rank and the period that the faculty members were on staff over the years subject to evaluation (see full explanation, in “The 2004–2010 Italian research assessment exercise, VQR ” section).
This refers to products authored by the researchers but not selected for submission.
In reality no Italian institution independently prepares exhaustive databases of all the products of its researchers, or of their publications indexed in the WoS. The simulation of scenarios 2 and 3 is possible using the ORP databases.
The university actually requested the “extra” publications only to permit resolution of potential problems of co-authorship.
References
Abramo, G., & D’Angelo, C. A. (2010). Evaluating research: From informed peer review to bibliometrics. Scientometrics, 87(3), 499–514.
Abramo, G., D’Angelo, C. A., & Caprasecca, A. (2009). Allocative efficiency in public research funding: Can bibliometrics help? Research Policy, 38(1), 206–215.
Council of Canadian Academies. (2012). Expert Panel on Science Performance and Research Funding Informing research choices: Indicators and judgment/The Expert Panel on Science Performance and Research Funding. Ottawa: Council of Canadian Academies.
D’Angelo, C. A., Giuffrida, C., & Abramo, G. (2011). A heuristic approach to author name disambiguation in bibliometrics databases for large-scale research assessments. Journal of the American Society for Information Science and Technology, 62(2), 257–269.
Excellence in Research for Australia (ERA). (2013). Retrieved July 15, 2013 from http://www.arc.gov.au/era/default.htm.
Geuna, A., & Martin, B. R. (2003). University research evaluation and funding: An international comparison. Minerva, 41(4), 277–304.
Herbst, M. (2007). Financing public universities. Higher education dynamics (Vol. 18). The Netherlands: Springer.
Hicks, D. (2012). Performance-based university research funding systems. Research Policy, 41(2), 251–261.
Horrobin, D. F. (1990). The philosophical basis of peer review and the suppression of innovation. Journal of the American Medical Association, 263(10), 1438–1441.
Jordan, G. B., & Malone, E. L. (2001). Performance assessment. In US Department of Energy Office of Science and Office of Planning and Analysis (Ed.), Management benchmark study. Washington, D.C.: US Department of Energy.
Moxham, H., & Anderson, J. (1992). Peer review. A view from the inside. Science and Technology Policy, 5(1), 7–15.
Research Excellence Framework (REF). (2013). Retrieved July 15, 2013 from http://www.ref.ac.uk/.
VQR. (2012). Valutazione della Qualità della Ricerca (2004–2010). Retrieved Oct 31, 2013 from www.anvur.org/attachments/article/122/bando_vqr_def_07_11.pdf.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Abramo, G., D’Angelo, C.A. & Di Costa, F. Inefficiency in selecting products for submission to national research assessment exercises. Scientometrics 98, 2069–2086 (2014). https://doi.org/10.1007/s11192-013-1177-3
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-013-1177-3