Absolute and specific measures of research group excellence
A desirable goal of scientific management is to introduce, if it exists, a simple and reliable way to measure the scientific excellence of publicly funded research institutions and universities to serve as a basis for their ranking and financing. While citation-based indicators and metrics are easily accessible, they are far from being universally accepted as way to automate or inform evaluation processes or to replace evaluations based on peer review. Here we consider absolute measurements of research excellence at an amalgamated, institutional level and specific measures of research excellence as performance per head. Using biology research institutions in the UK as a test case, we examine the correlations between peer review-based and citation-based measures of research excellence on these two scales. We find that citation-based indicators are very highly correlated with peer-evaluated measures of group strength, but are poorly correlated with group quality. Thus, and almost paradoxically, our analysis indicates that citation counts could possibly form a basis for deciding on, how to fund research institutions, but they should not be used as a basis for ranking them in terms of quality.
KeywordsScientometrics Scientific evaluation Higher education
This work was supported in part by the 7th FP, IRSES project No. 269139 “Dynamics and cooperative phenomena in complex physical and biological environments” and IRSES project No. 295302 “Statistical physics in diverse realizations”. The authors thank Jonathan Adams from Evidence for the data and Ihor Mryglod for fruitful discussions.
- De Bellis, N. (2009). Bibliometrics and citation analysis: from the science citation index to cybermetrics. USA: The Scarecrow Press, Inc., Lanham, Maryland, Toronto, Plymouth (UK).Google Scholar
- Nature Editorial (2010). Nature, and metrics special. http://www.nature.com/metrics. Accessed April 2012. (465, p. 845).
- Evidence (a business) report. (2010). The future of the UK university research base. Google Scholar
- Kenna, R., Berche, B. (2011a). Critical masses for academic research groups and consequences for higher education research policy and management. Higher Education Management and Policy, 23(3),1–21.Google Scholar
- The official web-page of the RAE. (2008a). http://www.rae.ac.uk/. Accessed 18 October 2012.
- The panel criteria and working methods, available on the official web-page of RAE. (2008b). http://www.rae.ac.uk/pubs/2006/01/docs/dall.pdf. Accessed 18 October 2012.
- The official web-page of the RAE. (2009) Biological Sciences. http://www.rae.ac.uk/pubs/2009/pro/uoas/uoa%2014%20-%20all%20submissions.pdf. Accessed 18 October 2012.
- The official web-page of the Higher Education Funding Council for England. Funding for universities and colleges in 2009--10 (2009). Electronic Publication 01/2009 in the ADMINHEFCE Archives. Accessed 18 October 2012Google Scholar
- Holmes, A., Oppenheim, C. (2001). Use of citation analysis to predict the outcome of the 2001 research assessment exercise for unit of assessment (UoA) 61: library and information management. Information Research, 6(2). http://informationr.net/ir/6-2/paper103.html.
- The official web-page of Evidence Thomson Reuters. (2012). http://www.evidence.co.uk. Accessed 18 October 2012.
- Evidence report. (2011). Funding research excellence: research group size, critical mass & performance. University Alliance report (2011). http://www.unialliance.ac.uk/wpcontent/uploads/2011/07/University-Alliance-Funding-Research-Excellence-July-2011.pdf Accessed 18 October 2012.
- The overall quality profile, and sub-profiles for research outputs, research environment and esteem indicators for each submission for RAE. (2008c). http://www.rae.ac.uk/pubs/2009/pro/Quality_profiles.xls.