Scientometrics

, Volume 95, Issue 1, pp 115–127

Absolute and specific measures of research group excellence

Article

Abstract

A desirable goal of scientific management is to introduce, if it exists, a simple and reliable way to measure the scientific excellence of publicly funded research institutions and universities to serve as a basis for their ranking and financing. While citation-based indicators and metrics are easily accessible, they are far from being universally accepted as way to automate or inform evaluation processes or to replace evaluations based on peer review. Here we consider absolute measurements of research excellence at an amalgamated, institutional level and specific measures of research excellence as performance per head. Using biology research institutions in the UK as a test case, we examine the correlations between peer review-based and citation-based measures of research excellence on these two scales. We find that citation-based indicators are very highly correlated with peer-evaluated measures of group strength, but are poorly correlated with group quality. Thus, and almost paradoxically, our analysis indicates that citation counts could possibly form a basis for deciding on, how to fund research institutions, but they should not be used as a basis for ranking them in terms of quality.

Keywords

Scientometrics Scientific evaluation Higher education 

References

  1. De Bellis, N. (2009). Bibliometrics and citation analysis: from the science citation index to cybermetrics. USA: The Scarecrow Press, Inc., Lanham, Maryland, Toronto, Plymouth (UK).Google Scholar
  2. Nature Editorial (2010). Nature, and metrics special. http://www.nature.com/metrics. Accessed April 2012. (465, p. 845).
  3. Evidence (a business) report. (2010). The future of the UK university research base. Google Scholar
  4. Vinkler, P. (2001). An attempt for defining some basic categories of scientometrics and classifying the indicators of evaluative scientometrics. Scientometrics, 50(3), 539–544.CrossRefGoogle Scholar
  5. Vinkler, P. (2003). Relations of relative scientometric indicators. Scientometrics, 58(3),687–694.CrossRefGoogle Scholar
  6. Kenna, R., Berche, B. (2010). Critical mass and the dependency of research quality on group size. Scientometrics, 86(2),527–540.CrossRefGoogle Scholar
  7. Kenna, R., Berche, B. (2011a). Critical masses for academic research groups and consequences for higher education research policy and management. Higher Education Management and Policy, 23(3),1–21.Google Scholar
  8. The official web-page of the RAE. (2008a). http://www.rae.ac.uk/. Accessed 18 October 2012.
  9. The panel criteria and working methods, available on the official web-page of RAE. (2008b). http://www.rae.ac.uk/pubs/2006/01/docs/dall.pdf. Accessed 18 October 2012.
  10. The official web-page of the RAE. (2009) Biological Sciences. http://www.rae.ac.uk/pubs/2009/pro/uoas/uoa%2014%20-%20all%20submissions.pdf. Accessed 18 October 2012.
  11. The official web-page of the Higher Education Funding Council for England. Funding for universities and colleges in 2009--10 (2009). Electronic Publication 01/2009 in the ADMINHEFCE Archives. Accessed 18 October 2012Google Scholar
  12. Oppenheim, Ch. (1996). Do citations count? Citation indexing and the research assessment exercise (RAE). Serials The Journal for the Serials Community, 9(2),155–161.CrossRefGoogle Scholar
  13. Norrism M., Oppenheim, Ch. (2003). Citation counts and the research assessment exercise V: archaeology and the 2001 RAE. Journal of Documentation, 59(6),709–730.CrossRefGoogle Scholar
  14. Holmes, A., Oppenheim, C. (2001). Use of citation analysis to predict the outcome of the 2001 research assessment exercise for unit of assessment (UoA) 61: library and information management. Information Research, 6(2). http://informationr.net/ir/6-2/paper103.html.
  15. MacRoberts, M.H., MacRoberts, B.R. (1989). Problems of citation analysis: a critical review. Journal of the American Society for Information Science , 40(5):342–349.CrossRefGoogle Scholar
  16. The official web-page of Evidence Thomson Reuters. (2012). http://www.evidence.co.uk. Accessed 18 October 2012.
  17. Evidence report. (2011). Funding research excellence: research group size, critical mass & performance. University Alliance report (2011). http://www.unialliance.ac.uk/wpcontent/uploads/2011/07/University-Alliance-Funding-Research-Excellence-July-2011.pdf Accessed 18 October 2012.
  18. Schubert, A., Braun, T. (1996). Cross-field normalization of scientometric indicators. Scientometrics, 36(3),311–324.CrossRefGoogle Scholar
  19. Kenna, R., Berche, B. (2011b). Normalization of research evaluation results across academic disciplines. Research evaluation, 20, 107–116.CrossRefGoogle Scholar
  20. The overall quality profile, and sub-profiles for research outputs, research environment and esteem indicators for each submission for RAE. (2008c). http://www.rae.ac.uk/pubs/2009/pro/Quality_profiles.xls.

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2012

Authors and Affiliations

  • O. Mryglod
    • 1
  • R. Kenna
    • 2
  • Yu. Holovatch
    • 1
  • B. Berche
    • 3
  1. 1.Institute for Condensed Matter Physics of the NAS of UkraineLvivUkraine
  2. 2.Applied Mathematics Research CentreCoventry UniversityCoventryEngland, UK
  3. 3.Université de Lorraine, Campus de NancyVandoeuvre lés Nancy CedexFrance

Personalised recommendations