Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Predicting results of the Research Excellence Framework using departmental h-index

Abstract

We compare estimates for past institutional research performances coming from two bibliometric indicators to the results of the UK’s Research Assessment Exercise which last took place in 2008. We demonstrate that a version of the departmental h-index is better correlated with the actual results of that peer-review exercise than a competing metric known as the normalised citation-based indicator. We then determine the corresponding h-indices for 2008–2013, the period examined in the UK’s Research Excellence Framework (REF) 2014. We place herewith the resulting predictions on the arXiv in advance of the REF results being published (December 2014). These may be considered as unbiased predictions of relative performances in that exercise. We will revisit this paper after the REF results are available and comment on the reliability or otherwise of these bibliometrics as compared with peer review.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3

Notes

  1. 1.

    The future of the UK university research base. Evidence (a Thomson Reuters business) report, July 2010.

  2. 2.

    Funding research excellence: Research group size, critical mass and performance. A University Alliance report, July 2011.

References

  1. Bishop, D. (2014). BishopBlog. http://deevybee.blogspot.co.at/2013/01/an-alternative-to-ref2014.html. Accessed November 7, 2014.

  2. Enderby, J. (2011). Thick or thin? The funding dilemma. Editorial in the Journal of the Foundation for Science and Technology, 20(6), 3–4.

  3. Higher Education Funding Council for England. (2014). http://www.hefce.ac.uk/ Accessed October 31, 2014.

  4. Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences USA, 102(46), 16569–16572.

  5. Holmes, A., Oppenheim, C. (2001). Use of citation analysis to predict the outcome of the 2001 Research Assessment Exercise for unit of assessment (UoA) 61: Library and information management. Information Research 6(2). http://informationr.net/ir/6-2/paper103.html Accessed October 31, 2014.

  6. MacRoberts, M. H., & MacRoberts, B. R. (1989). Problems of citation analysis: A critical review. Journal of the American Society for Information Science, 40(5), 342–349.

  7. Molinari, J.-F., & Molinari, A. (2008). A new methodology for ranking scientific institutions. Scientometrics, 75(1), 163–174.

  8. Mryglod, O., Kenna, R., Holovatch, Yu., & Berche, B. (2013). Absolute and specific measures of research group excellence. Scientometrics, 95(1), 115–127. doi:10.1007/s11192-012-0874-7.

  9. Mryglod, O., Kenna, R., Holovatch, Yu., & Berche, B. (2013). Comparison of a citation-based indicator and peer review for absolute and specific measures of research-group excellence. Scientometrics, 97, 767–777. doi:10.1007/s11192-013-1058-9.

  10. Oppenheim, C. H. (1996). Do citations count? Citation indexing and the Research Assessment Exercise (RAE). Serials: The Journal for the Serials Community, 9(2), 155–161.

  11. RAE 2008. (2008a). Research Assessment Exercise. http://www.rae.ac.uk/ Accessed October 31, 2014.

  12. RAE 2008. (2008b). The panel criteria and working methods. Panel E. (2006). http://www.rae.ac.uk/pubs/2006/01/docs/eall. Accessed October 31, 2014.

  13. REF 2014. (2014). Research Excellence Framework. http://www.ref.ac.uk/ Accessed October 31, 2014.

  14. Research Analytics. (2013). http://www.evidence.co.uk. Accessed October 31, 2014.

  15. Schreiber, M. (2014). A variant of the h-index to measure recent performance. arXiv preprint arXiv:1409.3379.

  16. Schubert, A., & Braun, T. (1996). Cross-field normalization of scientometric indicators. Scientometrics, 36(3), 311–324.

  17. Scopus. (2014). http://www.scopus.com/. Accessed October 31, 2014.

  18. Van Raan, A. F. J. (2006). Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups. Scientometrics, 67(3), 491–502.

Download references

Acknowledgments

This work was supported by the 7th FP, IRSES Project No. 269139 “Dynamics and cooperative phenomena in complex physical and biological environments” and IRSES Project No. 295302 “Statistical physics in diverse realizations”.

Conflict of interest

None.

Author information

Correspondence to O. Mryglod.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Mryglod, O., Kenna, R., Holovatch, Y. et al. Predicting results of the Research Excellence Framework using departmental h-index. Scientometrics 102, 2165–2180 (2015). https://doi.org/10.1007/s11192-014-1512-3

Download citation

Keywords

  • Peer review
  • Hirsch index
  • Normalised citation-based indicator
  • Research Assessment Exercise (RAE)
  • Research Excellence Framework (REF)