Skip to main content
SpringerLink
Log in

The suitability of h and g indexes for measuring the research performance of institutions

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

It is becoming ever more common to use bibliometric indicators to evaluate the performance of research institutions, however there is often a failure to recognize the limits and drawbacks of such indicators. Since performance measurement is aimed at supporting critical decisions by research administrators and policy makers, it is essential to carry out empirical testing of the robustness of the indicators used. In this work we examine the accuracy of the popular “h” and “g” indexes for measuring university research performance by comparing the ranking lists derived from their application to the ranking list from a third indicator that better meets the requirements for robust and reliable assessment of institutional productivity. The test population is all Italian universities in the hard sciences, observed over the period 2001–2005. The analysis quantifies the correlations between the three university rankings (by discipline) and the shifts that occur under changing indicators, to measure the distortion inherent in use of the h and g indexes and their comparative accuracy for assessing institutions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. The “h-index” represents the maximum number h of works by a scientist that have at least h citations each.

  2. The g-index represents the highest number “g” of articles that together received g 2 or more citations (Egghe 2006).

  3. Mathematics and computer sciences; physics; chemistry; earth sciences; biology; medicine; agricultural and veterinary sciences; civil engineering; industrial and information engineering.

  4. The list is accessible on http://www.disp.uniroma2.it/laboratorioRTT/TESTI/Indicators/ssd2.html.

  5. http://cercauniversita.cineca.it/php5/docenti/cerca.php, last accessed on September 30, 2012.

  6. As frequently observed in literature (Lundberg 2007), standardization of citations with respect to median value rather than to the average is justified by the fact that distribution of citations is highly skewed in almost all disciplines.

  7. A publication co-authored by researchers of the same SDS and university is considered only once.

  8. See note 5.

  9. These percentages for weighting were assigned following the results of interviews of top Italian professors in the life sciences: the values could be changed to suit practices in other national contexts.

  10. p value < 0.01.

  11. http://www.rae.ac.uk/, last accessed on September 30, 2012.

References

  • Abramo, G., Cicero, T., & D’Angelo, C. A. (2012). A sensitivity analysis of researchers’ productivity rankings to the time of citation observation. Journal of Informetrics, 6(2), 192–201.

    Article  Google Scholar 

  • Abramo, G., & D’Angelo, C. A. (2007). Measuring science: Irresistible temptations, easy shortcuts and dangerous consequences. Current Science, 93(6), 762–766.

    Google Scholar 

  • Abramo, G., & D’Angelo, C. A. (2011). National-scale research performance assessment at the individual level. Scientometrics, 86(2), 347–364.

    Article  Google Scholar 

  • Abramo, G., D’Angelo, C. A., & Viel, F. (2013). Assessing the accuracy of the h and g indexes for measuring researchers’ productivity. Journal of the American Society for Information Science and Technology. doi:10.1002/asi.22828.

  • Batista, P. D., Campiteli, M. G., Kinouchi, O., & Martinez, A. S. (2006). Is it possible to compare researchers with different scientific interests? Scientometrics, 68(1), 179–189.

    Article  Google Scholar 

  • Braun, T., Glanzel, W., & Schubert, A. (2006). A Hirsch-type index for journals. Scientometrics, 69(1), 169–173.

    Article  Google Scholar 

  • Carbon, C. C. (2011). The carbon_h-factor: predicting individuals’ research impact at early stages of their career. PLoS ONE, 6(12), e28770.

    Article  Google Scholar 

  • D’Angelo, C. A., Giuffrida, C., & Abramo, G. (2011). A heuristic approach to author name disambiguation in large-scale bibliometric databases. Journal of the American Society for Information Science and Technology, 62(2), 257–269.

    Article  Google Scholar 

  • Egghe, L. (2006). Theory and practise of the g-index. Scientometrics, 69(1), 131–152.

    Article  MathSciNet  Google Scholar 

  • Franceschini, F., & Maisano, D. (2011). Structured evaluation of the scientific output of academic research groups by recent h-based indicators. Journal of Informetrics, 5(1), 64–74.

    Article  Google Scholar 

  • Guan, J., & Gao, X. (2008). Comparison and evaluation of Chinese research performance in the field of bioinformatics. Scientometrics, 75(2), 357–379.

    Article  Google Scholar 

  • Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16569–16572.

    Article  Google Scholar 

  • Hirsch, J. E. (2007). Does the h-index have predictive power? Proceedings of the National Academy of Sciences of the United States of America, 104(49), 19193–19198.

    Article  Google Scholar 

  • Hönekopp, J., & Khan, J. (2012). Future publication success in science is better predicted by traditional measures than by the h-index. Scientometrics, 90(3), 843–853.

    Article  Google Scholar 

  • Hönekopp, J., & Klebe, J. (2008). Sometimes the impact factor outshines the H-index. Retrovirology, 5, 88.

    Article  Google Scholar 

  • Huang, M., & Lin, C. (2012). Counting methods & university ranking by h-index. Proceedings of the National Academy of Sciences of the United States of America, 48(1), 1–6.

    Google Scholar 

  • Iglesias, J. E., & Pecharromán, C. (2007). Scaling the h-index for different scientific ISI fields. Scientometrics, 73(3), 303–320.

    Article  Google Scholar 

  • Jensen, P., Rouquier, J., & Croissant, Y. (2009). Testing bibliometric indicators by their prediction of scientists promotions. Scientometrics, 78(3), 467–479.

    Article  Google Scholar 

  • Kuan, G., Huang, M., & Chen, D. (2012). A two-dimensional approach to performance evaluation for a large number of research institutions. Journal of the American Society for Information Science and Technology, 63(4), 817–828.

    Article  Google Scholar 

  • Lazaridis, T. (2010). Ranking university departments using the mean h-index. Scientometrics, 82(2), 211–216.

    Article  Google Scholar 

  • Lundberg, J. (2007). Lifting the crown-citation z-score. Journal of Informetrics, 1(2), 145–154.

    Article  Google Scholar 

  • Marchant, T., & Bouyssou, D. (2011). Ranking scientists and departments in a consistent manner. Journal of the American Society for Information Science and Technology, 62(9), 1761–1769.

    Article  Google Scholar 

  • Molinari, A., & Molinari, J. (2008). Mathematical aspects of a new criterion for ranking scientific institutions based on the h-index. Scientometrics, 75(2), 339–356.

    Article  MathSciNet  Google Scholar 

  • Radicchi, F., Fortunato, S., & Castellano, C. (2008). Universality of citation distributions: Toward an objective measure of scientific impact. Proceedings of the National Academy of Sciences of the United States of America, 105(45), 17268–17272.

    Article  Google Scholar 

  • Ravallion, M., & Wagstaff, A. (2011). On measuring scholarly influence by citations. Scientometrics, 88(1), 321–337.

    Article  Google Scholar 

  • Rezek, I., McDonald, R. J., & Kallmes, D. F. (2011). Is the h-index predictive of greater NIH funding success among academic radiologists? Academic Radiology, 18(11), 1337–1340.

    Article  Google Scholar 

  • Schreiber, M., Malesios, C. C., & Psarakis, S. (2012). Exploratory factor analysis for the Hirsch index, 17 h-type variants, and some traditional bibliometric indicators. Journal of Informetrics, 6(3), 347–358.

    Article  Google Scholar 

  • Sombatsompop, N., Markpin, T., Ratchatahirun, P., Yochai, W., Ittiritmeechai, S., Premkamolnetr, N., et al. (2011). Research productivity and impact of ASEAN countries and universities in the field of energy and fuel. Malaysian Journal of Library and Information Science, 16(1), 35–46.

    Google Scholar 

  • Van Raan, A. F. J. (2006). Comparison of the Hirsc h-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups. Scientometrics, 67(3), 491–502.

    Google Scholar 

  • Vanclay, J. (2008). Ranking forestry journals using the h-index. Journal of Informetrics, 2(4), 326–334.

    Article  Google Scholar 

  • Vinkler, P. (2013). Would it be possible to increase the Hirsch-index, p-index or CDS-index by increasing the number of publications or citations only by unity? Journal of Informetrics, 7(1), 72–83.

    Article  Google Scholar 

  • Waltman, L., & Van Eck, N. J. (2012). The inconsistency of the h-index. Journal of the American Society for Information Science and Technology, 63(2), 406–415.

    Article  Google Scholar 

  • Ye, F. Y., & Rousseau, R. (2010). Probing the h-core: An investigation of the tail-core ratio for rank distributions. Scientometrics, 84(2), 431–439.

    Article  Google Scholar 

  • Zhang, C. T. (2009). The e-index, complementing the h-index for excess citations. PLoS ONE, 4(5), e5429.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Laboratory for Studies of Research and Technology Transfer - Institute for System Analysis and Computer Science (IASI-CNR), National Research Council of Italy, Viale Manzoni 30, 00185, Rome, Italy

    Giovanni Abramo

  2. Department of Management, School of Engineering, University of Rome “Tor Vergata”, Via del Politecnico 1, 00133, Rome, Italy

    Giovanni Abramo & Ciriaco Andrea D’Angelo

  3. LAIT S.p.A., Via Adelaide Bono Cairoli 68, 00145, Rome, Italy

    Fulvio Viel

Authors
  1. Giovanni Abramo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ciriaco Andrea D’Angelo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fulvio Viel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Giovanni Abramo.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Abramo, G., D’Angelo, C.A. & Viel, F. The suitability of h and g indexes for measuring the research performance of institutions. Scientometrics 97, 555–570 (2013). https://doi.org/10.1007/s11192-013-1026-4

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-013-1026-4

Keywords

Search

Navigation