, Volume 105, Issue 3, pp 1743–1762 | Cite as

Comparison of bibliometric measures for assessing relative importance of researchers

  • Rok Blagus
  • Brane L. Leskošek
  • Janez Stare


Quantitative evaluation of citation data to support funding decisions has become widespread. For this purpose there exist many measures (indices) and while their properties were well studied there is little comprehensive experimental comparison of the ranking lists obtained when using different methods. A further problem of the existing studies is that lack of available data about net citations prevents researchers from studying the effect of measuring scientific impact by using net citations (all citations minus self-citations). In this paper we use simulated data to study factors that could potentially influence the degree of agreement between the rankings obtained when using different indices with the emphasis given to the comparison of the number of net citations per author to other more established indices. We observe that the researchers publishing papers with a large number of co-authors are systematically ranked higher when using h-index or total citations (TC) instead of the number of citations per author (TCA), that the researchers who publish a small proportion of papers which receive many citations while the rest of their papers receive only few citations are systematically ranked higher when using TCA or TC instead of h-index, and that the authors who have lower proportion of self-citations are ranked higher when considering indices which include the number of net citations in comparison with indices considering only the total citation count. Results are verified and illustrated also by analyzing a large dataset from the field of medical science in Slovenia for the period 1986–2007.


Bibliometric evaluation Number of citations per author Net citations h-index 



Our sincere thanks of gratitude goes to Dr. Hristovski who developed a programme to automatically analyze the Science Citation Index database (and later Web of Science).

Supplementary material

11192_2015_1622_MOESM1_ESM.pdf (88 kb)
Supplementary material 1 (pdf 87 KB)
11192_2015_1622_MOESM2_ESM.pdf (63 kb)
Supplementary material 2 (pdf 62 KB)


  1. Bartneck, C., & Kokkelmans, S. (2011). Detecting h-index manipulation through self-citation analysis. Scientometrics, 87(1), 85–98.CrossRefGoogle Scholar
  2. Batista, P. D., Campiteli, M. G., & Kinouchi, O. (2006). Is it possible to compare researchers with different scientific interests? Scientometrics, 68(1), 179–189.CrossRefGoogle Scholar
  3. Bland, J. M., & Altman, D. G. (2010). Statistical methods for assessing agreement between two methods of clinical measurement. International Journal of Nursing Studies, 47(8), 931–936.CrossRefGoogle Scholar
  4. Bornmann, L., & Daniel, H. D. (2007). What do we know about the h index? Journal of the American Society for Information Science and Technology, 58(9), 1381–1385. doi: 10.1002/asi.20609.CrossRefGoogle Scholar
  5. Bornmann, L., Mutz, R., & Daniel, H. D. (2008). Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine. Journal of the American Society for Information Science and Technology, 59(5), 830–837.CrossRefGoogle Scholar
  6. Bornmann, L., Mutz, R., Daniel, H. D., Wallon, G., & Ledin, A. (2009). Are there really two types of h index variants? A validation study by using molecular life sciences data. Research Evaluation, 18(3), 185–190.CrossRefGoogle Scholar
  7. Bornmann, L., Mutz, R., Hug, S. E., & Daniel, H. D. (2011). A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants. Journal of Informetrics, 5(3), 346–359.CrossRefGoogle Scholar
  8. Bras-Amorós, M., Domingo-Ferrer, J., & Torra, V. (2011). A bibliometric index based on the collaboration distance between cited and citing authors. Journal of Informetrics, 5(2), 248–264.CrossRefGoogle Scholar
  9. Cole, J. R., & Cole, S. (1973). Social stratification in science. Chicago: The University of Chicago Press.Google Scholar
  10. Egghe, L. (1987). An exact calculation of price’s law for the law of lotka. Scientometrics, 11(1–2), 81–97.CrossRefGoogle Scholar
  11. Egghe, L. (1991). The exact place of zipf’s and pareto’s law amongst the classical informetric laws. Scientometrics, 20(1), 93–106.CrossRefGoogle Scholar
  12. Egghe, L. (1998). Mathematical theories of citation. Scientometrics, 43(1), 57–62.MathSciNetCrossRefGoogle Scholar
  13. Egghe, L. (2005). A characterization of the law of lotka in terms of sampling. Scientometrics, 62(3), 321–328.CrossRefGoogle Scholar
  14. Egghe, L. (2006). Theory and practise of the g-index. Scientometrics, 69(1), 131–152.MathSciNetCrossRefGoogle Scholar
  15. Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science, 178(4060), 471–479.CrossRefGoogle Scholar
  16. Glanzel, W. (2006). On the h-index—A mathematical approach to a new measure of publication activity and citation impact. Scientometrics, 67(2), 315–321.CrossRefGoogle Scholar
  17. Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16,569–16,572.CrossRefGoogle Scholar
  18. Hirsch, J. E. (2007). Does the h index have predictive power? Proceedings of the National Academy of Sciences, 104(49), 19,193–19,198. doi: 10.1073/pnas.0707962104.CrossRefGoogle Scholar
  19. Hristovski, D., Rožić, A., & Adamič, Š. (1996) A decision support system for biomedical research evaluation. In Medical informatics Europe ‘96: Human facets in information technologies, pp 609–613Google Scholar
  20. Iglesias, J. E., & Pecharroman, C. (2006). Scaling the h-index for different scientific isi fields. arXiv:physics/0607224.
  21. Imperial, J., & Rodrguez-Navarro, A. (2007). Usefulness of hirsch’s h-index to evaluate scientific research in spain. Scientometrics, 71(2), 271–282. doi: 10.1007/s11192-007-1665-4.CrossRefGoogle Scholar
  22. Nerur, S., Sikora, R., Mangalaraj, G., & Balijepally, V. (2005). Assessing the relative influence of journals in a citation network. Commun ACM, 48(11), 71–74.CrossRefGoogle Scholar
  23. Panaretos, J., & Malesios, C. (2009). Assessing scientific research performance and impact with single indices. Scientometrics, 81(3), 635–670.CrossRefGoogle Scholar
  24. Pareto, V. (1897). Course d’économie politique. Lausanne: F. Rouge.Google Scholar
  25. R Core Team. (2013). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.Google Scholar
  26. Roediger, H. L. (2006). The h index in science: A new measure of scholarly contribution. APS Observer 19.Google Scholar
  27. Saam, N., & Reiter, L. (1999). Lotka’s law reconsidered: The evolution of publication and citation distributions in scientific fields. Scientometrics, 44(2), 135–155.CrossRefGoogle Scholar
  28. Schreiber, M. (2007). Self-citation corrections for the hirsch index. EPL (Europhysics Letters), 78(3), 30,002.CrossRefGoogle Scholar
  29. Schreiber, M. (2008a). A modification of the h-index: The hm-index accounts for multi-authored manuscripts. Journal of Informetrics, 2(3), 211–216.MathSciNetCrossRefGoogle Scholar
  30. Schreiber, M. (2008b). To share the fame in a fair way, \(h_m\) modifies h for multi-authored manuscripts. New Journal of Physics, 10(4), 040,201.CrossRefGoogle Scholar
  31. Silagadze, Z. (2010). Citation entropy and research impact estimation. Acta Physica Polonica B, 41(11), 2325–2333.Google Scholar
  32. Tol, R. (2011). Credit where credits due: Accounting for co-authorship in citation counts. Scientometrics, 89(1), 291–299.CrossRefGoogle Scholar
  33. Wan, J. K., Hua, Ph, & Rousseau, R. (2007). The pure h-index: Calculating an authors h-index by taking co-authors into account. Collnet Journal of Scientometrics and Information Management, 1(2), 1–5.CrossRefGoogle Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2015

Authors and Affiliations

  1. 1.Institute for Biostatistics and Medical InformaticsUniversity of LjubljanaLjubljanaSlovenia

Personalised recommendations