Advertisement

Scientometrics

, 81:635 | Cite as

Assessing scientific research performance and impact with single indices

  • John Panaretos
  • Chrisovaladis Malesios
Article

Abstract

We provide a comprehensive and critical review of the h-index and its most important modifications proposed in the literature, as well as of other similar indicators measuring research output and impact. Extensions of some of these indices are presented and illustrated.

Keywords

Bibliometric Indicator Scientific Performance Hirsch Index Citation Metrics Citation Source 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. Adler, R., Ewing, J., Taylor, P. (2008), Citation Statistics. Joint IMU/ICIAM/IMS-Committee on Quantitative Assessment of Research. (Available at: http://www.mathunion.org/fileadmin/IMU/Report/CitationStatistics.pdf).
  2. Amin, M., Mabe, M. (2000), Impact factor: Use and abuse. Perspectives in Publishing, 1: 1–6. (Available at: http://www.elsevier.com/framework_editors/pdfs/Perspectives1.pdf).Google Scholar
  3. Arencibia-jorge, R., Barrios-Almaguer, I., Fernandez-Hernandez, S., Carvajal-Espino, R. (2008), Applying successive H indices in the institutional evaluation: A case study. Journal of the American Society for Information Science and Technology, 59(1): 155–157. (Available at: http://eprints.rclis.org/archive/00013532/01/English_version.pdf).CrossRefGoogle Scholar
  4. Ball, P. (2005), Index aims for fair ranking of scientists. Nature, 436: 900. (Available at: http://dx.doi.org/10.1038/436900a).CrossRefGoogle Scholar
  5. Banks, M. G. (2006), An extension of the Hirsch Index: Indexing scientific topics and compounds. Scientometrics, 69(1): 161–168. (Available at: http://www.springerlink.com/content/cq62k71162274586).CrossRefGoogle Scholar
  6. Barendse, W. (2007), The strike rate index: A new index for journal quality based on journal size and the hindex of citations. Biomedical Digital Libraries, 4: 3. (Available at: http://www.pubmedcentral.nih.gov/picrender.fcgi?artid=1868756&blobtype=pdf).CrossRefGoogle Scholar
  7. Batista, P. D., Campiteli, M. G., Kinouchi, O., Martinez, A. S. (2005), Universal Behaviour of a Research Productivity Index. ArXiv:physics/0510142. (Available at: http://arxiv.org/ftp/physics/papers/0509/0509048.pdf).
  8. Beirlant, J., Einmahl, J. H. J. (2007), Asymptotics for the Hirsch Index. (Available at: http://econpapers.repec.org/paper/dgrkubcen/200786.htm).
  9. Bornmann, L., Daniel, H. D. (2007), What do we know about the h index? Journal of the American Society for Information Science and Technology, 58(9): 1381–1385. (Available at: http://dx.doi.org/10.1002/asi.20609).CrossRefGoogle Scholar
  10. Bornmann, L., Mutz, R., Daniel, H. D. (2008), Are there better indices for evaluation purposes than the hindex? A comparison of nine different variants of the h index using data from biomedicine. Journal of the American Society for Information Science and Technology, 59(5): 1–8. (Available at: http://dx.doi.org/10.1002/asi.20806).CrossRefGoogle Scholar
  11. Braun, T., Glänzel, W., Schubert, A. (2005), A Hirsch-type Index for journals. The Scientist, 19: 8. (Available at: http://www.steunpuntoos.be/WG_Papers/Scientist_19_22_8.pdf).Google Scholar
  12. Braun, T., Glänzel, W., Schubert, A. (2006), A Hirsch-type Index for journals. Scientometrics, 69(1): 169–173. (Available at: http://dx.doi.org/10.1007/s11192-006-0147-4).CrossRefGoogle Scholar
  13. Burrell, Q. L. (1992), A simple model for linked informetric processes. Information Processing and Management, 28: 637–645. (Available at: http://dx.doi.org/10.1016/0306-4573(92)90033-V).CrossRefGoogle Scholar
  14. Burrell, Q. L. (2007a), Hirsch’s h-index: A stochastic model. Journal of Informetrics, 1: 16–25. (Available at: http://dx.doi.org/10.1016/j.joi.2006.07.001).CrossRefGoogle Scholar
  15. Burrell, Q. L. (2007b), On the h-index, the size of the Hirsch core and Jin’s A-index. Journal of Informetrics, 1: 170–177. (Available at: http://dx.doi.org/10.1016/j.joi.2007.01.003).CrossRefGoogle Scholar
  16. Chai, J.-C., Hua, P.-H., Rousseau, R., Wan, J.-K. (2008), The Adapted Pure h-Index. Proccedings of WIS 2008, Berlin Fourth International Conference on Webometrics, Informetrics and Scientometrics & Ninth COLLNET Meeting. (Available at: http://www.collnet.de/Berlin-2008/ChaiWIS2008aph.pdf).
  17. Chapron, G., Husté, A. (2006), Open, fair, and free journal ranking for researchers. BioScience, 56(7): 558–559. (Available at: http://www.carnivoreconservation.org/files/guillaume/bioscience_2006.pdf).CrossRefGoogle Scholar
  18. Cole, S., Cole, J. R. (1973), Social Stratification in Science. University of Chicago Press, Chicago.Google Scholar
  19. Costas, R., Bordons, M. (2007), The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level. Journal of Informetrics, 1: 193–203. (Available at: http://dx.doi.org/10.1016/j.joi.2007.02.001).CrossRefGoogle Scholar
  20. Cronin, B., Meho, L. (2006), Using the h-index to rank influential information scientists. Journal of the American Society for Information Science and Technology, 59: 1275–1278. (Available at: http://www.slis.indiana.edu/faculty/meho/cronin-meho-01.pdf).CrossRefGoogle Scholar
  21. Cozzens, S. E. (1989), What do citations count? The rhetoric-first model. Scientometrics, 15(5): 437–447. (Available at: http://dx.doi.org/10.1007/BF02017064).CrossRefMathSciNetGoogle Scholar
  22. Egghe, L. (2005), Power Laws in the Information Production Process: Lotkaian Informetrics. Elsevier, Oxford.Google Scholar
  23. Egghe, L. (2006a), How to improve the h-index. The Scientist, 20(3): 14. (Available at: http://www.accessmylibrary.com/coms2/summary_0286-14029249_ITM).Google Scholar
  24. Egghe, L. (2006b), An improvement of the H-index: the G-index. ISSI Newsletter, 2(1): 8–9. (Available at: https://wiki-sop.inria.fr/wiki/pub/Ce/IndicateurEvaluation/animprovement.pdf).MathSciNetGoogle Scholar
  25. Egghe, L. (2006c), Theory and practice of the g-index. Scientometrics, 69(1): 131–152. (Available at: http://dx.doi.org/10.1007/s11192-006-0144-7).CrossRefMathSciNetGoogle Scholar
  26. Egghe, L. (2008a), Modelling successive h-indices. Scientometrics (to appear). (Available at: http://dx.doi.org/10.1007/s11192-007-1968-5).
  27. Egghe, L. (2008b), Dynamic h-index: The Hirsch Index in function of time. Journal of the American Society for Information Science and Technology (to appear). (Available at: http://dx.doi.org/10.1002/asi.v58:3).
  28. Egghe, L. (2008c), Mathematical theory of the h- and g-Index in case of fractional counting of authorship. Journal of the American Society for Information Science and Technology, 59(10): 1608–1616. (Available at: http://dx.doi.org/10.1002/asi.20845).CrossRefGoogle Scholar
  29. Egghe, L., Rao, R. (2008), Study of different h-indices for groups of authors. Journal of the American Society for Information Science and Technology, 59(8): 1276–1281. (Available at: http://dx.doi.org/10.1002/asi.20809).CrossRefGoogle Scholar
  30. Egghe, L., Rousseau, R. (2006), An informetric model for the Hirsch Index. Scientometrics, 69(1): 121–129. (Available at: http://dx.doi.org/10.1007/s11192-006-0143-8).CrossRefGoogle Scholar
  31. Egghe, L., Rousseau, R. (2007), An h-index weighted by citation impact. Information Processing & Management, 44(2): 770–780. (Available at: 10.1016/j.ipm.2007.05.003).CrossRefGoogle Scholar
  32. Egghe, L., Rousseau, R., van Hooydonk, G. (2000), Methods for accrediting publications to authors or countries: Consequences for evaluation studies. Journal of the American Society for Information Science, 51(2): 145–157. (Available at: http://dx.doi.org/10.1002/(SICI)1097-4571(2000)51:2〈145::AID-ASI6〉3.0.CO;2-9).CrossRefGoogle Scholar
  33. Evidence Report (2007), The Use of Bibliometrics to Measure Research Quality in the UK Higher Education System (A Report Produced for the Research Policy Committee of Universities, UK, by Evidence Ltd., a Company Specializing in Research Performance Analysis and Interpretation). (Available at: http://bookshop.universitiesuk.ac.uk/downloads/bibliometrics.pdf).
  34. Gumbel, E. J. (1958), Statistics of Extremes. Columbia University Press.Google Scholar
  35. Garfield, E. (1955), Citation indexes to science: A new dimension in documentation through association of ideas. Science, 122: 108–111. (Available at: http://ije.oxfordjournals.org/cgi/reprint/35/5/1123).CrossRefGoogle Scholar
  36. Garfield, E. (2006), The history and meaning of the journal impact factor. Journal of the American Medical Association, 295: 90–93. (Available at: http://jama.ama-assn.org/cgi/reprint/295/1/90).CrossRefGoogle Scholar
  37. Glänzel, W. (2006), On the opportunities and limitations of the H-index. Science Focus, 1(1): 10–11. (Available at: http://eprints.rclis.org/archive/00009535/01/H_Index_opprtunities.pdf).Google Scholar
  38. Goldstein, H., Spiegelhalter, D. J. (1996), League tables and their limitations: statistical issues in comparisons of institutional performance. Journal of the Royal Statistical Society, Series A. 159(3): 385–443. (Available at: http://www.cmm.bristol.ac.uk/team/HG_Personal/limitations-of-league-tables.pdf).Google Scholar
  39. Hirsch, J. E. (2005), An Index to Quantify an Individual’s Scientific Research Output. Proceedings of the National Academy of Sciences, USA, 102(46): 16569–16572. (Available at: http://arxiv.org/PS_cache/physics/pdf/0508/0508025v5.pdf).CrossRefGoogle Scholar
  40. Hirsch, J. E. (2007), Does the h-index have a Predictive Power? (Available at: http://www.pnas.org/content/104/49/19193.full.pdf+html).
  41. Iglesias, J. E., Pecharromán, C. (2007a), Scaling the h-index for different scientific ISI fields. Scientometrics, 73(3): 303–320. (Available at: http://arxiv.org/ftp/physics/papers/0607/0607224.pdf).CrossRefGoogle Scholar
  42. Iglesias, J. E., Pecharromán, C. (2007b), Comparing h-indices for Scientists in Different ISI Fields. Proceedings of the 11th Conference of the International Society for Scientometrics and Informetrics. Madrid, Spain: Spanish Research Council (CSIC), 862–863.Google Scholar
  43. Imperial, J., Rodriguez-Navarro, A. (2007), Usefulness of Hirsch’s h-index to evaluate scientific research in Spain. Scientometrics, 71(2): 271–282. (Available at: http://dx.doi.org/10.1007/s11192-007-1665-4).CrossRefGoogle Scholar
  44. Jacsó, P. (2008a), Testing the calculation of a realistic h-index in Google Scholar, Scopus, and Web of Science for F. W. Lancaster. Library Trends. (Available at: http://findarticles.com/p/articles/mi_m1387/is_/ai_n28076518).
  45. Jacsó, P. (2008b), The pros and cons of computing the h-index using web of science. Online Information Review, 32(5): 673–688. (Available at: http://dx.doi.org/10.1108/14684520810914043).CrossRefGoogle Scholar
  46. Jin, B.-H. (2006), H-index: An evaluation indicator proposed by scientist. Science Focus, 1(1): 8–9.Google Scholar
  47. Jin, B.-H., Liang, L., Rousseau, R., Egghe, L. (2007), The R- and AR- indices: Complementing the h-index. Chinese Science Bulletin, 52: 855–863. (Available at: http://dx.doi.org/10.1007/s11434-007-0145-9).CrossRefGoogle Scholar
  48. Kelly, C. D., Jennions, M. D. (2006), The h index and career assessment by numbers. Trends in Ecology and Evolution, 21(4): 167–170. (Available at: http://dx.doi.org/10.1016/j.tree.2006.01.005).CrossRefGoogle Scholar
  49. Kinney, A. L. (2007), National scientific facilities and their science impact on nonbiomedical research. Proceedings of the National Academy of Sciences USA, 104(46): 17943–17947. (Available at: http://dx.doi.org/10.1073/pnas.0704416104).CrossRefGoogle Scholar
  50. Kosmulski, M. (2006), A new Hirsch-type Index saves time and works equally well as the original h-index. ISSI Newsletter, 2(3): 4–6.Google Scholar
  51. Kosmulski, M. (2007), MAXPROD — A new index for assessment of the scientific output of an individual, and a comparison with the h-index. International Journal of Scientometrics, Informetrics and Bibliometrics, 11(1). (Available at: http://cybermetrics.cindoc.csic.es/articles/v11i1p5.pdf).
  52. Lehmann, S., Jackson, A. D., Lautrup, B. E. (2006), Measures for Measures. Nature, 444: 1003–1004. (Available at: http://dx.doi.org/10.1038/4441003a).CrossRefGoogle Scholar
  53. Lehmann, S., Jackson, A. D., Lautrup, B. E. (2005), Measures and Missmeasures of Scientific Quality (to appear). (Available at: http://arxiv.org/abs/physics/0512238).
  54. Liang, L. (2006), H-index sequence and H-index matrix: Constructions and applications. Scientometrics, 69(1): 153–159. (Available at: http://dx.doi.org/10.1007/s11192-006-0145-6).CrossRefGoogle Scholar
  55. Liang L., Rousseau, R., Egghe, L. (2007), The R- and AR-indices: Complementing the h-index. Chinese Science Bulletin, 52(6): 855–863. (Available at: http://dx.doi.org/10.1007/s11434-007-0145-9).CrossRefGoogle Scholar
  56. Liu Y., Rousseau, R. (2007), Hirsch-type Indices and Library Management: The Case of Tongji University Library. Proceedings of ISSI 2007. Madrid: CINDOC-CSIC, 514–522. (Available at: http://users.pandora.be/ronald.rousseau/Liu_Rousseau_ISSI_2007.pdf).Google Scholar
  57. Macroberts, M. H., Macroberts, B. R. (1989), Problems of citation analysis: A critical review. Journal of the American Society for Information Science, 40: 342–349. (Available at: http://dx.doi.org/10.1002/(SICI)1097-4571(198909)40:5〈342::AID-ASI7〉3.0.CO;2-U).CrossRefGoogle Scholar
  58. Martin, B. R., Irvine, J. (1983), Assessing basic research: Some partial indicators of scientific progress in radio astronomy. Research Policy, 12: 61–90. (Available at: http://dx.doi.org/10.1016/0048-7333(93)90049-N-abstract only).CrossRefGoogle Scholar
  59. Meho, L. I. (2007), The rise and rise of citation analysis. Physics World, 29(1): 32–36. (Available at: http://dlist.sir.arizona.edu/1703).Google Scholar
  60. Meho, L. I., Yang, K. (2007), Impact of data sources on citation counts and rankings of lis faculty: Web of science vs. scopus and google scholar. Journal of the American Society for Information Science and Technology, 58(13): 2105–2125. (Available at: http://www3.interscience.wiley.com/journal/116328907/abstract).CrossRefGoogle Scholar
  61. Miller, C. W. (2006), Superiority of the h-index Over the Impact Factor for Physics. eprint arXiv:physics/0608183. (Available at: http://arxiv.org/PS_cache/physics/pdf/0608/0608183v1.pdf).
  62. Moed, H., van Leeuwen, T. (1996), Impact factors can mislead. Nature, 381: 186.CrossRefGoogle Scholar
  63. Molinari, J.-F., Molinari, A. (2008), A new methodology for ranking scientific institutions. Scientometrics, 75(1): 163–174. (Available at: http://dx.doi.org/10.1007/s11192-007-1853-2).CrossRefGoogle Scholar
  64. Orbay, M., Karamustafaoğlu, O., Öner, F. (2007), What does hirsch index evolution explain us? A Case Study: Turkish Journal of Chemistry. (Available at: http://arxiv.org/ftp/arxiv/papers/0704/0704.1602.pdf).
  65. Panaretos, J., Malesios, C. C. (2008), Assessing Scientific Research Performance and Impact with Single Indices. Technical Report no 235 (March 2008), Department of Statistics, Athens University of Economics and Business.Google Scholar
  66. Persson, O., Glänzel, W., Danell, R. (2008), Inflationary bibliometric values: The role of scientometric collaboration and the need for relative indicators in evaluative studies. Scientometrics (to appear). (Available at: http://dx.doi.org/10.1023/B:SCIE.0000034384.35498.7d).
  67. Podlubny, I. (2005), Comparison of scientific impact expressed by the number of citations in different fields of science. Scientometrics, 64(1): 95–99. (Available at: http://dx.doi.org/10.1007/s11192-005-0240-0).CrossRefGoogle Scholar
  68. Podlubny, I., Kassayova, K. (2006), Towards a better list of citation superstars: compiling a multidisciplinary list of highly cited researchers. Research Evaluation, 15(3): 154–162 (Available at: http://uk.arxiv.org/abs/math.ST/0603024).CrossRefGoogle Scholar
  69. Prathap, G. (2006), Hirsch-type indices for ranking institutions’ scientific research output. Current Science, 91(11): 1439. (Available at: http://www.ias.ac.in/currsci/dec102006/1439.pdf).Google Scholar
  70. Publish or Perish (2007), User’s Manual, http://www.harzing.com/resources.htm#/pop.htm, accessed on June 12, 2007.
  71. Roediger, H. L. (2006), The h-index in science: A new measure of scholarly contribution. Observer: The Academic Observer, 19(4): 37–40.Google Scholar
  72. Rousseau, R. (2006a), Simple models and the corresponding h- and g-index. Science Technology Development (to appear). (Available at: http://eprints.rclis.org/archive/00006153).
  73. Rousseau, R. (2006b), New developments related to the Hirsch index. Science Focus, 1(4): 23–25 (in Chinese). (English version available at: http://eprints.rclis.org/archive/00006376).Google Scholar
  74. Rousseau, R. (2007a), A case study: Evolution of JASIS’ Hirsch index, Science Focus. (Available at: http://eprints.rclis.org/archive/00005430/01/Evolution_of_h_JASIS_rev.pdf).
  75. Rousseau, R. (2007b), The influence of missing publications on the Hirsch index. Journal of Informetrics, 1: 2–7. (Available at: http://dx.doi.org/10.1016/j.joi.2006.05.001).CrossRefGoogle Scholar
  76. Saad, G. (2006), Exploring the h-index at the author and journal levels using bibliometric data of productive consumer scholars and business-related journals respectively. Scientometrics, 69(1): 117–120. (Available at: http://dx.doi.org/10.1007/s11192-006-0142-9).CrossRefGoogle Scholar
  77. Saisana, M., d’Hombres, B. (2008), Higher Education Rankings: Robustness Issues and Critical Assessment, JRC Scientific and Technical Reports 23487 EN. (Available at: http://crell.jrc.ec.europa.eu/Publications/CRELL%20Research%20Papers/EUR23487.pdf).
  78. Sanderson, M. (2008), Revisiting h measured on UK LIS Academics. Journal of the American Society for Information Science and Technology, 59(7): 1184–1190. (Available at: http://dx.doi.org/10.1002/asi.20771).CrossRefGoogle Scholar
  79. Symonds M. R., Gemmell N. J., Braisher T. L., Gorringe K. L., Elgar M. A. (2006), Gender differences in publication output: Towards an unbiased metric of research performance. PLoS ONE 1e127: 1–5. (Available at: http://dx.doi.org/10.1371/journal.pone.0000127).Google Scholar
  80. Schreiber, M. (2007a), Self-citation corrections for the Hirsch index. EPL 78 30002, 1-6. (Available at: http://dx.doi.org/10.1209/0295-5075/78/30002).
  81. Schreiber, M. (2007b), The Influence of Self-citations on Egghe’s g Index. (Available at: http://arxiv.org/PS_cache/arxiv/pdf/0707/0707.4577v1.pdf).
  82. Schreiber, M. (2008a), A modification of the h-index: The hm-index accounts for multi-authored manuscripts. Journal of Informetrics, 2(3): 211–216. (Available at: http://dx.doi.org/10.1016 /j.joi.2008.05.001).CrossRefMathSciNetGoogle Scholar
  83. Schreiber, M. (2008b), To share the fame in a fair way, hm modifies h for multi-authored manuscripts. New Journal of Physics, 10: 9. (Available at: http://dx.doi.org/10.1088/1367-2630/10/4/040201).CrossRefGoogle Scholar
  84. Schubert, A. (2007), Successive h-indices. Scientometrics, 70(1): 201–205. (Available at: http://dx.doi.org/10.1007/s11192-007-0112-x).CrossRefGoogle Scholar
  85. Schubert, A., Glänzel, W. (2007), A systematic analysis of Hirsch-type indices for journals. Journal of Informetrics, 1: 179–184. (Available at: http://dx.doi.org/10.1016/j.joi.2006.12.002).CrossRefGoogle Scholar
  86. Sidiropoulos, A., Katsaros, D., Manolopoulos, Y. (2006), Generalized h-index for disclosing latent facts in citation networks. Scientometrics, 72(2): 253–280. (Available at: http://dx.doi.org/10.1007/s11192-007-1722-z).CrossRefGoogle Scholar
  87. Tol, R. S. J. (2007), Of the h-index and its Alternatives: An Application to the 100 Most Prolific Economists. Working paper FNU-146. (Available at: http://www.fnu.zmaw.de/fileadmin/fnu-files/publication/working-papers/alternativeindexwp.pdf).
  88. Toro-Alves, N., Herculano, R. D., Terçariol, C. A. S., Filho, O. K., Graeff, C. F. O. (2007), Hirsch’s Index: Case Study Conducted at the Faculdade de Filosofia, Ciências e Letras de Ribeirão Preto, Universidade de São Paulo. Brazilian Journal of Medical and Biological Research, 40: 1529–1536. (Available at: http://dx.doi.org/10.1590/S0100-879X2007001100013).CrossRefGoogle Scholar
  89. van Raan, A. J. (2005), Measurement of central aspects of scientific research: performance, Interdisciplinary, structure. Measurement: Interdisciplinary Research and Perspectives, 3(1): 1–19. (Available at: http://www.cwts.nl/TvR/documents/AvR-Measurem.pdf).CrossRefGoogle Scholar
  90. van Raan, A. J. (2006), Comparison of the Hirsch-index with standard bibliometric indicators and with Peer Judgment for 147 Chemistry Research Groups. Scientometrics, 67(3): 491–502. (Available at: http://www.cwts.nl/cwts/Hirsch.pdf).Google Scholar
  91. Vanclay, J. K. (2007), On the robustness of the h-index. Journal of the American Society for Information Science and Technology, 58(10): 1547–1550. (Available at: http://dx.doi.org/10.1002/asi.20616).CrossRefGoogle Scholar
  92. Vinkler, P. (2007), Eminence of scientists in the light of the h-index and other scientometric indicators. Journal of Information Science, 33: 481–491. (Available at: http://dx.doi.org/10.1177 /0165551506072165).CrossRefMathSciNetGoogle Scholar
  93. Wan, J.-K., Hua, P.-H., Rousseau, R. (2007), The Pure h-index: Calculating an Author’s h-index by Taking Co-authors Into Account. Preprint. (Available at: http://eprints.rclis.org/archive/00011401 /01/pure_h.pdf).
  94. Wu, Q. (2008), The w-index: A significant improvement of the h-index. (Available at: http://arxiv.org/ftp/arxiv/papers/0805/0805.4650.pdf).

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2009

Authors and Affiliations

  1. 1.Athens University of Economics and BusinessAthensGreece

Personalised recommendations