Advertisement

Scientometrics

, Volume 121, Issue 3, pp 1269–1291 | Cite as

P-score: a reputation bibliographic index that complements citation counts

  • João Mateus de Freitas VenerosoEmail author
  • Marlon Dias
  • Alberto Ueda
  • Sabir Ribas
  • Berthier Ribeiro-Neto
  • Nivio Ziviani
  • Edmundo de Souza e Silva
Article

Abstract

The notions of reputation and popularity in academia are critical for taking decisions on research grants, faculty position tenure, and research excellence awards. These notions are almost always associated with the publication track records of researchers. Thus, it is important to assess publication track records quantitatively. To quantify publication records, bibliographic indices are usually adopted and, among these, citation-based indices such as the H-index are frequently considered. In this paper we study the correlation between P-score, a publication record index and H-index, a very popular citation-based index, in the setting of conference ranking. While H-indices reflect the popularity of a given publication or researcher in academia, P-scores can reflect the reputation of a publication or researcher among its peers, considering a reference set of reputable researchers. Popularity and reputation are frequently considered to be equivalent properties in the formulation of citation based indices, however these properties are not identical. Indeed, we first show that H-indices and P-scores are correlated with a Kendall-Tau coefficient that exceeds 0.5. However, we also notice that they show important differences. Particularly, we identify publication venues with high H-indices and low P-scores, as well as venues with low H-indices and high P-scores. We provide interpretations for these findings and discuss how they can be used by research funding councils and committees to better support their funding decisions.

Keywords

P-score H-index Bibliographic index Reputation flows 

Notes

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

References

  1. Babić, D., Kutlača, Đ., Živković, L., Štrbac, D., & Semenčenko, D. (2016). Evaluation of the quality of scientific performance of the selected countries of southeast europe. Scientometrics, 106(1), 405–434.CrossRefGoogle Scholar
  2. Baeza-Yates, R., & Ribeiro-Neto, B. (2011). Modern information retrieval: The concepts and technology behind search (2nd ed.). Boston: Addison-Wesley Publishing Company.Google Scholar
  3. Balaban, A. T. (2012). Positive and negative aspects of citation indices and journal impact factors. Scientometrics, 92(2), 241–247.CrossRefGoogle Scholar
  4. Bar-Ilan, J. (2008). Which h-index?—A comparison of WoS, Scopus and Google Scholar. Scientometrics, 74(2), 257–271.CrossRefGoogle Scholar
  5. Benevenuto, F., Laender, A. H., & Alves, B. L. (2016). The H-index paradox: Your coauthors have a higher H-index than you do. Scientometrics, 106(1), 469–474.CrossRefGoogle Scholar
  6. Bollen, J., Rodriquez, M. A., & Van de Sompel, H. (2006). Journal status. Scientometrics, 69(3), 669–687.CrossRefGoogle Scholar
  7. Bornmann, L., & Daniel, H. D. (2005). Does the h-index for ranking of scientists really work? Scientometrics, 65(3), 391–392.CrossRefGoogle Scholar
  8. Bornmann, L., & Marx, W. (2011). The h-index as a research performance indicator. European Science Editing, 37(3), 77–80.Google Scholar
  9. Braun, T., Glänzel, W., & Schubert, A. (2006). A Hirsch-type index for journals. Scientometrics, 69(1), 169–173.CrossRefGoogle Scholar
  10. Brin, S., & Page, L. (1998). The anatomy of a large scale hypertextual Web search engine. Computer Networks and ISDN Systems, 30(1–7), 107–117.CrossRefGoogle Scholar
  11. Chen, J., & Konstan, J. A. (2010). Conference paper selectivity and impact. Communications of the ACM, 53(6), 79–83.CrossRefGoogle Scholar
  12. Ding, Y., & Cronin, B. (2011). Popular and/or prestigious? Measures of scholarly esteem. Information Processing and Management, 47(1), 80–96.CrossRefGoogle Scholar
  13. Egghe, L. (2006). Theory and practise of the g-index. Scientometrics, 69(1), 131–152.MathSciNetCrossRefGoogle Scholar
  14. Egghe, L. (2008). The influence of transformations on the h-index and the g-index. Journal of the American Society for Information Science and Technology, 59(8), 1304–1312.CrossRefGoogle Scholar
  15. Garfield, E. (1955). Citation indexes for science. Science, 122(3159), 108–111.CrossRefGoogle Scholar
  16. Gonçalves, G. D., Figueiredo, F., Almeida, J. M., & Gonçalves, M. A. (2014). Characterizing scholar popularity: A case study in the computer science research community. In Proceedings of the 14th ACM/IEEE-CS joint conference on digital libraries (pp. 57–66).Google Scholar
  17. Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences, 102(46), 16569–16572.CrossRefGoogle Scholar
  18. Hirsch, J. E. (2007). Does the h index have predictive power? Proceedings of the National Academy of Sciences, 104(49), 19193–19198.CrossRefGoogle Scholar
  19. Järvelin, K., & Kekäläinen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4), 422–446.CrossRefGoogle Scholar
  20. Kellner, A. W. A., & Ponciano, L. C. M. O. (2008). H-index in the Brazilian academy of sciences—Comments and concerns. Anais da Academia Brasileira de Ciências, 80(4), 771–781.CrossRefGoogle Scholar
  21. Kendall, M. G. (1955). Rank correlation methods (2nd ed.). New York: Hafner Publishing Company.zbMATHGoogle Scholar
  22. Langville, A. N., & Meyer, C. D. (2006). Google’s pagerank and beyond: The science of search engine rankings. Princeton: Princeton University Press.CrossRefGoogle Scholar
  23. Lee, D. H. (2019). Predictive power of conference-related factors on citation rates of conference papers. Scientometrics, 118(1), 281–304.  https://doi.org/10.1007/s11192-018-2943-z.CrossRefGoogle Scholar
  24. Leydesdorff, L. (2009). How are new citation-based journal indicators adding to the bibliometric toolbox? Journal of the American Society for Information Science and Technology, 60(7), 1327–1336.CrossRefGoogle Scholar
  25. Leydesdorff, L., Zhou, P., & Bornmann, L. (2013). How can journal impact factors be normalized across fields of science? An assessment in terms of percentile ranks and fractional counts. Journal of the American Society for Information Science and Technology, 64(1), 96–107.CrossRefGoogle Scholar
  26. Martins, W. S., Gonçalves, M. A., Laender, A. H., & Pappa, G. L. (2009). Learning to assess the quality of scientific conferences: A case study in computer science. In Proceedings of the 9th ACM/IEEE-CS joint conference on digital libraries (pp. 193–202).Google Scholar
  27. Meyer, C. (1989). Stochastic complementation, uncoupling Markov chains, and the theory of nearly reducible systems. SIAM Review, 31(2), 240–272.MathSciNetCrossRefGoogle Scholar
  28. Nane, T. (2015). Time to first citation estimation in the presence of additional information. In Proceedings of the 15th international society of scientometrics and informetrics conference (pp. 249–260).Google Scholar
  29. Nature (2016). Time to remodel the journal impact factor. Nature  535(7613), 466.Google Scholar
  30. Nelakuditi, S., Gray, C., & Choudhury, R. R. (2011). Snap judgement of publication quality: How to convince a dean that you are a good researcher. ACM SIGMOBILE Mobile Computing and Communications Review, 15(2), 20–23.CrossRefGoogle Scholar
  31. Newman, M. (2010). Networks: An introduction. Oxford: Oxford University Press.CrossRefGoogle Scholar
  32. Pinski, G., & Narin, F. (1976). Citation influence for journal aggregates of scientific publications: Theory, with application to the literature of physics. Information Processing and Management, 12(5), 297–312.CrossRefGoogle Scholar
  33. Piwowar, H. A. (2013). Value all research products. Nature, 493(7431), 159.CrossRefGoogle Scholar
  34. Ribas, S., Ribeiro-Neto, B., de Souza e Silva, E., Ueda, A. H., & Ziviani, N. (2015a). Using reference groups to assess academic productivity in computer science. In Proceedings of the 24th international conference on world wide web, WWW ’15 companion (pp. 603–608). New York, NY, USA: ACM.Google Scholar
  35. Ribas, S., Ribeiro-Neto, B., Santos, R. L., de Souza e Silva, E., Ueda, A., & Ziviani, N. (2015b). Random walks on the reputation graph. In Proceedings of the 2015 international conference on the theory of information retrieval, ICTIR ’15 (pp. 181–190). New York, NY, USA: ACM.Google Scholar
  36. Riikonen, P., & Vihinen, M. (2008). National research contributions: A case study on Finnish biomedical research. Scientometrics, 77(2), 207–222.CrossRefGoogle Scholar
  37. Rossner, M., Van Epps, H., & Hill, E. (2007). Show me the data. The Journal of Cell Biology, 179(6), 1091–1092.CrossRefGoogle Scholar
  38. Saha, S., Saint, S., & Christakis, D. A. (2003). Impact factor: A valid measure of journal quality? Journal of the Medical Library Association, 91(1), 42–46.Google Scholar
  39. Sun, Y., & Giles, C. L. (2007). Popularity weighted ranking for academic digital libraries. In Proceedings of the 29th European conference on IR research, ECIR’07 (pp. 605–612). Berlin: Springer.Google Scholar
  40. Wainer, J., Eckmann, M., Goldenstein, S., & Rocha, A. (2013). How productivity and impact differ across computer science subareas. Communications of the ACM, 56(8), 67–73.CrossRefGoogle Scholar
  41. Waltman, L., & Eck, N. (2013). Source normalized indicators of citation impact: An overview of different approaches and an empirical comparison. Scientometrics, 96(3), 699–716.CrossRefGoogle Scholar
  42. Wendl, M. C. (2007). H-index: However ranked, citations need context. Nature, 449(7161), 403.CrossRefGoogle Scholar
  43. Yan, E., Ding, Y., & Sugimoto, C. R. (2011). P-rank: An indicator measuring prestige in heterogeneous scholarly networks. Journal of the American Society for Information Science and Technology, 62(3), 467–477.Google Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2019

Authors and Affiliations

  1. 1.Departamento de Ciência da Computação Sala 4304Universidade Federal de Minas GeraisBelo HorizonteBrazil
  2. 2.Universidade Federal do Rio de JaneiroRio de JaneiroBrazil

Personalised recommendations