, Volume 118, Issue 2, pp 453–478 | Cite as

Scholarly impact assessment: a survey of citation weighting solutions

  • Liwei Cai
  • Jiahao Tian
  • Jiaying Liu
  • Xiaomei Bai
  • Ivan Lee
  • Xiangjie KongEmail author
  • Feng Xia


Scholarly impact assessment has always been a hot issue. It has played an important role in evaluating researchers, scientific papers, scientific teams, and institutions within science of science. Scholarly impact assessment is also used to address fundamental issues, such as reward evaluation, funding allocation, promotion and recruitment decision. Scholars generally agree that it is more reasonable to use weighted citations to assess the scholarly impact. Although a great number of researchers use weighted citations to access the scholarly impact, there is a lack of a systematic summary of citation weighting methods. To fill the gap, this paper summarizes the existing classical indicators and weighting methods used in measuring scholarly impact from the perspectives of articles, authors and journals. We also summarize the focus of the indicators involved in this paper and the weighting factors that involved in the weighting methods. Finally, we discuss the open issues to try to discover the hidden trends of citation weighting. Through this paper, we can not only have a clearer understanding of the weighting methods in the scholarly impact assessment, but also think more deeply about the weighting factors to be explored.


Weighted citations Scholarly impact assessment Weighting factors 



This work was partially supported by the National Natural Science Foundation of China (61872054), the Fund for Promoting the Reform of Higher Education by Using Big Data Technology, Energizing Teachers and Students to Explore the Future (2017A01002), and the Fundamental Research Funds for the Central Universities (DUT18JC09).


  1. Abujbara, A., Ezra, J., & Radev, D. R. (2013). Purpose and polarity of citation: Towards NLP-based bibliometrics. In Proceedings of the 2013 conference of the North American chapter of the association for computational linguistics: Human language technologies (pp. 596–606).Google Scholar
  2. Alonso, S., Cabrerizo, F. J., Herrera-Viedma, E., & Herrera, F. (2010). hg-index: A new index to characterize the scientific output of researchers based on the h- and g-indices. Scientometrics, 82(2), 391–400.Google Scholar
  3. Anderson, T. R., Hankin, R. K. S., & Killworth, P. D. (2008). Beyond the Durfee square: Enhancing the h-index to score total publication output. Scientometrics, 76(3), 577–588.Google Scholar
  4. Bai, X., Feng, X., Lee, I., Zhang, J., & Ning, Z. (2016a). Identifying anomalous citations for objective evaluation of scholarly article impact. PLoS ONE, 11(9), 1–15.Google Scholar
  5. Bai, X., Hou, J., Du, H., Kong, X., & Xia, F. (2017a). Evaluating the impact of articles with geographical distances between institutions. In International conference on world wide web companion (pp. 1243–1244).Google Scholar
  6. Bai, X., Liu, H., Zhang, F., Ning, Z., Kong, X., Lee, I., et al. (2017b). An overview on evaluating and predicting scholarly article impact. Information, 8(3), 73–86.Google Scholar
  7. Bai, X., Zhang, J., Cui, H., Ning, Z., & Xia, F. (2016b). PNCOIRank: Evaluating the impact of scholarly articles with positive and negative citations. In International conference companion on world wide web (pp. 9–10).Google Scholar
  8. Batista, P. D., Campiteli, M. G., & Kinouchi, O. (2006). Is it possible to compare researchers with different scientific interests. Scientometrics, 68(1), 179–189.Google Scholar
  9. Bergstrom, C. (2007). Eigenfactor: Measuring the value and prestige of scholarly journals. College and Research Libraries News, 68(5), 314–316.Google Scholar
  10. Bollen, J., Rodriquez, M. A., & Van de Sompel, H. (2006). Journal status. Scientometrics, 69(3), 669–687.Google Scholar
  11. Bollen, J., Sompel, H. V. D., Hagberg, A., & Chute, R. (2009). A principal component analysis of 39 scientific impact measures. PLoS ONE, 4(6), e6022–e6032.Google Scholar
  12. Bornmann, L., & Marx, W. (2016). The journal impact factor and alternative metrics: A variety of bibliometric measures has been developed to supplant the impact factor to better assess the impact of individual research papers. EMBO Reports, 17(8), 1094–1097.Google Scholar
  13. Braun, T., Glänzel, W., & Schubert, A. (2006). A Hirsch-type index for journals. Scientometrics, 69(1), 169–173.Google Scholar
  14. Brown, R. J. C. (2009). A simple method for excluding self-citation from the h-index: The b-index. Online Information Review, 33(6), 1129–1136.Google Scholar
  15. Callaway, E. (2016). Publishing elite turns against impact factor. Nature, 535(7611), 210–211.Google Scholar
  16. Cantín, M., Muñoz, M., & Roa, I. (2015). Comparison between impact factor, eigenfactor score, and scimago journal rank indicator in anatomy and morphology journals. International Journal of Morphology, 33(3), 1183–1188.Google Scholar
  17. Casadevall, A., Bertuzzi, S., Buchmeier, M. J., Davis, R. J., Drake, H., Fang, F. C., et al. (2016). Asm journals eliminate impact factor information from journal websites. Applied and Environmental Microbiology, 1(4), 2407–2408.Google Scholar
  18. Casadevall, A., & Fang, F. C. (2014). Causes for the persistence of impact factor mania. MBio, 5(2), e00,064–14.Google Scholar
  19. Cavalcanti, D. C., Prudncio, R. B. C., Pradhan, S. S., Shah, J. Y., & Pietrobon, R. S. (2011). Good to be bad? Distinguishing between positive and negative citations in scientific impact. In IEEE international conference on TOOLS with artificial intelligence (pp. 156–162).Google Scholar
  20. Costas, R., & Bordons, M. O. (2007). The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level. Journal of Informetrics, 1(3), 193–203.Google Scholar
  21. Costas, R., Zahedi, Z., & Wouters, P. (2015). Do altmetrics correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Association for Information Science and Technology, 66(10), 2003–2019.Google Scholar
  22. Davis, P. M. (2008). Eigenfactor: Does the principle of repeated improvement result in better estimates than raw citation counts? Journal of the American Society for Information Science and Technology, 59(13), 2186–2188.Google Scholar
  23. Dellavalle, R. P., Schilling, L. M., Rodriguez, M. A., van de Sompel, H., & Bollen, J. (2007). Refining dermatology journal impact factors using PageRank. Journal of the American Academy of Dermatology, 57(1), 116–119.Google Scholar
  24. Ding, Y., & Yan, E. (2010). Measuring scholarly impact in heterogeneous networks. Proceedings of the Association for Information Science and Technology, 47(1), 1–7.Google Scholar
  25. Dong, P., Loh, M., & Mondry, A. (2005). The “impact factor” revisited. Biomedical digital libraries, 2(1), 1–8.Google Scholar
  26. Eck, N. J. V., & Waltman, L. (2008). Generalizing the h- and g-indices. Journal of Informetrics, 2(4), 263–271.Google Scholar
  27. Egghe, L. (2006). An improvement of the H-index: The G-index. ISSI, 2(1), 8–9.MathSciNetGoogle Scholar
  28. Elliott, D. B. (2014). The impact factor: A useful indicator of journal quality or fatally flawed? Ophthalmic and Physiological Optics, 34(1), 4–7.Google Scholar
  29. Fersht, A. R. (2009). The most influential journals: Impact factor and eigenfactor. Proceedings of the National Academy of Sciences of the United States of America, 106(17), 6883–6884.Google Scholar
  30. Fiala, D. (2012). Time-aware pagerank for bibliographic networks. Journal of Informetrics, 6(3), 370–388.Google Scholar
  31. Fiala, D. (2014). Current index: A proposal for a dynamic rating system for researchers. Journal of the American Society for Information Science and Technology, 65(4), 850–855.Google Scholar
  32. Fiala, D., Rousselot, F., & Jezek, K. (2008). Pagerank for bibliographic networks. Scientometrics, 76(1), 135–158.Google Scholar
  33. Franceschet, M. (2010). The difference between popularity and prestige in the sciences and in the social sciences: A bibliometric analysis. Journal of Informetrics, 4(1), 55–63.Google Scholar
  34. Franceschet, M., & Colavizza, G. (2018). Timerank: A dynamic approach to rate scholars using citations. Journal of Informetrics, 11(4), 1128–1141.Google Scholar
  35. Fujimagari, H., & Fujita, K. (2015). Detecting research fronts using neural network model for weighted citation network analysis. Journal of Information Processing, 23(6), 753–758.Google Scholar
  36. Fujita, K., Kajikawa, Y., Mori, J., & Sakata, I. (2014). Detecting research fronts using different types of weighted citation networks. Journal of Engineering and Technology Management, 32, 129–146.Google Scholar
  37. Galam, S. (2011). Tailor based allocations for multiple authorship: A fractional gh-index. Scientometrics, 89(1), 365–379.Google Scholar
  38. Garfield, E. (2006). The history and meaning of the journal impact factor. JAMA, 295(1), 90–93.Google Scholar
  39. Gómez-Núñez, A. J., Batagelj, V., Vargas-Quesada, B., Moya-Anegón, F., & Chinchilla-Rodríguez, Z. (2014). Optimizing SCImago journal & country rank classification by community detection. Journal of Informetrics, 8(2), 369–383.Google Scholar
  40. Gonzalezpereira, B., Guerrerobote, V. P., & De Moyaanegon, F. (2010). A new approach to the metric of journals scientific prestige: The SJR indicator. Journal of Informetrics, 4(3), 379–391.Google Scholar
  41. Guerrero-Bote, V. P., & Moya-Anegón, F. (2012). A further step forward in measuring journals scientific prestige: The SJR2 indicator. Journal of Informetrics, 6(4), 674–688.Google Scholar
  42. Habibzadeh, F., & Yadollahie, M. (2008). Journal weighted impact factor: A proposal. Journal of Informetrics, 2(2), 164–172.Google Scholar
  43. Hagen, N. T. (2010). Harmonic publication and citation counting: Sharing authorship credit equitably not equally, geometrically or arithmetically. Scientometrics, 84(3), 785–793.Google Scholar
  44. Hall, C. M., & Page, S. J. (2015). Following the impact factor: Utilitarianism or academic compliance? Tourism Management, 51, 309–312.Google Scholar
  45. Harzing, A. W., & van der Wal, R. (2009). A google scholar h-index for journals: An alternative metric to measure journal impact in economics and business. Journal of the American Society for Information Science and Technology, 60(1), 41–46.Google Scholar
  46. Ha, T. C., Tan, S. B., & Soo, K. C. (2006). The journal impact factor: Too much of an impact? Annals-Academy of Medicine Singapore, 35(12), 911–916.Google Scholar
  47. Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16569–16572.zbMATHGoogle Scholar
  48. Jha, R., Jbara, A. A., Qazvinian, V., & Radev, D. R. (2017). NLP-driven citation analysis for scientometrics. Natural Language Engineering, 23(1), 93–130.Google Scholar
  49. Jin, B. (2006). H-Index: An evaluation indicator proposed by scientist. Science Focus, 1(1), 8–9.Google Scholar
  50. Jin, B., Liang, L., Rousseau, R., & Egghe, L. (2007). The R-and AR-indices: Complementing the h-index. Chinese Science Bulletin, 52(6), 855–863.Google Scholar
  51. Kianifar, H., Sadeghi, R., & Zarifmahmoudi, L. (2014). Comparison between impact factor, eigenfactor metrics, and scimago journal rank indicator of pediatric neurology journals. Acta Informatica Medica, 22(2), 103–106.Google Scholar
  52. Kong, X., Mao, M., Wang, W., Liu, J., & Xu, B. (2018). Voprec: Vector representation learning of papers with text information and structural identity for recommendation. IEEE Transactions on Emerging Topics in Computing, PP(99), 1.Google Scholar
  53. Kosmulski, M. (2006). A new Hirsch-type index saves time and works equally well as the original h-index. ISSI Newsletter, 2(3), 4–6.Google Scholar
  54. Leydesdorff, L., & Bornmann, L. (2011). How fractional counting of citations affects the impact factor: Normalization in terms of differences in citation potentials among fields of science. Journal of the American Society for Information Science and Technology, 62(2), 217–229.Google Scholar
  55. Leydesdorff, L., & Opthof, T. (2010). Normalization at the field level: Fractional counting of citations. Journal of Informetrics, 4(4), 644–646.Google Scholar
  56. Leydesdorff, L., & Opthof, T. (2011). Remaining problems with the “New Crown Indicator” (MNCS) of the CWTS. Journal of Informetrics, 5(1), 224–225.Google Scholar
  57. Lundberg, J. (2007). Lifting the crown—citation z-score. Journal of Informetrics, 1(2), 145–154.Google Scholar
  58. Luo, D., Gong, C., Hu, R., Duan, L., & Ma, S. (2016). Ensemble enabled weighted PageRank. arXiv preprint arXiv:160405462.Google Scholar
  59. Mathur, V. P., Sharma, A., et al. (2009). Impact factor and other standardized measures of journal citation: A perspective. Indian Journal of Dental Research, 20(1), 81–85.Google Scholar
  60. Mingers, J., & Yang, L. (2017). Evaluating journal quality: A review of journal citation indicators and ranking in business and management. European Journal of Operational Research, 257(1), 323–337.MathSciNetzbMATHGoogle Scholar
  61. Moed, H. F. (2010a). CWTS crown indicator measures citation impact of a research group’s publication oeuvre. Journal of Informetrics, 4(3), 436–438.Google Scholar
  62. Moed, H. F. (2010b). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4(3), 265–277.Google Scholar
  63. Moustafa, K. (2015). The disaster of the impact factor. Science and Engineering Ethics, 21(1), 139–142.MathSciNetGoogle Scholar
  64. Nykl, M., Campr, M., & Jezek, K. (2015). Author ranking based on personalized pagerank. Journal of Informetrics, 9(4), 777–799.Google Scholar
  65. Nykl, M., Jezek, K., Fiala, D., & Dostal, M. (2014). Pagerank variants in the evaluation of citation networks. Journal of Informetrics, 8(3), 683–692.Google Scholar
  66. Opthof, T., & Leydesdorff, L. (2010). Caveats for the journal and field normalizations in the CWTS (“Leiden”) evaluations of research performance. Journal of Informetrics, 4(3), 423–430.Google Scholar
  67. Ordunamalea, E., & Lopezcozar, E. D. (2014). Google scholar metrics evolution: An analysis according to languages. Scientometrics, 98(3), 2353–2367.Google Scholar
  68. Pajić, D. (2015). On the stability of citation-based journal rankings. Journal of Informetrics, 9(4), 990–1006.Google Scholar
  69. Priem, J., & Hemminger, B. H. (2010). Scientometrics 2.0: New metrics of scholarly impact on the social web. First Monday, 15(7), 1.Google Scholar
  70. Ramin, S., & Shirazi, A. S. (2012). Comparison between Impact factor, SCImago journal rank indicator and eigenfactor score of nuclear medicine journals. Nuclear Medicine Review, 15(2), 132–136.Google Scholar
  71. Roldan-Valadez, E., Orbe-Arteaga, U., & Rios, C. (2018). Eigenfactor score and alternative bibliometrics surpass the impact factor in a 2-years ahead annual-citation calculation: A linear mixed design model analysis of radiology, nuclear medicine and medical imaging journals. La Radiologia Medica, 123, 1–11.Google Scholar
  72. Rousseau, R. (2008). Woeginger’s axiomatisation of the h-index and its relation to the g-index, the h(2)-index and the R2-index. Journal of Informetrics, 2(4), 335–340.Google Scholar
  73. Ruiz-Castillo, J., & Waltman, L. (2015). Field-normalized citation impact indicators using algorithmically constructed classification systems of science. Journal of Informetrics, 9(1), 102–117.Google Scholar
  74. Saad, G. (2006). Exploring the h-index at the author and journal levels using bibliometric data of productive consumer scholars and business-related journals respectively. Scientometrics, 69(1), 117–120.MathSciNetGoogle Scholar
  75. Sayyadi, H., & Getoor, L. (2009). Futurerank: Ranking scientific articles by predicting their future PageRank. In Siam international conference on data mining (pp. 533–544).Google Scholar
  76. Schreiber, M. (2008). EDITORIAL: To share the fame in a fair way, hm modifies h for multi-authored manuscripts. New Journal of Physics, 10(4), 1131–1137.MathSciNetGoogle Scholar
  77. Schreiber, M. (2010). The influence of self-citation corrections and the fractionalised counting of multi-authored manuscripts on the Hirsch index. Annalen Der Physik, 18(9), 607–621.Google Scholar
  78. Schubert, A., Glänzel, W., & Thijs, B. (2006). The weight of author self-citations. A fractional approach to self-citation counting. Scientometrics, 67(3), 503–514.Google Scholar
  79. Sekercioglu, C. H. (2008). Quantifying coauthor contributions. Science, 322(5900), 371–371.Google Scholar
  80. Smolinsky, L. (2016). Expected number of citations and the crown indicator. Journal of Informetrics, 10(1), 43–47.Google Scholar
  81. Su, C., Pan, Y. T., Zhen, Y. N., Ma, Z., Yuan, J. P., Guo, H., et al. (2011). PrestigeRank: A new evaluation method for papers and journals. Journal of Informetrics, 5(1), 1–13.Google Scholar
  82. Vaccario, G., Medo, M., Wider, N., & Mariani, M. S. (2017). Quantifying and suppressing ranking bias in a large citation network. Journal of Informetrics, 11(3), 766–782.Google Scholar
  83. Valenzuela, M., Ha, V., & Etzioni, O. (2015). Identifying meaningful citations. In Association for the advancement of artificial intelligence WS-15-13 (pp. 21–26).Google Scholar
  84. Van, H. B. A., Phelps, J., Barnes, M., & Suk, W. A. (2000). Evaluating scientific impact. Environmental Health Perspectives, 108(9), 392–393.Google Scholar
  85. Walker, D., Xie, H., Yan, K. K., & Maslov, S. (2007). Ranking scientific publications using a model of network traffic. Journal of Statistical Mechanics: Theory and Experiment, 6(6), 1–5.Google Scholar
  86. Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of Informetrics, 10(2), 365–391.Google Scholar
  87. Waltman, L., & van Eck, N. J. (2008). Some comments on the journal weighted impact factor proposed by Habibzadeh and Yadollahie. Journal of Informetrics, 2(4), 369–372.Google Scholar
  88. Waltman, L., & Van Eck, N. J. (2010). The relation between eigenfactor, audience factor, and influence weight. Journal of the Association for Information Science and Technology, 61(7), 1476–1486.Google Scholar
  89. Waltman, L., & van Eck, N. J. (2012). A new methodology for constructing a publication-level classification system of science. Journal of the American Society for Information Science and Technology, 63(12), 2378–2392.Google Scholar
  90. Waltman, L., & van Eck, N. J. (2013a). Source normalized indicators of citation impact: An overview of different approaches and an empirical comparison. Scientometrics, 96(3), 699–716.Google Scholar
  91. Waltman, L., & Van Eck, N. J. (2013b). A systematic empirical comparison of different approaches for normalizing citation impact indicators. Journal of Informetrics, 7(4), 833–849.Google Scholar
  92. Waltman, L., Van Eck, N. J., Van Leeuwen, T. N., & Visser, M. S. (2013). Some modifications to the SNIP journal impact indicator. Journal of Informetrics, 7(2), 272–285.Google Scholar
  93. Waltman, L., Van Eck, N. J., Van Leeuwen, T. N., Visser, M. S., & Van Raan, A. F. J. (2011a). Towards a new crown indicator: Some theoretical considerations. Journal of Informetrics, 5(1), 37–47.Google Scholar
  94. Waltman, L., Yan, E., & van Eck, N. J. (2011b). A recursive field-normalized bibliometric performance indicator: An application to the field of library and information science. Scientometrics, 89(1), 301–314.Google Scholar
  95. Wang, l. (2013). A new h-type index A+ index. Information Magazine, 1, 55–58.Google Scholar
  96. Wang, Y., Tong, Y., & Zeng, M. (2013). Ranking scientific articles by exploiting citations, authors, journals, and time information. In AAAI conference on artificial intelligence (pp. 933–939).Google Scholar
  97. Wang, S., Xie, S., Zhang, X., Li, Z., Yu, P. S., & Shu, X. (2016). Future influence ranking of scientific literature. Computer Science, 9, 1–9.Google Scholar
  98. Wan, X., & Liu, F. (2014). Are all literature citations equally important? Automatic citation strength estimation and its applications. Journal of the Association for Information Science & Technology, 65(9), 1929–1938.Google Scholar
  99. Wesley-Smith, I., Bergstrom, C. T., & West, J. D. (2016). Static ranking of scholarly papers using article-level eigenfactor (ALEF). arXiv preprint arXiv:160608534.Google Scholar
  100. West, J. D., Bergstrom, T. C., & Bergstrom, C. T. (2010). The eigenfactor metrics™: A network approach to assessing scholarly journals. College and Research Libraries, 71(3), 236–244.Google Scholar
  101. West, J. D., Jensen, M. C., Dandrea, R. J., Gordon, G. J., & Bergstrom, C. T. (2013). Author-level eigenfactor metrics: Evaluating the influence of authors, institutions, and countries within the social science research network community. Journal of the Association for Information Science and Technology, 64(4), 787–801.Google Scholar
  102. Wu, Q. (2010). The w-index: A measure to assess scientific impact by focusing on widely cited papers. Journal of the American Society for Information Science and Technology, 61(3), 609–614.Google Scholar
  103. Yan, E., & Ding, Y. (2011). Discovering author impact: A PageRank perspective. Information Processing and Management, 47(1), 125–134.Google Scholar
  104. Zhang, C. T. (2009a). A proposal for calculating weighted citations based on author rank. Embo Reports, 10(5), 416–417.Google Scholar
  105. Zhang, C. T. (2009b). The e-index, complementing the h-index for excess citations. PLoS ONE, 4(5), e5429–e5432.Google Scholar
  106. Zhou, D., Orshanskiy, S. A., Zha, H., & Giles, C. L. (2007). Co-ranking authors and documents in a heterogeneous network. In IEEE international conference on data mining (pp. 739–744).Google Scholar
  107. Zhu, X., Turney, P., Lemire, D., & Vellino, A. (2015). Measuring academic influence: Not all citations are equal. Journal of the Association for Information Science and Technology, 66(2), 408–427.Google Scholar
  108. Zitt, M. (2010). Citing-side normalization of journal impact: A robust variant of the Audience Factor. Journal of Informetrics, 4(3), 392–406.Google Scholar
  109. Zitt, M., Ramanana-Rahary, S., & Bassecoulard, E. (2005). Relativity of citation performance and excellence measures: From cross-field to cross-scale effects of field-normalisation. Scientometrics, 63(2), 373–401.Google Scholar
  110. Zitt, M., & Small, H. (2008). Modifying the journal impact factor by fractional citation weighting: The audience factor. Journal of the American Society for Information Science and Technology, 59(11), 1856–1860.Google Scholar
  111. Zyczkowski, K. (2010). Citation graph, weighted impact factors and performance indices. Scientometrics, 85(1), 301–315.Google Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2018

Authors and Affiliations

  1. 1.Key Laboratory for Ubiquitous Network and Service Software of Liaoning Province, School of SoftwareDalian University of TechnologyDalianChina
  2. 2.Computing CenterAnshan Normal UniversityAnshanChina
  3. 3.School of Information Technology and Mathematical SciencesUniversity of South AustraliaAdelaideAustralia

Personalised recommendations