An effectiveness analysis of altmetrics indices for different levels of artificial intelligence publications

  • Xi ZhangEmail author
  • Xianhai Wang
  • Hongke Zhao
  • Patricia Ordóñez de Pablos
  • Yongqiang Sun
  • Hui Xiong


Altmetrics indices are increasingly applied to measure scholarly influence in recent years because they can reflect the influence of research outputs more timely comparing with traditional measurements. Simultaneously, artificial intelligence (AI), as an emerging interdiscipline, has a rapid development in these years. Traditional indices can’t reflect the influence of the AI research outputs quickly, thus more timely altmetrics indices are needed. In this paper, we conduct four studies about altmetrics indices and AI research outputs based on the datasets collected from and Scopus database. First, we provide a review of the research status in the AI field. Second, we show the AI researches that attracted the most attention. Third, we demonstrate the general effectiveness of altmetrics indices in the AI field. Last, we examine the effectiveness of altmetrics indices for different levels of AI journal papers and AI conference papers. Our results indicate that there is a rapid increase of AI publications and the public has paid more attention to AI research outputs since 2011. It is found that altmetrics indices are effective to discriminate highly cited publications and publications whose citation counts increase quickly. Among all Altmetric sub-indicators, Number of Mendeley readers is the most effective. Moreover, the results indicate that altmetrics indices are more effective in high levels of AI journal papers and AI conference papers. The main contribution of this paper is investigating the effectiveness of altmetrics indices from the perspective of different levels of publications. This study lays the foundation for further investigations about effectiveness of altmetrics indices from new perspectives, and it has important implication for the studies about the impact of social media on the scientific community.


Altmetrics Bibliometrics Artificial intelligence Highly cited publication Increase of citation count Citation analysis 



The study is supported by funds from National Natural Science Foundation of China (Nos: 71722005 and 71571133 and 71790594 and 71790590). And from Natural Science Foundation of Tianjin (No. 18JCJQJC45900), the Humanities and Social Sciences Foundation of the Ministry of Education, China (Project No. 16YJC870011). We are grateful to for providing the data.


  1. Adie, E., & Roe, W. (2013). Altmetric: Enriching scholarly content with article-level discussion and metrics. Learned Publishing, 26(1), 11–17.CrossRefGoogle Scholar
  2. (2017). Sources of attention-altmetric track a unique range of online sources to capture the conversations relating to research outputs.
  3. Bando, K. (2014). Importance of Mendeley readership on altmetrics based on the Altmetric score. Journal of Molecular Structure, 435(2), 123–132.Google Scholar
  4. Bench-Capon, T. J., & Dunne, P. E. (2007). Argumentation in artificial intelligence. Artificial Intelligence, 171(10–15), 619–641.MathSciNetCrossRefzbMATHGoogle Scholar
  5. Bornmann, L. (2014). Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of Informetrics, 8(4), 895–903.CrossRefGoogle Scholar
  6. Boyd, S., & Vandenberghe, L. (2004). Convex optimization. Cambridge: Cambridge University Press.CrossRefzbMATHGoogle Scholar
  7. Colledge, L., Moya-Anegón, F. D., Guerrero-Bote, V., López-Illescas, C., Aisati, M. H. E., & Moed, H. (2010). SJR and SNIP: Two new journal metrics in Elsevier’s Scopus. Serials the Journal for the Serials Community, 23(3), 215–221.CrossRefGoogle Scholar
  8. Costas, R., Zahedi, Z., & Wouters, P. (2015). Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Association for Information Science & Technology, 66(10), 2003–2019.CrossRefGoogle Scholar
  9. Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In Paper presented at the IEEE computer society conference on computer vision and pattern recognition.Google Scholar
  10. Daud, A., Ahmad, M., Malik, M. S., & Che, D. (2015). Using machine learning techniques for rising star prediction in co-author network. Scientometrics, 102(2), 1687–1711.CrossRefGoogle Scholar
  11. González-Pereira, B., Guerrero-Bote, V. P., & Moya-Anegón, F. (2010). A new approach to the metric of journals’ scientific prestige: The SJR indicator. Journal of Informetrics, 4(3), 379–391.CrossRefGoogle Scholar
  12. Gunther, E. (2011). Can tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact. Journal of Medical Internet Research, 13(4), e123.CrossRefGoogle Scholar
  13. Hartley, R., & Zisserman, A. (2003). Multiple view geometry in computer vision. Cambridge: Cambridge University Press.zbMATHGoogle Scholar
  14. Harvey, C., Morris, H., Kelly, A., & Rowlinson, M. (2010). Academic journal quality guide. London: The Association of Business Schools.Google Scholar
  15. Hassan, S. U., & Gillani, U. A. (2016). Altmetrics of “altmetrics” using Google Scholar, Twitter, Mendeley, Facebook, Google-plus, CiteULike, Blogs and Wiki.Google Scholar
  16. Hassan, S. U., & Haddawy, P. (2013). Measuring international knowledge flows and scholarly impact of scientific research. Scientometrics, 94(1), 163–179.CrossRefGoogle Scholar
  17. Hassan, S. U., Imran, M., Gillani, U., Aljohani, N. R., Bowman, T. D., & Didegah, F. (2017). Measuring social media activity of scientific literature: An exhaustive comparison of scopus and novel altmetrics big data. Scientometrics, 113(1), 1–21.CrossRefGoogle Scholar
  18. Haykin, S. (1994). Neural networks: A comprehensive foundation. Upper Saddle River: Prentice Hall.zbMATHGoogle Scholar
  19. Heagerty, P. J., & Zheng, Y. (2005). Survival model predictive accuracy and ROC curves. Biometrics, 61(1), 92.MathSciNetCrossRefzbMATHGoogle Scholar
  20. Holmberg, K., & Thelwall, M. (2014). Disciplinary differences in Twitter scholarly communication. Scientometrics, 101(2), 1027–1042.CrossRefGoogle Scholar
  21. Hu, B., Lu, Z., Hang, L., & Chen, Q. (2014). Convolutional neural network architectures for matching natural language sentences. In Paper presented at the international conference on neural information processing systems.Google Scholar
  22. Kemp, C., Tenenbaum, J. B., Griffiths, T. L., Yamada, T., & Ueda, N. (2006). Learning systems of concepts with an infinite relational model. In Paper presented at the national conference on artificial intelligence. Google Scholar
  23. Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. In International conference on learning representations. arXiv:1412.6980.
  24. Kousha, K., & Thelwall, M. (2017). Are wikipedia citations important evidence of the impact of scholarly articles and books? Journal of the Association for Information Science & Technology, 68(3), 762–779.CrossRefGoogle Scholar
  25. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Paper presented at the Advances in neural information processing systems.Google Scholar
  26. Lavin, A., & Gray, S. (2016). Fast algorithms for convolutional neural networks. In Paper presented at the computer vision and pattern recognition. Google Scholar
  27. Lee, Y. G., Lee, J. D., Song, Y. I., & Lee, S. J. (2007). An in-depth empirical analysis of patent citation counts using zero-inflated count data model: The case of KIST. Scientometrics, 70(1), 27–39.CrossRefGoogle Scholar
  28. Ling, C. X., Huang, J., & Zhang, H. (2003). AUC: A better measure than accuracy in comparing learning algorithms. In Paper presented at the Canadian society for computational studies of intelligence conference on advances in artificial intelligence. Google Scholar
  29. Liu, X. Z., & Fang, H. (2017). What we can learn from tweets linking to research papers. Scientometrics, 111, 1–21.CrossRefGoogle Scholar
  30. Lyder, A. (2008). Mechanical design of Odin, an extendable heterogeneous deformable modular robot: Intelligent robots and systems. In Paper presented at the IEEE/RSJ international conference on intelligent robots and systems, 2008.Google Scholar
  31. Metz, C. E. (1978). Basic principles of ROC analysis. Seminars in Nuclear Medicine, 8(4), 283.CrossRefGoogle Scholar
  32. Ming, J., Han, J., & Danilevsky, M. (2011). Ranking-based classification of heterogeneous information networks. In Paper presented at the Acm Sigkdd international conference on knowledge discovery and data mining.Google Scholar
  33. Moed, H. F. (2009). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4(3), 265–277.CrossRefGoogle Scholar
  34. Moed, H. F., & Halevi, G. (2015). Multidimensional assessment of scholarly research impact. Journal of the Association for Information Science & Technology, 66(10), 1988–2002.CrossRefGoogle Scholar
  35. Nguyen, V., Tomatis, N., Tomatis, N., & Siegwart, R. (2007). A comparison of line extraction algorithms using 2D range data for indoor mobile robotics. Autonomous Robots, 23(2), 97–111.CrossRefGoogle Scholar
  36. Noorden, R. V. (2014). Scientists and the social network. Nature, 512, 126–129.CrossRefGoogle Scholar
  37. Peoples, B. K., Midway, S. R., Sackett, D., Lynch, A., & Cooney, P. B. (2016). Twitter predicts citation rates of ecological research. PLoS ONE, 11(11), e0166570.CrossRefGoogle Scholar
  38. Piwowar, H. (2013). Altmetrics: Value all research products. Nature, 493(7431), 159.Google Scholar
  39. Pooladian, A., & Borrego, Á. (2017). Methodological issues in measuring citations in Wikipedia: A case study in library and information science. Scientometrics, 113(3), 1–10.Google Scholar
  40. Priem, J., & Hemminger, B. H. (2010). Scientometrics 2.0: New metrics of scholarly impact on the social Web. First Monday, 15(7). Available at:
  41. Priem, J., Piwowar, H. A., & Hemminger, B. M. (2012). Altmetrics in the wild: Using social media to explore scholarly impact. arXiv:1203.4745.
  42. Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2010). Altmetrics: A manifesto.
  43. Qian, Y., Rong, W., Jiang, N., Tang, J., & Xiong, Z. (2017). Citation regression analysis of computer science publications in different ranking categories and subfields. Scientometrics, 110(3), 1–24.CrossRefGoogle Scholar
  44. Rendell, L. (1983). A new basis for state-space learning systems and a successful implementation. Artificial Intelligence, 20(4), 369–392.CrossRefGoogle Scholar
  45. Ringelhan, S., Wollersheim, J., & Welpe, I. M. (2015). I like, i cite? Do Facebook likes predict the impact of scientific work? PLoS ONE, 10(8), e0134389.CrossRefGoogle Scholar
  46. Russell, S. J., & Norvig, P. (2002). Artificial intelligence: A modern approach. Petaling Jaya: Pearson Education Limited.zbMATHGoogle Scholar
  47. Salvatori, H. (2014). Coverage and adoption of altmetrics sources in the bibliometric community. Scientometrics. Scientometrics, 101(2), 1145–1163.CrossRefGoogle Scholar
  48. Sause, M. G. R., Gribov, A., Unwin, A. R., & Horn, S. (2012). Pattern recognition approach to identify natural clusters of acoustic emission signals. Pattern Recognition Letters, 33(1), 17–23.CrossRefGoogle Scholar
  49. Shu, F., & Haustein, S. (2017). On the citation advantage of tweeted papers at the journal level. Proceedings of the Association for Information Science & Technology, 54(1), 366–372.CrossRefGoogle Scholar
  50. Sud, P., & Thelwall, M. (2014). Evaluating altmetrics. Scientometrics, 98(2), 1131–1143.CrossRefGoogle Scholar
  51. Sugimoto, C. R., Work, S., Larivière, V., & Haustein, S. (2017). Scholarly use of social media and altmetrics: A review of the literature. Journal of the Association for Information Science & Technology, 68(9), 2037–2062.CrossRefGoogle Scholar
  52. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In Paper presented at the proceedings of the IEEE conference on computer vision and pattern recognition. Google Scholar
  53. Teixeira, D. S., Jaime, A., & Memon, A. R. (2017). CiteScore: A cite for sore eyes, or a valuable, transparent metric? Scientometrics, 111(1), 1–4.CrossRefGoogle Scholar
  54. Thelwall, M., Haustein, S., Larivière, V., & Sugimoto, C. R. (2013). Do altmetrics work? Twitter and ten other social web services. PLoS ONE, 8(5), e64841.CrossRefGoogle Scholar
  55. Thelwall, M., & Wilson, P. (2016). Mendeley readership altmetrics for medical articles: An analysis of 45 fields. Journal of the Association for Information Science & Technology, 67(8), 1962–1972.CrossRefGoogle Scholar
  56. Winter, J. C. F. D. (2015). The relationship between tweets, citations, and article views for PLOS ONE articles. Scientometrics, 102(2), 1773–1779.CrossRefGoogle Scholar
  57. Wouters, P., & Costas, R. (2012). Users, narcissism and control—Tracking the impact of scholarly publications in the 21st century. Available at:
  58. Xia, F., Su, X., Wang, W., Zhang, C., Ning, Z., & Lee, I. (2016). Bibliographic analysis of nature based on Twitter and Facebook altmetrics data. PLoS ONE, 11(12), e0165997.CrossRefGoogle Scholar
  59. Xin, S., Alberto, P., & Johan, B. (2012). How the scientific community reacts to newly submitted preprints: Article downloads, Twitter mentions, and citations. PLoS ONE, 7(11), e47523.CrossRefGoogle Scholar
  60. Yu, H. (2017). Context of altmetrics data matters: An investigation of count type and user category. Scientometrics, 111(1), 1–17.CrossRefGoogle Scholar
  61. Zahedi, Z., Costas, R., & Wouters, P. (2013). What is the impact of the publications read by the different Mendeley users? Could they help to identify alternative types of impact. In Plos Alm workshop.Google Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2019

Authors and Affiliations

  • Xi Zhang
    • 1
    Email author
  • Xianhai Wang
    • 1
  • Hongke Zhao
    • 1
  • Patricia Ordóñez de Pablos
    • 2
  • Yongqiang Sun
    • 3
  • Hui Xiong
    • 4
  1. 1.College of Management and EconomicsTianjin UniversityTianjinChina
  2. 2.Department of Business AdministrationUniversity of OviedoOviedoSpain
  3. 3.School of Information ManagementWuhan UniversityWuhanChina
  4. 4.Rutgers Business School - Newark and New BrunswickRutgers UniversityNewarkUSA

Personalised recommendations