Abstract
Altmetrics has been widely recognized in the evaluation of articles, books, and journals. However, only a few of scholars have focused on author-level metrics which limits the wide application of altmetrics. Moreover, most of the commonly used altmetrics indicators are selected from international platforms, such as Google + , Facebook, and Twitter, which are not widely available to Chinese scholars. It will be biased if these indicators are directly used to evaluate Chinese scholars. This paper aims to construct an indicator system suitable for the evaluation of Chinese scholars based on altmetrics. We investigate representative platforms which are popular in China and similar with the frequently used altmetrics platforms and extract measurements from them. The principal component analysis is applied to reduce the multicollinearity among the selected indicators and objectively calculate their weights in the altmetrics system. Furthermore, the receiver operating characteristic curve is adopted to give evidence that the proposed altmetrics system is non-specific to the Social Sciences and the Natural Sciences, and the correlation analysis between the altmetrics scores and the traditional bibliometric indicators is implemented to verify its applicability to evaluate Chinese scholars.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Aguillo, I. F. (2020). Altmetrics of the open access institutional repositories: A webometrics approach. Scientometrics, 123(3), 1181–1192. https://doi.org/10.1007/s11192-020-03424-6
Altmetric.com. (2021). How is the Altmetric attention score calculated? https://help.altmetric.com/support/solutions/articles/6000233311-how-is-the-altmetric-attention-score-calculated-.
Álvarez-Bornstein, B., & Montesi, M. (2016). Researchers’ communication on Twitter. A virtual ethnography in the area of information science. Revista española de Documentación Científica, 39(4), 156. https://doi.org/10.3989/redc.2016.4.1352
Archambault, É., & Larivière, V. (2009). History of the journal impact factor: Contingencies and consequences. Scientometrics, 79(3), 635–649. https://doi.org/10.1007/s11192-007-2036-x
Bar-Ilan, J. (2008). Informetrics at the beginning of the 21st century—A review. Journal of Informetrics, 2(1), 1–52. https://doi.org/10.1016/j.joi.2007.11.001
Bornmann, L. (2014). Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of Informetrics, 8(4), 895–903. https://doi.org/10.1016/j.joi.2014.09.005
Bornmann, L. (2015a). Alternative metrics in scientometrics: A meta-analysis of research into three altmetrics. Scientometrics, 103(3), 1123–1144. https://doi.org/10.1007/s11192-015-1565-y
Bornmann, L. (2015b). Letter to the Editor: On the conceptualisation and theorisation of the impact caused by publications. Scientometrics, 103(3), 1145–1148. https://doi.org/10.1007/s11192-015-1588-4
Bornmann, L., & Haunschild, R. (2016). t factor: A metric for measuring impact on Twitter. Malaysian Journal of Library & Information Science, 21(2), 13–20. https://doi.org/10.22452/mjlis.vol21no2.2
Bornmann, L., & Haunschild, R. (2018). Do altmetrics correlate with the quality of papers? A large-scale empirical study based on F1000Prime data. PLoS ONE, 13(5), e0197133. https://doi.org/10.1371/journal.pone.0197133
Bornmann, L., Haunschild, R., & Adams, J. (2019). Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF). Journal of Informetrics, 13(1), 325–340. https://doi.org/10.1016/j.joi.2019.01.008
Cheng, N., & Dong, K. (2018). Knowledge communication on social media: A case study of biomedical science on Baidu Baike. Scientometrics, 116(3), 1749–1770. https://doi.org/10.1007/s11192-018-2828-1
Chua, A. Y. K., & Goh, D. H. (2010). A study of Web 2.0 applications in library websites. Library & Information Science Research, 32, 203–211.
Copiello, S. (2020). Multi-criteria altmetric scores are likely to be redundant with respect to a subset of the underlying information. Scientometrics, 124(1), 819–824. https://doi.org/10.1007/s11192-020-03491-9
Costas, R., Zahedi, Z., & Wouters, P. (2015). Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Association for Information Science and Technology, 66(10), 2003–2019. https://doi.org/10.1002/asi.23309
de Winter, J. C. F. (2015). The relationship between tweets, citations, and article views for PLOS ONE articles. Scientometrics, 102(2), 1773–1779. https://doi.org/10.1007/s11192-014-1445-x
Drongstrup, D., Malik, S., Aljohani, N. R., Alelyani, S., Safder, I., & Hassan, S. U. (2020). Can social media usage of scientific literature predict journal indices of AJG, SNIP and JCR? An altmetric study of economics. Scientometrics, 125(2), 1541–1558. https://doi.org/10.1007/s11192-020-03613-3
Dutta, B. (2016). Altmetric manifesto completes five years (2010–2015). Current Science, 110(1), 17.
Eastday. (2018). News website communication power ranking of China. http://sh.eastday.com/qhtyzt/wzcbl7/n973802/n973803/index.html.
Erdt, M., Aung, H. H., Aw, A. S., Rapple, C., & Theng, Y. L. (2017). Analysing researchers’ outreach efforts and the association with publication metrics: A case study of Kudos. PLoS ONE, 12(8), e0183217. https://doi.org/10.1371/journal.pone.0183217
Erdt, M., Nagarajan, A., Sin, S. C. J., & Theng, Y. L. (2016). Altmetrics: An analysis of the state-of-the-art in measuring research impact on social media. Scientometrics, 109(2), 1117–1166. https://doi.org/10.1007/s11192-016-2077-0
Erfanmanesh, M., Noorhidawati, A., & Abrizah, A. (2019). What can Bookmetrix tell us about the impact of Springer Nature’s books. Scientometrics, 121(1), 521–536. https://doi.org/10.1007/s11192-019-03198-6
Eysenbach, G. (2011). Can tweets predict citations? Metrics of social impact based on twitter and correlation with traditional metrics of scientific impact. Journal of Medical Internet Research, 13(4), e123. https://doi.org/10.2196/jmir.2012
Galligan, F., & Dyas-Correia, S. (2013). Altmetrics: Rethinking the way we measure. Serials Review, 39(1), 56–61. https://doi.org/10.1080/00987913.2013.10765486
Gantz, W. (1981). The influence of researcher methods on television and newspaper news credibility evaluations. Journal of Broadcasting, 25(2), 155–169. https://doi.org/10.1080/08838158109386439
Garfield, E. (1955). Citation indexes for science. Science, 122(3159), 108–111.
Garfield, E. (2006). The history and meaning of the journal impact factor. Journal of the American Medical Association, 295(1), 90–93. https://doi.org/10.1001/jama.295.1.90
Gumpenberger, C., Glänzel, W., & Gorraiz, J. (2016). The ecstasy and the agony of the altmetric score. Scientometrics, 108(2), 977–982. https://doi.org/10.1007/s11192-016-1991-5
Guo, Y., & Xiao, X. (2019). The evaluation of chinese scholars’ influence by altmetrics index. Information Studies: Theory & Application, 42(4), 64–70.
Hassan, S. U., Imran, M., Gillani, U., Aljohani, N. R., Bowman, T. D., & Didegah, F. (2017). Measuring social media activity of scientific literature: An exhaustive comparison of scopus and novel altmetrics big data. Scientometrics, 113(2), 1037–1057. https://doi.org/10.1007/s11192-017-2512-x
Haunschild, R., & Bornmann, L. (2016). Normalization of Mendeley reader counts for impact assessment. Journal of Informetrics, 10(1), 62–73. https://doi.org/10.1016/j.joi.2015.11.003
Haustein, S. (2014). Readership metrics. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: Harnessing multi-dimensional indicators of performance (pp. 327–344). MIT Press.
Haustein, S., Costas, R., & Larivière, V. (2015). Characterizing social media metrics of scholarly papers: the effect of document properties and collaboration patterns. PLoS ONE, 10(3), e0120495. https://doi.org/10.1371/journal.pone.0120495
Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences, 102(46), 16569–16572. https://doi.org/10.1073/pnas.0507655102
Holmberg, K., & Park, H. W. (2018). An altmetric investigation of the online visibility of South Korea-based scientific journals. Scientometrics, 117(1), 603–613. https://doi.org/10.1007/s11192-018-2874-8
Hood, W. W., & Wilson, C. S. (2001). The literature of bibliometrics, scientometrics, and informetrics. Scientometrics, 52(2), 291–314.
Htoo, T. H. H., & Na, J. C. (2017). Disciplinary differences in altmetrics for social sciences. Online Information Review, 41(2), 235–251. https://doi.org/10.1108/OIR-12-2015-0386
Jabaley, C. S., Groff, R. F., Stentz, M. J., Moll, V., Lynde, G. C., Blum, J. M., & O’Reilly-Shah, V. N. (2018). Highly visible sepsis publications from 2012 to 2017: Analysis and comparison of altmetrics and bibliometrics. Journal of Critical Care, 48, 357–371. https://doi.org/10.1016/j.jcrc.2018.09.033
Kaplan, A. M., & Haenlein, M. (2010). Users of the world, unite! The challenges and opportunities of social media. Business Horizons, 53(1), 59–68. https://doi.org/10.1016/j.bushor.2009.09.003
Karanatsiou, D., Misirlis, N., & Vlachopoulou, M. (2017). Bibliometrics and altmetrics literature review: Performance indicators and comparison analysis. Performance Measurement and Metrics, 18(1), 16–27. https://doi.org/10.1108/PMM-08-2016-0036
Kirkup, G. (2010). Academic blogging: Academic practice and academic identity. London Review of Education, 8(1), 75–84. https://doi.org/10.1080/14748460903557803
Kolahi, J., & Khazaei, S. (2016). Altmetric: Top 50 dental articles in 2014. British Dental Journal, 220(11), 569–574. https://doi.org/10.1038/sj.bdj.2016.411
Kunze, K. N., Polce, E. M., Vadhera, A., Williams, B. T., Nwachukwu, B. U., Nho, S. J., & Chahla, J. (2020). What is the predictive ability and academic impact of the altmetrics score and social media attention? The American Journal of Sports Medicine, 48(5), 1056–1062. https://doi.org/10.1177/0363546520903703
Lehmann, S., Jackson, A. D., & Lautrup, B. E. (2006). Measures for measures. Nature, 444(7122), 1003–1004. https://doi.org/10.1038/4441003a
Li, W., Aste, T., Caccioli, F., & Livan, G. (2019). Early coauthorship with top scientists predicts success in academic careers. Nature Communications, 10(1), 5170. https://doi.org/10.1038/s41467-019-13130-4
Maflahi, N., & Thelwall, M. (2016). When are readership counts as useful as citation counts? Scopus versus Mendeley for LIS journals. Journal of the Association for Information Science and Technology, 67(1), 191–199. https://doi.org/10.1002/asi.23369
Metz, C. E. (1978). Basic principles of ROC analysis. Seminars in Nuclear Medicine, 8(4), 283–298. https://doi.org/10.1016/S0001-2998(78)80014-2
Mohammadi, E., & Thelwall, M. (2014). Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows. Journal of the Association for Information Science and Technology, 65(8), 1627–1638. https://doi.org/10.1002/asi.23071
Nan, X., Li, M., & Shi, J. (2020). Using altmetrics for assessing impact of highly-cited books in Chinese book citation index. Scientometrics, 122(3), 1651–1669. https://doi.org/10.1007/s11192-020-03347-2
Narin, F., & Hamilton, K. S. (1996). Bibliometric performance measures. Scientometrics, 36(3), 293–310. https://doi.org/10.1007/BF02129596
Ortega, J. L. (2015). Relationship between altmetric and bibliometric indicators across academic social sites: The case of CSIC’s members. Journal of Informetrics, 9(1), 39–49. https://doi.org/10.1016/j.joi.2014.11.004
Ortega, J. L. (2020). Proposal of composed altmetric indicators based on prevalence and impact dimensions. Journal of Informetrics, 14(4), 101071. https://doi.org/10.1016/j.joi.2020.101071
Priem, J. (2010). I like the term #Altmetrics#. https://twitter.com/#!/jasonpriem/status/25844968813.
Priem, J., Taraborelli, D., Groth, P., & Nevlon, C. (2010). Altmetrics: A manifesto. http://altmetrics.org/manifesto.
Correia, A. M. R., & Teixeira, J. C. (2005). Reforming scholarly publishing and knowledge communication: From the advent of the scholarly journal to the challenges of open access. Online Information Review, 29(4), 349–364. https://doi.org/10.1108/14684520510617802
Sotudeh, H., Dehdarirad, T., & Freer, J. (2018). Gender differences in scientific productivity and visibility in core neurosurgery journals: Citations and social media metrics. Research Evaluation. https://doi.org/10.1093/reseval/rvy003
Sotudeh, H., Mazarei, Z., & Mirzabeigi, M. (2015). CiteULike bookmarks are correlated to citations at journal and author levels in library and information science. Scientometrics, 105(3), 2237–2248. https://doi.org/10.1007/s11192-015-1745-9
Sud, P., & Thelwall, M. (2014). Evaluating altmetrics. Scientometrics, 98(2), 1131–1143. https://doi.org/10.1007/s11192-013-1117-2
Taylor, M. (2020). An altmetric attention advantage for open access books in the humanities and social sciences. Scientometrics, 125(3), 2523–2543. https://doi.org/10.1007/s11192-020-03735-8
Thelwall, M. (2008). Bibliometrics to webometrics. Journal of Information Science, 34(4), 605–621. https://doi.org/10.1177/0165551507087238
Thelwall, M., & Kousha, K. (2014). Academia. edu: Social network or academic network? Journal of the Association for Information Science and Technology, 65(4), 721–731. https://doi.org/10.1002/asi.23038
Thelwall, M., Haustein, S., Larivière, V., & Sugimoto, C. R. (2013). Do altmetrics work? twitter and ten other social web services. PLoS ONE, 8(5), e64841. https://doi.org/10.1371/journal.pone.0064841
Timilsina, M., Davis, B., Taylor, M., & Hayes, C. (2016). Towards predicting academic impact from mainstream news and weblogs: A heterogeneous graph based approach. IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), 2016, 1388–1389. https://doi.org/10.1109/ASONAM.2016.7752425
Torres-Salinas, D., Cabezas-Clavijo, Á., & Jiménez-Contreras, E. (2013). Altmetrics: New indicators for scientific communication in Web 2.0. Comunicar, 21(41), 53–60. https://doi.org/10.3916/C41-2013-05
Torres-Salinas, D., Robinson-Garcia, N., & Gorraiz, J. (2017). Filling the citation gap: Measuring the multidimensional impact of the academic book at institutional level with PlumX. Scientometrics, 113(3), 1371–1384. https://doi.org/10.1007/s11192-017-2539-z
Veletsianos, G., & Kimmons, R. (2012). Networked Participatory Scholarship: Emergent techno-cultural pressures toward open and digital scholarship in online networks. Computers & Education, 58(2), 766–774. https://doi.org/10.1016/j.compedu.2011.10.001
Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E. C. M., Tijssen, R. J. W., van Eck, N. J., van Leeuwen, T. N., van Raan, A. F. J., Visser, M. S., & Wouters, P. (2012). The Leiden ranking 2011/2012: Data collection, indicators, and interpretation. Journal of the American Society for Information Science and Technology, 63(12), 2419–2432. https://doi.org/10.1002/asi.22708
Wang, M., Wang, Z., & Chen, G. (2019). Which can better predict the future success of articles? Bibliometric indices or alternative metrics. Scientometrics, 119(3), 1575–1595. https://doi.org/10.1007/s11192-019-03052-9
Wang, X., Liu, C., Mao, W., & Fang, Z. (2015). The open access advantage considering citation, article usage and social media attention. Scientometrics, 103(2), 555–564. https://doi.org/10.1007/s11192-015-1547-0
Warren, H. R., Raison, N., & Dasgupta, P. (2016). The rise of altmetrics. Journal of the American Medical Association, 317(2), 131–132. https://doi.org/10.1001/jama.2016.18346
Yu, H., Xu, S., Xiao, T., Hemminger, B. M., & Yang, S. (2017). Global science discussed in local altmetrics: Weibo and its comparison with twitter. Journal of Informetrics, 11(2), 466–482. https://doi.org/10.1016/j.joi.2017.02.011
Yu, M. C., Wu, Y. C. J., Alhalabi, W., Kao, H. Y., & Wu, W. H. (2016). ResearchGate: An effective altmetric indicator for active researchers? Computers in Human Behavior, 55, 1001–1006. https://doi.org/10.1016/j.chb.2015.11.007
Zahedi, Z., Costas, R., & Wouters, P. (2014). How well developed are altmetrics? A cross-disciplinary analysis of the presence of “alternative metrics” in scientific publications. Scientometrics, 101(2), 1491–1513. https://doi.org/10.1007/s11192-014-1264-0
Zammuto, R. F., Griffith, T. L., Majchrzak, A., Dougherty, D. J., & Faraj, S. (2007). Information technology and the changing fabric of organization. Organization Science, 18(5), 749–762. https://doi.org/10.1287/orsc.1070.0307
Zhang, X., Wang, X., Zhao, H., Ordóñez de Pablos, P., Sun, Y., & Xiong, H. (2019). An effectiveness analysis of altmetrics indices for different levels of artificial intelligence publications. Scientometrics, 119(3), 1311–1344. https://doi.org/10.1007/s11192-019-03088-x
Zhao, R., & Wei, M. (2017). Academic impact evaluation of Wechat in view of social media perspective. Scientometrics, 112(3), 1777–1791. https://doi.org/10.1007/s11192-017-2439-2
Zoller, D., Doerfel, S., Jäschke, R., Stumme, G., & Hotho, A. (2016). Posted, visited, exported: Altmetrics in the social tagging system BibSonomy. Journal of Informetrics, 10(3), 732–749. https://doi.org/10.1016/j.joi.2016.03.005
Zuccala, A. A., Verleysen, F. T., Cornacchia, R., & Engels, T. C. E. (2015). Altmetrics for the humanities: Comparing goodreads reader ratings with citations to history books. Aslib Journal of Information Management, 67(3), 320–336. https://doi.org/10.1108/AJIM-11-2014-0152
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Guo, Y., Xiao, X. Author-level altmetrics for the evaluation of Chinese scholars. Scientometrics 127, 973–990 (2022). https://doi.org/10.1007/s11192-021-04228-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-021-04228-y