Abstract
This study analyzed 16,799 journal papers and 98,773 conference papers published by IEEE Xplore in 2016 to investigate the relationships among usage counts, Mendeley readership, and citations through descriptive, regression, and mediation analyses. Differences in the relationship among these metrics between journal and conference papers are also studied. Results showed that there is no significant difference between journal and conference papers in the distribution patterns and accumulation rates of the three metrics. However, the correlation coefficients of the interrelationships between the three metrics were lower in conference papers compared to journal papers. Secondly, funding, international collaboration, and open access are positively associated with all three metrics, except for the case of funding on the usage metrics of conference papers. Furthermore, early Mendeley readership is a better predictor of citations than early usage counts and performs better for journal papers. Finally, we reveal that early Mendeley readership partially mediates between early usage counts and citation counts in the journal and conference papers. The main difference is that conference papers rely more on the direct effect of early usage counts on citations. This study contributes to expanding the existing knowledge on the relationships among usage counts, Mendeley readership, and citations in journal and conference papers, providing new insights into the relationship between the three metrics through mediation analysis.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
Data and codes can be accessed from this URL: https://github.com/Tianwencan/IEEE_usage
References
Aduku, K. J., Thelwall, M., & Kousha, K. (2017). Do Mendeley reader counts reflect the scholarly impact of conference papers? An investigation of computer science and engineering. Scientometrics, 112(1), 573–581. https://doi.org/10.1007/s11192-017-2367-1
Alvarez-Bornstein, B., & Bordons, M. (2021). Is funding related to higher research impact? Exploring its relationship and the mediating role of collaboration in several disciplines. Journal of Informetrics, 15(1), 101102. https://doi.org/10.1016/j.joi.2020.101102
Baron, R., & Kenny, D. (1986). The moderator mediator variable distinction in social psychological-research—Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51(6), 1173–1182. https://doi.org/10.1037/0022-3514.51.6.1173
Bollen, J., Luce, R., Vemulapalli, S. S., et al. (2003). Usage analysis for the identification of research trends in digital libraries. D-Lib Magazine, 9(5), e1395. https://doi.org/10.1045/may2003-bollen
Bollen, J., & van de Sompel, H. (2008). Usage impact factor: The effects of sample characteristics on usage-based impact metrics. Journal of the American Society for Information Science and Technology, 59(1), 136–149. https://doi.org/10.1002/asi.20746
Breitzman, A. (2021). The relationship between web usage and citation statistics for electronics and information technology articles. Scientometrics, 126(3), 2085–2105. https://doi.org/10.1007/s11192-020-03851-5
Chen, W. M. Y., Bukhari, M., Cockshull, F., et al. (2020). The relationship between citations, downloads and alternative metrics in rheumatology publications: A bibliometric study. Rheumatology, 59(2), 277–280. https://doi.org/10.1093/rheumatology/kez163
Chi, P.-S., & Glanzel, W. (2017). An empirical investigation of the associations among usage, scientific collaboration and citation impact. Scientometrics, 112(1), 403–412. https://doi.org/10.1007/s11192-017-2356-4
Chi, P.-S., & Glänzel, W. (2018). Comparison of citation and usage indicators in research assessment in scientific disciplines and journals. Scientometrics, 116(1), 537–554. https://doi.org/10.1007/s11192-018-2708-8
Chi, P.-S., Gorraiz, J., & Glanzel, W. (2019). Comparing capture, usage and citation indicators: An altmetric analysis of journal papers in chemistry disciplines. Scientometrics, 120(3), 1461–1473. https://doi.org/10.1007/s11192-019-03168-y
Costas, R., Perianes-Rodríguez, A., & Ruiz-Castillo, J. (2017). On the quest for currencies of science: Field “exchange rates” for citations and Mendeley readership. Aslib Journal of Information Management, 69(5), 557–575. https://doi.org/10.1108/AJIM-01-2017-0023
Costas, R., Zahedi, Z., & Wouters, P. (2015). Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Association for Information Science and Technology, 66(10), 2003–2019. https://doi.org/10.1002/asi.23309
Cui, Y., Wang, Y., Liu, X., et al. (2023). Multidimensional scholarly citations: Characterizing and understanding scholars’ citation behaviors. Journal of the Association for Information Science and Technology, 74(1), 115–127. https://doi.org/10.1002/asi.24709
Ding, Y., Dong, X., Bu, Y., et al. (2021). Revisiting the relationship between downloads and citations: A perspective from papers with different citation patterns in the case of the Lancet. Scientometrics, 126(9), 7609–7621. https://doi.org/10.1007/s11192-021-04099-3
Ebrahimy, S., Mehrad, J., Setareh, F., & Hosseinchari, M. (2016). Path analysis of the relationship between visibility and citation: The mediating roles of save, discussion, and recommendation metrics. Scientometrics, 109(3), 1497–1510. https://doi.org/10.1007/s11192-016-2130-z
Erdt, M., Nagarajan, A., Sin, S.-C.J., et al. (2016). Altmetrics: An analysis of the state-of-the-art in measuring research impact on social media. Scientometrics, 109(2), 1117–1166. https://doi.org/10.1007/s11192-016-2077-0
Fang, Z., Costas, R., Tian, W., et al. (2020a). An extensive analysis of the presence of altmetric data for Web of Science publications across subject fields and research topics. Scientometrics, 124(3), 2519–2549. https://doi.org/10.1007/s11192-020-03564-9
Fang, Z., Costas, R., Tian, W., et al. (2021). How is science clicked on Twitter? Click metrics for Bitly short links to scientific publications. Journal of the Association for Information Science and Technology, 72(7), 918–932. https://doi.org/10.1002/asi.24458
Fang, Z., Costas, R., & Wouters, P. (2022). User engagement with scholarly tweets of scientific papers: A large-scale and cross-disciplinary analysis. Scientometrics, 127(8), 4523–4546. https://doi.org/10.1007/s11192-022-04468-6
Fang, Z., Dudek, J., & Costas, R. (2020b). The stability of Twitter metrics: A study on unavailable Twitter mentions of scientific publications. Journal of the Association for Information Science and Technology, 71(12), 1455–1469. https://doi.org/10.1002/asi.24344
Freyne, J., Coyle, L., Smyth, B., & Cunningham, P. (2010). Relative status of journal and conference publications in computer science. Communications of the ACM, 53(11), 124–132. https://doi.org/10.1145/1839676.1839701
Geng, Y., Cao, R., Han, X., et al. (2022). Scientists are working overtime: When do scientists download scientific papers? Scientometrics, 127(11), 6413–6429. https://doi.org/10.1007/s11192-022-04524-1
Glaenzel, W., & Gorraiz, J. (2015). Usage metrics versus altmetrics: Confusing terminology? Scientometrics, 102(3), 2161–2164. https://doi.org/10.1007/s11192-014-1472-7
Glänzel, W., & Chi, P.-S. (2020). The big challenge of Scientometrics 2.0: Exploring the broader impact of scientific research in public health. Scientometrics, 125(2), 1011–1031. https://doi.org/10.1007/s11192-020-03473-x
Guerrero-Bote, V. P., & Moya-Anegón, F. (2014). Relationship between downloads and citations at journal and paper levels, and the influence of language. Scientometrics, 101(2), 1043–1065. https://doi.org/10.1007/s11192-014-1243-5
Haunschild, R., & Bornmann, L. (2023). Identification of potential young talented individuals in the natural and life sciences: A bibliometric approach. Journal of Informetrics, 17(3), 101394. https://doi.org/10.1016/j.joi.2023.101394
Haustein, S., Bowman, T. D., & Costas, R. (2016). Interpreting ‘altmetrics’: Viewing acts on social media through the lens of citation and social theories. In S. R. Sugimoto (Ed.), Theories of informetrics and scholarly communication (pp. 372–406). De Gruyter Saur.
Haustein, S., Peters, I., Sugimoto, C. R., et al. (2014). Tweeting biomedicine: An analysis of tweets and citations in the biomedical literature. Journal of the Association for Information Science and Technology, 65(4), 656–669. https://doi.org/10.1002/asi.23101
Hicks, D., Wouters, P., Waltman, L., et al. (2015). Bibliometrics: The Leiden manifesto for research metrics. Nature, 520(7548), 429–431. https://doi.org/10.1038/520429a
Holmberg, K., Hedman, J., Bowman, T. D., et al. (2020). Do articles in open access journals have more frequent altmetric activity than articles in subscription-based journals? An investigation of the research output of Finnish universities. Scientometrics, 122(1), 645–659. https://doi.org/10.1007/s11192-019-03301-x
Iacobucci, D., Saldanha, N., & Deng, X. (2007). A meditation on mediation: Evidence that structural equations models perform better than regressions. Journal of Consumer Psychology, 17(2), 139–153. https://doi.org/10.1016/S1057-7408(07)70020-7
Khan, M. S., & Younas, M. (2017). Analyzing readers behavior in downloading articles from IEEE digital library: A study of two selected journals in the field of education. Scientometrics, 110(3), 1523–1537. https://doi.org/10.1007/s11192-016-2232-7
Kudlow, P., Cockerill, M., Toccalino, D., et al. (2017). Online distribution channel increases article usage on Mendeley: A randomized controlled trial. Scientometrics, 112(3), 1537–1556. https://doi.org/10.1007/s11192-017-2438-3
Li, X., Thelwall, M., & Giustini, D. (2012). Validating online reference managers for scholarly impact measurement. Scientometrics, 91(2), 461–471. https://doi.org/10.1007/s11192-011-0580-x
Lippi, G., & Favaloro, E. J. (2013). Article downloads and citations: Is there any relationship? Clinica Chimica Acta, 415, 195–195. https://doi.org/10.1016/j.cca.2012.10.037
McGillivray, B., & Astell, M. (2019). The relationship between usage and citations in an open access mega-journal. Scientometrics, 121(2), 817–838. https://doi.org/10.1007/s11192-019-03228-3
Mehmetoglu, M. (2018). Medsem: A Stata package for statistical mediation analysis. International Journal of Computational Economics and Econometrics, 8(1), 63–78. https://doi.org/10.1504/IJCEE.2018.088321
Mohammadi, E., & Thelwall, M. (2014). Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows. Journal of the Association for Information Science and Technology, 65(8), 1627–1638. https://doi.org/10.1002/asi.23071
Mohammadi, E., Thelwall, M., Haustein, S., et al. (2015). Who reads research articles? An altmetrics analysis of Mendeley user categories. Journal of the Association for Information Science and Technology, 66(9), 1832–1846. https://doi.org/10.1002/asi.23286
Mohammadi, E., Thelwall, M., & Kousha, K. (2016). Can Mendeley bookmarks reflect readership? A survey of user motivations. Journal of the Association for Information Science and Technology, 67(5), 1198–1209. https://doi.org/10.1002/asi.23477
Raguseo, E., Pigni, F., & Vitari, C. (2021). Streams of digital data and competitive advantage: The mediation effects of process efficiency and product effectiveness. Information & Management, 58(4), 103451. https://doi.org/10.1016/j.im.2021.103451
Rowlands, I., & Nicholas, D. (2007). The missing link: Journal usage metrics. Aslib Proceedings, 59(3), 222–228. https://doi.org/10.1108/00012530710752025
Schloegl, C., & Gorraiz, J. (2010). Comparison of citation and usage indicators: The case of oncology journals. Scientometrics, 82(3), 567–580. https://doi.org/10.1007/s11192-010-0172-1
Schloegl, C., Gorraiz, J., Gumpenberger, C., et al. (2014). Comparison of downloads, citations and readership data for two information systems journals. Scientometrics, 101(2), 1113–1128. https://doi.org/10.1007/s11192-014-1365-9
Singh, R., Charan, P., & Chattopadhyay, M. (2023). Relational capabilities and performance: Examining the moderation-mediation effect of organisation structures and dynamic capability. Knowledge Management Research & Practice, 21(1), 92–106. https://doi.org/10.1080/14778238.2020.1843984
Sugimoto, C. R., Work, S., Larivière, V., et al. (2017). Scholarly use of social media and altmetrics: A review of the literature. Journal of the Association for Information Science and Technology, 68(9), 2037–2062. https://doi.org/10.1002/asi.23833
Taylor, M. (2023). Slow, slow, quick, quick, slow: Five altmetric sources observed over a decade show evolving trends, by research age, attention source maturity and open access status. Scientometrics, 128(4), 2175–2200. https://doi.org/10.1007/s11192-023-04653-1
Thelwall, M. (2017a). Are Mendeley reader counts high enough for research evaluations when articles are published? Aslib Journal of Information Management, 69(2), 174–183. https://doi.org/10.1108/AJIM-01-2017-0028
Thelwall, M. (2017b). Are Mendeley reader counts useful impact indicators in all fields? Scientometrics, 113(3), 1721–1731. https://doi.org/10.1007/s11192-017-2557-x
Thelwall, M. (2018). Early Mendeley readers correlate with later citation counts. Scientometrics, 115(3), 1231–1240. https://doi.org/10.1007/s11192-018-2715-9
Thelwall, M. (2020). Mendeley reader counts for US computer science conference papers and journal articles. Quantitative Science Studies, 1(1), 347–359. https://doi.org/10.1162/qss_a_00010
Thelwall, M., & Kousha, K. (2017). ResearchGate articles: Age, discipline, audience size, and impact. Journal of the Association for Information Science and Technology, 68(2), 468–479. https://doi.org/10.1002/asi.23675
Thelwall, M., & Maflahi, N. (2015). Are scholarly articles disproportionately read in their own country? An analysis of Mendeley readers. Journal of the Association for Information Science and Technology, 66(6), 1124–1135. https://doi.org/10.1002/asi.23252
Tian, W., Hu, Z., & Wang, X. (2019). Upgrading from 3G to 5G: Topic evolution and persistence among scientists. In: Proceedings of the 17th international conference on scientometrics and informetrics (pp. 1156–1165).
Vaughan, L., Tang, J., & Yang, R. (2017). Investigating disciplinary differences in the relationships between citations and downloads. Scientometrics, 111(3), 1533–1545. https://doi.org/10.1007/s11192-017-2308-z
Vilchez-Roman, C., & Vara-Horna, A. (2021). Usage, content and citation in open access publication: Any interaction effects? Scientometrics, 126(12), 9457–9476. https://doi.org/10.1007/s11192-021-04178-5
Vrettas, G., & Sanderson, M. (2015). Conferences versus journals in computer science. Journal of the Association for Information Science and Technology, 66(12), 2674–2684. https://doi.org/10.1002/asi.23349
Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of Informetrics, 10(2), 365–391. https://doi.org/10.1016/j.joi.2016.02.007
Wan, J., Hua, P., Rousseau, R., et al. (2010). The journal download immediacy index (DII): Experiences using a Chinese full-text database. Scientometrics, 82(3), 555–566. https://doi.org/10.1007/s11192-010-0171-2
Wang, X., Fang, Z., & Sun, X. (2016). Usage patterns of scholarly articles on Web of Science: A study on Web of Science usage count. Scientometrics, 109(2), 917–926. https://doi.org/10.1007/s11192-016-2093-0
Wang, X., Liu, C., Mao, W., et al. (2015). The open access advantage considering citation, article usage and social media attention. Scientometrics, 103(3), 1149–1149. https://doi.org/10.1007/s11192-015-1589-3
Wang, X., Mao, W., Xu, S., et al. (2014). Usage history of scientific literature: Nature metrics and metrics of Nature publications. Scientometrics, 98(3), 1923–1933. https://doi.org/10.1007/s11192-013-1167-5
Wang, Z., Chen, Y., & Glanzel, W. (2020). Preprints as accelerator of scholarly communication: An empirical analysis in Mathematics. Journal of Informetrics, 14(4), 101097. https://doi.org/10.1016/j.joi.2020.101097
Zahedi, Z., & Costas, R. (2018). General discussion of data quality challenges in social media metrics: Extensive comparison of four major altmetric data aggregators. PLoS ONE, 13(5), e0197326. https://doi.org/10.1371/journal.pone.0197326
Zahedi, Z., Costas, R., & Wouters, P. (2014). How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications. Scientometrics, 101(2), 1491–1513. https://doi.org/10.1007/s11192-014-1264-0
Zahedi, Z., Costas, R., & Wouters, P. (2017). Mendeley readership as a filtering tool to identify highly cited publications. Journal of the Association for Information Science and Technology, 68(10), 2511–2521. https://doi.org/10.1002/asi.23883
Zahedi, Z., & Haustein, S. (2018). On the relationships between bibliographic characteristics of scientific documents and citation and Mendeley readership counts: A large-scale analysis of Web of Science publications. Journal of Informetrics, 12(1), 191–202. https://doi.org/10.1016/j.joi.2017.12.005
Zhang, G., Wang, Y., Xie, W., et al. (2021). The open access usage advantage: A temporal and spatial analysis. Scientometrics, 126(7), 6187–6199. https://doi.org/10.1007/s11192-020-03836-4
Zhao, X., Lynch, J. G., & Chen, Q. (2010). Reconsidering Baron and Kenny: Myths and truths about mediation analysis. Journal of Consumer Research, 37(2), 197–206. https://doi.org/10.1086/651257
Acknowledgements
The authors are grateful to the anonymous reviewers for their helpful comments and suggestions.
Funding
This study is supported by the National Natural Science Foundation of China (Grant No. 71974029). Rodrigo Costas is partially funded by the South African DSI-NRF Centre of Excellence in Scientometrics and Science, Technology and Innovation Policy (SciSTIP). Wencan Tian is financially supported by the China Scholarship Council (Grant No. 202106060134). Zhichao Fang is funded by the Scientific Research Funding of Renmin University of China (Grant No. 23XNF037).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
One of the authors (Rodrigo Costas) is a member of the Distinguished Reviewers Board of the journal Scientometrics.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix
Appendix
See Tables
6,
7,
8 and
9; Fig.
Path coefficients for the mediation effects models for conference papers in various disciplines. The total effects are displayed in a larger font and in bold, and the direct and indirect effects are in smaller font sizes
10.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Tian, W., Fang, Z., Wang, X. et al. A multi-dimensional analysis of usage counts, Mendeley readership, and citations for journal and conference papers. Scientometrics 129, 985–1013 (2024). https://doi.org/10.1007/s11192-023-04909-w
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-023-04909-w