Abstract
Since the Altmetric Manifesto was published in 2010, a large number of studies have quantitatively examined altmetric-citation associations to assess if and how altmetrics differ from citations. These efforts resulted in a diverse array of observations that varies within and between indicators. These studies also differ in the measurement environment, potentially introducing confounding variables. We sought here to clarify this diversity with a meta-analysis and examine if any of several factors moderated the altmetric-citation relationship. We identified relevant literature examining altmetric-citation correlations in the Web of Science and extracted from each study the correlation coefficient, sample size, and potential moderator variables, such as publication year, field, and citation source. We collated 914 correlation coefficients from 111 studies examining 13 altmetric indicators. We fitted multi-level random-effects meta-analysis models for each altmetric indicator individually and examined moderators for Mendeley, Twitter, Altmetric Attention Score, and usage metrics. Mendeley, usage metrics, ResearchGate, Twitter, and peer ratings showed the strongest association with citations, while Facebook, Wikipedia, Google + , blogs, news, and Reddit were only weakly associated with citations. No variable consistently moderated the altmetric-citation relationship, suggesting that, when associations were observed, they were structural and persistent. Altmetrics that did not demonstrate an association evidently differ from citations in the impact construct measured. Hence, our study highlights the diversity among altmetrics channels, as the characteristics of each channel partly define the potential mentions of research in these channels, suggesting a nuanced application of the diverse channels.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Amath, A., Ambacher, K., Leddy, J. J., Wood, T. J., & Ramnana, C. J. (2017). Comparing alternative and traditional dissemination metrics in medical education. Medical Education, 51, 935–941. https://doi.org/10.1111/medu.13359
Assink, M., & Wibbelink, C. (2016). Fitting three-level meta-analytic models in R: A step-by-step tutorial. The Quantitative Methods for Psychology, 12, 154–174. https://doi.org/10.20982/tqmp.12.3.p154
Bornmann, L. (2014). Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of Informetrics, 8, 895–903. https://doi.org/10.1016/j.joi.2014.09.005
Bornmann, L. (2015). Alternative metrics in scientometrics: A meta-analysis of research into three altmetrics. Scientometrics, 103, 1123–1144. https://doi.org/10.1007/s11192-015-1565-y
Bornmann, L., & Haunschild, R. (2018). Do altmetrics correlate with the quality of papers? A large-scale empirical study based on F1000 Prime data. PLoS ONE, 13(5), e0197133. https://doi.org/10.1371/journal.pone.0197133
Buttliere, B., & Buder, J. (2017). Personalizing papers using altmetrics: Comparing paper ‘quality’ or ‘impact’ to person ‘intelligence’ or ‘personality.’ Scientometrics, 111, 219–239. https://doi.org/10.1007/s11192-017-2246-9
Cho, J. (2021). Altmetrics analysis of highly cited academic papers in the field of library and information science. Scientometrics, 126, 7623–7635. https://doi.org/10.1007/s11192-021-04084-w
Costas, R., Zahedi, Z., & Wouters, P. (2015). Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Association for Information Science and Technology, 66, 2003–2019. https://doi.org/10.1002/asi.23309
Erdt, M., Nagarajan, A., Sin, S.-C.J., & Theng, Y.-L. (2016). Altmetrics: An analysis of the state-of-the-art in measuring research impact on social media. Scientometrics, 109, 1117–1166. https://doi.org/10.1007/s11192-016-2077-0
Eysenbach, G. (2011). Can tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact. Journal of Medical Internet Research, 13, e123. https://doi.org/10.2196/jmir.2012
Gilpin, A. R. (1993). Table for conversion of Kendall’s tau to Spearman’s rho within the context of measures of magnitude of effect for meta-analysis. Educational and Psychological Measurement, 53, 87–92. https://doi.org/10.1177/0013164493053001007
Gorraiz, J., Blahous, B., & Wieland, M. (2018). Monitoring the broader impact of the journal publication output on country level: A case study for Austria. In Erdt M., Sesagiri Raamkumar A., Rasmussen E., & YL. Theng (Eds.), Altmetrics for research outputs measurement and scholarly information management (pp. 39–62). Springer. https://doi.org/10.1007/978-981-13-1053-9_4
Harrer, M., Cuijpers, P., Furukawa, T., & Ebert, D. (2021). Doing meta-analysis with R: A hands-on guide (1st ed.). Chapman & Hall/CRC Press.
Haustein, S., Peters, I., Sugimoto, C., Thelwall, M., & Larivière, V. (2014). Tweeting biomedicine: An analysis of tweets and citations in the biomedical literature. Journal of the Association for Information Science and Technology, 65, 656–669. https://doi.org/10.1002/asi.23101
Kolahi, J., Khazaei, S., Iranmanesh, P., Kim, J., Bang, H., & Khademi, A. (2021). Meta-analysis of correlations between altmetric attention score and citations in health sciences. BioMed Research International. https://doi.org/10.1155/2021/6680764
Malecki, A. (2015). PubMed and arXiv vs. Gold Open Access: Citation, Mendeley, and Twitter uptake of academic articles of Iran (Salah A. A., Tonta Y., Akdag Salah A. A., C. Sugimoto, & U. Al, Eds.; pp. 46–58). Bogaziçi University Printhouse.
J. Priem, D. Taraborelli, P. Groth, C. Neylon (2010), Altmetrics: A manifesto, 26 October 2010. http://altmetrics.org/manifesto
Rodgers, M., & Pustejovsky, J. E. (2021). Evaluating meta-analytic methods to detect selective reporting in the presence of dependent effect sizes. Psychological Methods, 26, 141–160. https://doi.org/10.1037/met0000300
Shen, H., Xie, J., Li, J., & Cheng, Y. (2021). The correlation between scientific collaboration and citation count at the paper level: A meta-analysis. Scientometrics, 126, 3443–3470. https://doi.org/10.1007/s11192-021-03888-0
Stahlschmidt, S., & Stephen, D. (2022). From indexation policies through citation networks to normalized citation impacts: Web of Science, Scopus, and Dimensions as varying resonance chambers. Scientometrics, 127, 2413–2431. https://doi.org/10.1007/s11192-022-04309-6
Sugimoto, C., Work, S., Larivière, V., & Haustein, S. (2017). Scholarly use of social media and altmetrics: A review of the literature. Journal of the Association for Information Science and Technology, 68, 2037–2062. https://doi.org/10.1002/asi.23833
Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–48. https://doi.org/10.18637/jss.v036.i03
Wang, J. (2013). Citation time window choice for research impact evaluation. Scientometrics, 94(3), 851–872. https://doi.org/10.1007/s11192-012-0775-9
Wooldridge, J., & King, M. B. (2019). Altmetric scores: An early indicator of research impact. Journal of the Association for Information Science and Technology, 70(3), 271–282. https://doi.org/10.1002/asi.24122
Xia, F., Su, X., Wang, W., Zhang, C., Ning, Z., & Lee, I. (2016). Bibliographic analysis of nature based on Twitter and Facebook altmetrics data. PLoS ONE, 11, e0165997. https://doi.org/10.1371/journal.pone.0165997
Acknowledgements
A shortened version of this study was presented under the title “Confounding cross-correlation: A meta-analysis of the association between citations and altmetrics” at the 18th International Society of Scientometrics and Informetrics Conference (ISSI 2023) in Bloomington, USA, 2-5 July 2023.
Funding
This study was funded by the Federal Ministry of Education and Research of Germany (BMBF) under grant number 16PU17024.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing interests
The authors have no relevant financial or non-financial interests to disclose.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Stephen, D., Stahlschmidt, S. Contrasting cross-correlation: Meta-analyses of the associations between citations and 13 altmetrics, incorporating moderating variables. Scientometrics (2024). https://doi.org/10.1007/s11192-024-05006-2
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11192-024-05006-2