Skip to main content
SpringerLink
Log in

Exploring the relationships between altmetric counts and citations of papers in different academic fields based on co-occurrence analysis

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Altmetrics is an emerging method for observing academic communication. Many researchers have explored the relationships between altmetric counts and citations through correlation analysis, but there is no consistent result. The varied results may come from the divergence of data sources, disciplinary differences, and the characteristics of data distribution. To reduce the influence of the interfering factors above, the Co-occurrence analysis was proposed in this study as the method to explore the relationships between citations and altmetric counts. We observe the overlap between the highly-cited papers and the high altmetric count papers, alongside with the coverage of each collection from various altmetric sources in different academic fields. The results show that Mendeley has the highest correlation with citations among all the altmetric sources in the five academic fields, and might be the only one having the opportunity to be an indicator for academic evaluation. The other altmetric counts from different sources do not show strong relationships with citations in general.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Adam, D. (2002). The counting house. Nature, 415(6873), 726–729. https://doi.org/10.1038/415726a

    Article  Google Scholar 

  • Alhoori, H., & Furuta, R. (2014). Do altmetrics follow the crowd or does the crowd follow altmetrics?. In IEEE/ACM Joint Conference on Digital Libraries (pp. 375–378). IEEE. https://doi.org/10.1109/JCDL.2014.6970193.

  • Borgman, C. L., & Furner, J. (2002). Scholarly communication and bibliometrics. Annual Review of Information Science and Technology, 36(1), 2–72.

    Article  Google Scholar 

  • Bornmann, L. (2014). Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of Informetrics, 8(4), 895–903.

    Article  Google Scholar 

  • Bornmann, L., Haunschild, R., & Adams, J. (2019). Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF). Journal of Informetrics, 13(1), 325–340. https://doi.org/10.1016/j.joi.2019.01.008

    Article  Google Scholar 

  • Bowman, T. D. (2015). Investigating the use of affordances and framing techniques by scholars to manage personal and professional impressions on Twitter [Doctoral dissertation, Indiana University]. tdbowman.com. Retrieved October 11, 2019, from http://www.tdbowman.com/pdf/2015_07_TDBowman_Dissertation.pdf.

  • Bradshaw, C. J. A., & Brook, B. W. (2016). How to rank journals. PLoS ONE. https://doi.org/10.1371/journal.pone.0149852

    Article  Google Scholar 

  • Carpenter, J. (2012). Researchers of Tomorrow: The research behaviour of Generation Y doctoral students. Information Services and Use, 32(1–2), 3–17. https://doi.org/10.3233/ISU-2012-0637

    Article  Google Scholar 

  • Costas, R., Zahedi, Z., & Wouters, P. (2014). Do Altmetrics correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Association for Information Science and Technology, 66(10), 2003–2019.

    Article  Google Scholar 

  • Costas, R., Zahedi, Z., & Wouters, P. (2015). The thematic orientation of publications mentioned on social media: Large-scale disciplinary comparison of social media metrics with citations. Aslib Journal of Information Management, 67(3), 260–288.

    Article  Google Scholar 

  • Cronin, B., & Snyder, H. W. (1998). Invoked on the web. Journal of the American Society for Information Science, 49(14), 1319–1328. https://doi.org/10.1108/AJIM-12-2014-0173

    Article  Google Scholar 

  • Crotty, D. (2014). Altmetrics: Finding meaningful needles in the data haystack. Serials Review, 40(3), 141–146. https://doi.org/10.1080/00987913.2014.947839

    Article  Google Scholar 

  • de Oliveira Silva, D., Taborda, B., Pazzinatto, M. F., Ardern, C. L., & Barton, C. J. (2021). The altmetric score has a stronger relationship with article citations than journal impact factor and open access status: A cross-sectional analysis of 4022 sport sciences articles. Journal of Orthopaedic & Sports Physical Therapy, 51(11), 536–541.

    Article  Google Scholar 

  • Didegah, F., Bowman, T. D., & Holmberg, K. (2018). On the differences between citations and altmetrics : An investigation of factors driving altmetrics versus citations for finnish articles. Journal of the Association for Information Science and Technology, 69(6), 832–843. https://doi.org/10.1002/asi.23934

    Article  Google Scholar 

  • Elmore, S. A. (2018). The altmetric attention score : What does it mean and why should i care ? Toxicologic Pathology, 46(3), 252–255. https://doi.org/10.1177/0192623318758294

    Article  MathSciNet  Google Scholar 

  • Eysenbach, G. (2011). Can tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact. Journal of Medical Internet Research. https://doi.org/10.2196/jmir.2012

    Article  Google Scholar 

  • Fenner, M. (2013). What can article-level metrics do for you? PLoS Biology, 11(10), e1001687.

    Article  Google Scholar 

  • Galligan, F., & Dyas-Correia, S. (2013). Altmetrics rethinking the way we measure. Serials Review, 39(1), 56–61.

    Article  Google Scholar 

  • García-Villar, C. (2021). A critical review on altmetrics: Can we measure the social impact factor? Insights into Imaging, 12(1), 1–10.

    Article  Google Scholar 

  • Garfield, E. (1979). Is citation analysis a legitimate evaluation tool? Scientometrics, 1(4), 359–375. https://doi.org/10.1007/BF02019306

    Article  Google Scholar 

  • Gauch, S., & Blümel, C. (2018). Quantitative User Valuation Studies as means of scrutinization–The case of Altmetrics [Paper presentation]. The 23rd International Conference on Science and Technology Indicators (STI 2018), Leiden, The Netherlands. Retrieved August 27, 2019, from https://scholarlypublications.universiteitleiden.nl/access/item%3A2722383/view.

  • Hammarfelt, B. (2014). Using altmetrics for assessing research impact in the humanities. Scientometrics, 101(2), 1419–1430. https://doi.org/10.1007/s11192-014-1261-3

    Article  Google Scholar 

  • Harley, D., Acord, S. K., Earl-Novell, S., Lawrence, S., & King, C. J. (2010). Assessing the future landscape of scholarly communication An exploration of faculty values and needs in seven disciplines. Berkeley: Univ Of California Press.

  • Hassan, S. U., Iqbal, S., Aljohani, N. R., Alelyani, S., & Zuccala, A. (2020). Introducing the’alt-index’for measuring the social visibility of scientific research. Scientometrics, 123(3), 1407–1419.

    Article  Google Scholar 

  • Haustein, S. (2016). Grand challenges in altmetrics: Heterogeneity, data quality and dependencies. Scientometrics, 108(1), 413–423. https://doi.org/10.1007/s11192-016-1910-9

    Article  Google Scholar 

  • Haustein, S., Bowman, T. D., Holmberg, K., Tsou, A., Sugimoto, C. R., & Larivière, V. (2016). Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter. Journal of the Association for Information Science and Technology, 67(1), 232–238.

    Article  Google Scholar 

  • Haustein, S., Costas, R., & Larivière, V. (2015). Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(3), 1–21. https://doi.org/10.1371/journal.pone.0120495

    Article  Google Scholar 

  • Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (2014a). Tweets vs. Mendeley readers: How do these two social media metrics differ? IT-Information Technology, 56(5), 207–215.

    Article  Google Scholar 

  • Haustein, S., Peters, I., Bar-Ilan, J., Priem, J., Shema, H., & Terliesner, J. (2014b). Coverage and adoption of altmetrics sources in the bibliometric community. Scientometrics, 101(2), 1145–1163. https://doi.org/10.1007/s11192-013-1221-3

    Article  Google Scholar 

  • Haustein, S., Peters, I., Sugimoto, C. R., Thelwall, M., & Larivière, V. (2014c). Tweeting biomedicine: An analysis of tweets and citations in the biomedical literature. Journal of the Association for Information Science and Technology, 65(4), 656–669.

    Article  Google Scholar 

  • Herzog, C., Sorensen, A., & Taylor, M. (2016) Forward-looking analysis based on grants data and machine learning based research classification as an analytical tool [Paper presentation]. OECD Blue Sky 2016 Forum, Ghent, Belgium. Retrieved December 5, 2019, from https://www.oecd.org/sti/093%20-%20OECDForward-lookinganalysisbasedongrantsdataandmachinelearningbasedresearchclassificationsasananalyticaltool%20(1).pdf.

  • Kaplan, A. M., & Haenlein, M. (2010). Users of the world, unite! The challenges and opportunities of Social Media. Business Horizons, 53(1), 59–68.

    Article  Google Scholar 

  • Kietzmann, J. H., Hermkens, K., McCarthy, I. P., & Silvestre, B. S. (2011). Social media? Get serious! Understanding the functional building blocks of social media. Business Horizons, 54(3), 241–251. https://doi.org/10.1016/j.bushor.2011.01.005

    Article  Google Scholar 

  • Lin, J., & Fenner, M. (2013). Altmetrics in evolution: Defining and redefining the ontology of article-level metrics. Information Standards Quarterly, 25(2), 20–26.

    Article  Google Scholar 

  • Liu, J., & Adie, E. (2013). Five challenges in altmetrics: A toolmaker’s perspective. Bulletin of the American Society for Information Science and Technology, 39(4), 31–34.

    Article  Google Scholar 

  • MacRoberts, M. H., & MacRoberts, B. R. (1996). Problems of citation analysis. Scientometrics, 36(3), 435–444. https://doi.org/10.1007/BF02129604

    Article  Google Scholar 

  • Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. University of Chicago press.

    Google Scholar 

  • Ortega, J. L. (2018). Reliability and accuracy of altmetric providers: A comparison among Altmetric.com, PlumX and and Crossref Event Data. Scientometrics, 116(3), 2123–2138. https://doi.org/10.1007/s11192-018-2838-z

    Article  Google Scholar 

  • Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2010). Alt-metrics: A Manifesto. Altmetrics. http://altmetrics.org/manifesto/.

  • Priem, J., Piwowar, H., & Hemminger, B. (2011). Altmetrics in the wild: An exploratory study of impact metrics based on social media [Paper presentation]. Metrics 2011: Symposium on Informetric and Scientometric Research, New Orleans, USA. Retrieved May 27, 2019, from http://jasonpriem.com/self-archived/PLoS-altmetrics-sigmetrics11-abstract.pdf.

  • Repiso, R., Castillo-Esparcia, A., & Torres-Salinas, D. (2019). Altmetrics, alternative indicators for Web of Science Communication studies journals. Scientometrics, 119(2), 941–958. https://doi.org/10.1007/s11192-019-03070-7

    Article  Google Scholar 

  • Robinson-Garcia, N., van Leeuwen, T. N., & Ràfols, I. (2018). Using altmetrics for contextualised mapping of societal impact: From hits to networks. Science and Public Policy, 45(6), 815–826. https://doi.org/10.1093/SCIPOL/SCY024

    Article  Google Scholar 

  • Small, H. (2004). On the shoulders of Robert Merton: Towards a normative theory of citation. Scientometrics, 60(1), 71–79.

    Article  Google Scholar 

  • Sugimoto, C. R., Work, S., Larivière, V., & Haustein, S. (2017). Scholarly use of social media and altmetrics: A review of the literature. Journal of the Association for Information Science and Technology, 68(9), 2037–2062.

    Article  Google Scholar 

  • Taylor, M. (2013). Towards a common model of citation: Some thoughts on merging altmetrics and bibliometrics. Research Trends, 35, 19–22.

    Google Scholar 

  • Thelwall, M. (2008). Bibliometrics to webometrics. Journal of Information Science, 34(4), 605–621.

    Article  Google Scholar 

  • Thelwall, M., Haustein, S., Larivière, V., & Sugimoto, C. R. (2013a). Do Altmetrics Work? Twitter and Ten Other Social Web Services. PLoS ONE, 8(5), 1–7. https://doi.org/10.1371/journal.pone.0064841

    Article  Google Scholar 

  • Thelwall, M., & Wilson, P. (2016). Mendeley readership altmetrics for medical articles: An analysis of 45 fields. Journal of the Association for Information Science and Technology, 67(8), 1962–1972. https://doi.org/10.1002/asi.23501

    Article  Google Scholar 

  • Xu, S. (2018). Issues in the Interpretation of “Altmetrics” Digital Traces: A Review. Frontiers in Research Metrics and Analytics, 3(October), 1–8. https://doi.org/10.3389/frma.2018.00029

    Article  Google Scholar 

  • Zahedi, Z., Costas, R., & Wouters, P. (2014). How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications. Scientometrics, 101(2), 1491–1513. https://doi.org/10.1007/s11192-014-1264-0

    Article  Google Scholar 

Download references

Acknowledgements

This work was financially supported by the Center for Research in Econometric Theory and Applications (Grant No. 111L900204) which is under the Featured Areas Research Center Program by Higher Education Sprout Project of Ministry of Education (MOE) in Taiwan, the Universities and Colleges Humanities and Social Sciences Benchmarking Project (Grant No. 111L9A002), and the Ministry of Science and Technology (MOST), Taiwan, with Grant No. 111-2634-F-002-018-.

Author information

Authors and Affiliations

  1. Institute of Nuclear Energy Research, Atomic Energy Council, Executive Yuan, Taoyuan, Taiwan

    Chieh Liu

  2. Department of Library and Information Science, National Taiwan University, Taipei, Taiwan

    Chieh Liu & Mu-Hsuan Huang

Authors
  1. Chieh Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mu-Hsuan Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mu-Hsuan Huang.

Ethics declarations

Conflict of interest

The authors have no conflict of interest to declare that are relevant to the content of this article.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, C., Huang, MH. Exploring the relationships between altmetric counts and citations of papers in different academic fields based on co-occurrence analysis. Scientometrics 127, 4939–4958 (2022). https://doi.org/10.1007/s11192-022-04456-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-022-04456-w

Keywords

Search

Navigation