Advertisement

Scientometrics

, Volume 103, Issue 3, pp 1123–1144 | Cite as

Alternative metrics in scientometrics: a meta-analysis of research into three altmetrics

  • Lutz Bornmann
Article

Abstract

Alternative metrics are currently one of the most popular research topics in scientometric research. This paper provides an overview of research into three of the most important altmetrics: microblogging (Twitter), online reference managers (Mendeley and CiteULike) and blogging. The literature is discussed in relation to the possible use of altmetrics in research evaluation. Since the research was particularly interested in the correlation between altmetrics counts and citation counts, this overview focuses particularly on this correlation. For each altmetric, a meta-analysis is calculated for its correlation with traditional citation counts. As the results of the meta-analyses show, the correlation with traditional citations for micro-blogging counts is negligible (pooled r = 0.003), for blog counts it is small (pooled r = 0.12) and for bookmark counts from online reference managers, medium to large (CiteULike pooled r = 0.23; Mendeley pooled r = 0.51).

Keywords

Altmetrics Twitter Microblogging Online reference managers Mendeley Blogging Meta-analysis 

Notes

Acknowledgments

I would like to thank Hadas Shema for discussing the concept of this paper.

References

  1. Bar-Ilan, J. (2012a). JASIST 2001–2010. Bulletin of the American Society for Information Science and Technology, 38(6), 24–28. doi: 10.1002/bult.2012.1720380607.Google Scholar
  2. Bar-Ilan, J. (2012b). JASIST@mendeley. Paper presented at the altmetrics12: An ACM Web Science Conference 2012 Workshop, Evanston, IL, USA.Google Scholar
  3. Bar-Ilan, J., Haustein, S., Peters, I., Priem, J., Shema, H., & Terliesner, J. (2012). Beyond citations: Scholars’ visibility on the social Web. In É. Archambault, Y. Gingras, & V. Larivière (Eds.), Proceedings of the 17th international conference on science and technology indicators (pp. 98–109). Montreal, QC: Science-Metrix and OST.Google Scholar
  4. Bar-Ilan, J., Shema, H., & Thelwall, M. (2014). Bibliographic references in Web 2.0. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: Harnessing multi-dimensional indicators of performance (pp. 307–325). Cambridge, MA: MIT Press.Google Scholar
  5. Bik, H. M., & Goldstein, M. C. (2013). An introduction to social media for scientists. PLoS Biology, 11(4), e1001535. doi: 10.1371/journal.pbio.1001535.CrossRefGoogle Scholar
  6. Bonetta, L. (2007). Scientists enter the blogosphere. Cell, 129(3), 443–445.CrossRefGoogle Scholar
  7. Börner, K., Sanyal, S., & Vespignani, A. (2007). Network science. Annual Review of Information Science and Technology, 41(1), 537–607. doi: 10.1002/aris.2007.1440410119.CrossRefGoogle Scholar
  8. Bornmann, L. (2012). Measuring the societal impact of research. EMBO Reports, 13(8), 673–676.CrossRefGoogle Scholar
  9. Bornmann, L. (2013). What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society of Information Science and Technology, 64(2), 217–233.CrossRefGoogle Scholar
  10. Bornmann, L. (2014). Validity of altmetrics data for measuring societal impact: A study using data from Altmetric and F1000Prime. Journal of Informetrics, 8(4), 935–950.CrossRefGoogle Scholar
  11. Bornmann, L., & Daniel, H.-D. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64(1), 45–80. doi: 10.1108/00220410810844150.CrossRefGoogle Scholar
  12. Bornmann, L., Leydesdorff, L., & Mutz, R. (2013). The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits. Journal of Informetrics, 7(1), 158–165.CrossRefGoogle Scholar
  13. Bornmann, L., Stefaner, M., de Moya Anegón, F., & Mutz, R. (2014). What is the effect of country-specific characteristics on the research performance of scientific institutions? Using multi-level statistical models to rank and map universities and research-focused institutions worldwide. Journal of Informetrics, 8(3), 581–593. doi: 10.1016/j.joi.2014.04.008.CrossRefGoogle Scholar
  14. Bradburn, M. J., Deeks, J. J., & Altman, D. G. (1998). Metan—An alternative meta-analysis command. Stata Technical Bulletin, 44, 4–15.Google Scholar
  15. Cheung, M. W. L. (2014). Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach. Psychological Methods, 19(2), 211–229. doi: 10.1037/A0032968.CrossRefGoogle Scholar
  16. Colledge, L. (2014). Snowball metrics recipe book. Amsterdam: Snowball Metrics program partners.Google Scholar
  17. Colson, V. (2011). Science blogs as competing channels for the dissemination of science news. Journalism, 12(7), 889–902. doi: 10.1177/1464884911412834.CrossRefGoogle Scholar
  18. Costas, R., Zahedi, Z., & Wouters, P. (2014). Do altmetrics correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Retrieved February 11, from http://arxiv.org/abs/1401.4321
  19. Darling, E. S., Shiffman, D., Côté, I. M., & Drew, J. A. (2013). The role of Twitter in the life cycle of a scientific publication. PeerJ PrePrints, 1, e16v11. doi: 10.7287/peerj.preprints.16v1.Google Scholar
  20. de Bellis, N. (2014). History and evolution of (biblio)metrics. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: Harnessing multi-dimensional indicators of performance (pp. 23–44). Cambridge, MA: MIT Press.Google Scholar
  21. Dinsmore, A., Allen, L., & Dolby, K. (2014). Alternative perspectives on impact: The potential of ALMs and altmetrics to inform funders about research impact. PLoS Biology, 12(11), e1002003. doi: 10.1371/journal.pbio.1002003.CrossRefGoogle Scholar
  22. Duggan, M., & Smith, A. (2014). Social Media Update 2013. Retrieved March 28, from http://pewinternet.org/Reports/2013/Social-Media-Update.aspx
  23. Eagly, A. H. (2005). Refereeing literature review submissions to journals. In R. J. Sternberg (Ed.), Reviewing scientific works in psychology. Washington, DC: American Psychological Association (APA).Google Scholar
  24. Eysenbach, G. (2011). Can tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact. J Med Internet Res, 13(4), e123.CrossRefGoogle Scholar
  25. Fausto, S., Machado, F. A., Bento, L. F. J., Iamarino, A., Nahas, T. R., & Munger, D. S. (2012). Research blogging: Indexing and registering the change in Science 2.0. PLoS ONE, 7(12), e50109. doi: 10.1371/journal.pone.0050109.CrossRefGoogle Scholar
  26. Galloway, L. M., Pease, J. L., & Rauh, A. E. (2013). Introduction to altmetrics for science, technology, engineering, and mathematics (STEM) librarians. Science & Technology Libraries, 32(4), 335–345. doi: 10.1080/0194262X.2013.829762.CrossRefGoogle Scholar
  27. Glass, G. V. (1976). Primary, secondary, and meta-analysis. Educational Researcher, 5, 3–8.CrossRefGoogle Scholar
  28. Groth, P., & Gurney, T. (2010). Studying scientific discourse on the Web using bibliometrics: A chemistry blogging case study. Paper presented at the WebSci10: Extending the Frontiers of Society On-Line, Raleigh, NC, USA. http://wiki.few.vu.nl/sms/images/9/9c/Websci10-FINAL-29-4-2010f.pdf
  29. Gunn, W. (2013). Social signals reflect academic impact: What it means when a scholar adds a paper to mendeley. Information Standards Quarterly, 25(2), 33–39.CrossRefGoogle Scholar
  30. Hammarfelt, B. (2014). Using altmetrics for assessing research impact in the humanities. Scientometrics. doi: 10.1007/s11192-014-1261-3.Google Scholar
  31. Haustein, S. (2014). Readership metrics. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: Harnessing multi-dimensional indicators of performance (pp. 327–344). Cambridge, MA: MIT Press.Google Scholar
  32. Haustein, S., & Peters, I. (2012). Using social bookmarks and tags as alternative indicators of journal content description. firstmonday, 17(11).Google Scholar
  33. Haustein, S., Peters, I., Bar-Ilan, J., Priem, J., Shema, H., & Terliesner, J. (2014a). Coverage and adoption of altmetrics sources in the bibliometric community. Scientometrics. doi: 10.1007/s11192-013-1221-3.zbMATHGoogle Scholar
  34. Haustein, S., Peters, I., Sugimoto, C. R., Thelwall, M., & Larivière, V. (2014b). Tweeting biomedicine: An analysis of tweets and citations in the biomedical literature. Journal of the Association for Information Science and Technology, 65(4), 656–669. doi: 10.1002/asi.23101.CrossRefGoogle Scholar
  35. Haustein, S., & Siebenlist, T. (2011). Applying social bookmarking data to evaluate journal usage. Journal of Informetrics, 5(3), 446–457. doi: 10.1016/j.joi.2011.04.002.Google Scholar
  36. Henning, V. (2010). The top 10 journal articles published in 2009 by readership on Mendeley | Mendeley Blog. Retrieved June 27, 2014, from http://www.mendeley.com/blog/academic-features/the-top-10-journal-articles-published-in-2009-by-readership-on-mendeley/
  37. Holmberg, K., & Thelwall, M. (2014). Disciplinary differences in Twitter scholarly communication. Scientometrics. doi: 10.1007/s11192-014-1229-3.Google Scholar
  38. Kjellberg, S. (2010). I am a blogging researcher: Motivations for blogging in a scholarly context. First Monday, 15(8).Google Scholar
  39. Kovic, I., Lulic, I., & Brumini, G. (2008). Examining the medical blogosphere: An online survey of medical bloggers. J Med Internet Res, 10(3), e28.CrossRefGoogle Scholar
  40. Li, X., & Thelwall, M. (2012). F1000, Mendeley and traditional bibliometric indicators. In E. Archambault, Y. Gingras, & V. Lariviere (Eds.), The 17th international conference on science and technology indicators (pp. 541–551). Montreal: Repro-UQAM.Google Scholar
  41. Li, X., Thelwall, M., & Giustini, D. (2012). Validating online reference managers for scholarly impact measurement. Scientometrics, 91(2), 461–471. doi: 10.1007/s11192-011-0580-x.CrossRefGoogle Scholar
  42. Lin, J., & Fenner, M. (2013). The many faces of article-level metrics. Bulletin of the American Society for Information Science and Technology, 39(4), 27–30. doi: 10.1002/bult.2013.1720390409.CrossRefGoogle Scholar
  43. Liu, C. L., Xu, Y. Q., Wu, H., Chen, S. S., & Guo, J. J. (2013). Correlation and interaction visualization of altmetric indicators extracted from scholarly social network activities: Dimensions and structure. Journal of Medical Internet Research, 15(11), 17. doi: 10.2196/jmir.2707.CrossRefGoogle Scholar
  44. Luzón, M. J. (2013). Public communication of science in blogs: Recontextualizing scientific discourse for a diversified audience. Written Communication. doi: 10.1177/0741088313493610.zbMATHGoogle Scholar
  45. Mahrt, M., Weller, K., & Peters, I. (2012). Twitter in scholarly communication. In K. Weller, A. Bruns, J. Burgess, M. Mahrt, & C. Puschmann (Eds.), Twitter and society (pp. 399–410). New York, NY: Peter Lang.Google Scholar
  46. Marsh, H. W., Bornmann, L., Mutz, R., Daniel, H. D., & O’Mara, A. (2009). Gender effects in the peer reviews of grant proposals: A comprehensive meta-analysis comparing traditional and multilevel approaches. Review of Educational Research, 79(3), 1290–1326. doi: 10.3102/0034654309334143.CrossRefGoogle Scholar
  47. Matt, G. E., & Navarro, A. M. (1997). What meta-analyses have and have not taught us about psychotherapy effects: A review and future directions. Clinical Psychology Review, 17(1), 1–32.CrossRefGoogle Scholar
  48. Mewburn, I., & Thomson, P. (2013). Why do academics blog? An analysis of audiences, purposes and challenges. Studies in Higher Education, 38(8), 1105–1119. doi: 10.1080/03075079.2013.835624.CrossRefGoogle Scholar
  49. Mohammadi, E., & Thelwall, M. (2013). Assessing the Mendeley readership of social science and humanities research. In J. Gorraiz, E. Schiebel, C. Gumpenberger, & M. Ho (Eds.), Proceedings of ISSI 2013 Vienna: 14th International society of scientometrics and informetrics conference (pp. 200–214). Vienna, Austria: Austrian Institute of Technology GmbH.Google Scholar
  50. Mohammadi, E., & Thelwall, M. (2014). Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows. Journal of the Association for Information Science and Technology, n/a–n/a. doi: 10.1002/asi.23071.Google Scholar
  51. Neylon, C., Willmers, M., & King, T. (2014). Rethinking impact: Applying altmetrics to southern African research. Ottawa: International Development Research Centre.Google Scholar
  52. NISO Alternative Assessment Metrics Project. (2014). NISO Altmetrics Standards Project White Paper. Retrieved July 8, 2014, from http://www.niso.org/apps/group_public/document.php?document_id=13295&wg_abbrev=altmetrics
  53. Priem, J. (2014). Altmetrics. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: Harnessing multi-dimensional indicators of performance. Cambridge, MA: MIT Press.Google Scholar
  54. Priem, J., & Costello, K. L. (2010). How and why scholars cite on Twitter. Proceedings of the American Society for Information Science and Technology, 47(1), 1–4. doi: 10.1002/meet.14504701201.CrossRefGoogle Scholar
  55. Priem, J., Costello, K., & Dzuba, T. (2012, 2012/12/16). Prevalence and use of Twitter among scholars. Retrieved June 23, 2014, from http://figshare.com/articles/Prevalence_and_use_of_Twitter_among_scholars/104629
  56. Priem, J., & Hemminger, B. M. (2010). Scientometrics 2.0: Toward new metrics of scholarly impact on the social Web. First Monday, 15(7).Google Scholar
  57. Priem, J., Piwowar, H., & Hemminger, B. (2012). Altmetrics in the wild: Using social media to explore scholarly impact. Retrieved March 27, from http://arxiv.org/html/1203.4745
  58. Puschmann, C. (2014). (Micro)Blogging Science? Notes on potentials and constraints of new forms of scholarly communication. In S. Bartling & S. Friesike (Eds.), Opening science (pp. 89–106). Cham, Switzerland: Springer International Publishing.CrossRefGoogle Scholar
  59. Puschmann, C., & Mahrt, M. (2012). Scholarly blogging: A new form of publishing or science journalism 2.0? In A. Tokar, M. Beurskens, S. Keuneke, M. Mahrt, I. Peters, C. Puschmann, K. Weller, & T. van Treeck (Eds.), Science and the Internet (pp. 171–182). Düsseldorf University Press: Düsseldorf.Google Scholar
  60. Rodgers, E. P., & Barbrow, S. (2013). A look at altmetrics and its growing significance to research libraries. Ann Arbor, MI: The University of Michigan University Library.Google Scholar
  61. Schlögl, C., Gorraiz, J., Gumpenberger, C., Jack, K., & Kraker, P. (2013). Download vs. vitiation vs. readership data: The case of an information systems journal. In J. Gorraiz, E. Schiebel, C. Gumpenberger, M. Hörlesberger, & H. Moed (Eds.), Proceedings of the 14th international society of scientometrics and informetrics conference. Austria: Austrian Institute of Technology, Vienna.Google Scholar
  62. Schlögl, C., Gorraiz, J., Gumpenberger, C., Jack, K., & Kraker, P. (2014). A comparison of citations, downloads and readership data for an information systems journal. Research Trends (37), 14–18.Google Scholar
  63. Schubert, A., & Braun, T. (1986). Relative indicators and relational charts for comparative assessment of publication output and citation impact. Scientometrics, 9(5–6), 281–291.CrossRefGoogle Scholar
  64. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin Company.Google Scholar
  65. Shema, H. (2014). Scholarly blogs are a promising altmetric source. Research Trends(37), 11-13.Google Scholar
  66. Shema, H., Bar-Ilan, J., & Thelwall, M. (2012a). Research blogs and the discussion of scholarly information. PLoS ONE, 7(5), e35869. doi: 10.1371/journal.pone.0035869.CrossRefGoogle Scholar
  67. Shema, H., Bar-Ilan, J., & Thelwall, M. (2012b). Self-citation of bloggers in the science blogosphere. In A. Tokar, M. Beurskens, S. Keuneke, M. Mahrt, I. Peters, C. Puschmann, K. Weller, & T. van Treeck (Eds.), Science and the Internet (pp. 183–192). Düsseldorf: Düsseldorf University Press.Google Scholar
  68. Shema, H., Bar-Ilan, J., & Thelwall, M. (2014). Do blog citations correlate with a higher number of future citations? Research blogs as a potential source for alternative metrics. Journal of the Association for Information Science and Technology, 65(5), 1018–1027. doi: 10.1002/asi.23037.CrossRefGoogle Scholar
  69. Shema, H., Bar-Ilan, J., & Thelwall, M. (in press). How is research blogged? A content analysis approach. Journal of the Association for Information Science and Technology.Google Scholar
  70. Shuai, X., Pepe, A., & Bollen, J. (2012). How the scientific community reacts to newly submitted preprints: Article downloads, Twitter mentions, and citations. Plos One, 7(11). doi: ARTN e47523 doi: 10.1371/journal.pone.0047523.
  71. StataCorp. (2013). Stata statistical software: Release 13. College Station, TX: Stata Corporation.Google Scholar
  72. Sud, P., & Thelwall, M. (in press). Not all international collaboration is beneficial: The Mendeley readership and citation impact of biochemical research collaboration. Journal of the Association for Information Science and Technology.Google Scholar
  73. Taylor, M. (2013). Towards a common model of citation: Some thoughts on merging altmetrics and bibliometrics. Research Trends (35), 19–22.Google Scholar
  74. Thelwall, M. (2014, January 2). Five recommendations for using alternative metrics in the future UK Research Excellence Framework. Retrieved from http://blogs.lse.ac.uk/impactofsocialsciences/2014/10/23/alternative-metrics-future-uk-research-excellence-framework-thelwall/
  75. Thelwall, M., Haustein, S., Lariviere, V., & Sugimoto, C. R. (2013). Do altmetrics work? Twitter and ten other social web services. PLoS ONE,. doi: 10.1371/journal.pone.0064841.Google Scholar
  76. Thelwall, M., & Maflahi, N. (in press). Are scholarly articles disproportionately read in their own country? An analysis of Mendeley readers. Journal of the Association for Information Science and Technology.Google Scholar
  77. Thorsen, E. (2013). Blogging on the ice: Connecting audiences with climate-change sciences. International Journal of Media & Cultural Politics, 9(1), 87–101. doi: 10.1386/macp.9.1.87_1.CrossRefGoogle Scholar
  78. Torres-Salinas, D., Cabezas-Clavijo, A., & Jimenez-Contreras, E. (2013). Altmetrics: New indicators for scientific communication in Web 2.0. Comunicar, 41, 53–60.CrossRefGoogle Scholar
  79. Tramer, M. R., Reynolds, D. J. M., Moore, R. A., & McQuay, H. J. (1997). Impact of covert duplicate publication on meta-analysis: A case study. British Medical Journal, 315(7109), 635–640.CrossRefGoogle Scholar
  80. Wainer, J., & Vieira, P. (2013). Correlations between bibliometrics and peer evaluation for all disciplines: The evaluation of Brazilian scientists. Scientometrics, 96(2), 395–410. doi: 10.1007/s11192-013-0969-9.CrossRefGoogle Scholar
  81. Weller, K., Dröge, E., & Puschmann, C. (2011). Citation analysis in Twitter: Approaches for defining and measuring information flows within Tweets during scientific conferences. In M. Rowe, M. Stankovic, A.-S. Dadzie, & M. Hardey (Eds.), Making Sense of Microposts (MSM2011) (pp. 1–12). Heraklion: CEUR Workshop Proceedings.Google Scholar
  82. Weller, K., & Peters, I. (2012). Citations in Web 2.0. In A. Tokar, M. Beurskens, S. Keuneke, M. Mahrt, I. Peters, C. Puschmann, T. van Treeck, & K. Weller (Eds.), Science and the Internet (pp. 209–222). Düsseldorf: Düsseldorf University Press.Google Scholar
  83. Weller, K., & Puschmann, C. (2011, June 14–17 2011). Twitter for Scientific Communication: How Can Citations/References be Identified and Measured? Paper presented at the ACM WebSci’11, Koblenz, Germany.Google Scholar
  84. White, H. D. (2005). On extending informetrics: An opinion paper. In P. Ingwersen & B. Larsen (Eds.), Proceedings of the 10th International conference of the international society for scientometrics and informetrics (Vol. 2, pp. 442–449). Stockholm, Sweden: Karolinska University Press.Google Scholar
  85. Wolinsky, H. (2011). More than a blog. EMBO Reports, 12(11), 1102–1105.CrossRefGoogle Scholar
  86. Wouters, P. (2014). The citation: From culture to infrastructure. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: Harnessing multi-dimensional indicators of performance (pp. 47–66). Cambridge, MA: MIT Press.Google Scholar
  87. Yan, K. K., & Gerstein, M. (2011). The spread of scientific information: Insights from the web usage statistics in PLoS article-level metrics. PloS One, 6(5). doi: ARTN e19917 doi: 10.1371/journal.pone.0019917
  88. Zahedi, Z., Costas, R., & Wouters, P. (2014). How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications. Scientometrics. doi: 10.1007/s11192-014-1264-0.Google Scholar
  89. Zubiaga, A., Spina, D., Martínez, R., & Fresno, V. (2014). Real-time classification of twitter trends. Journal of the Association for Information Science and Technology, n/a–n/a. doi: 10.1002/asi.23186.Google Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2015

Authors and Affiliations

  1. 1.Division for Science and Innovation StudiesAdministrative Headquarters of the Max Planck SocietyMunichGermany

Personalised recommendations