, Volume 108, Issue 1, pp 413–423 | Cite as

Grand challenges in altmetrics: heterogeneity, data quality and dependencies

  • Stefanie HausteinEmail author


With increasing uptake among researchers, social media are finding their way into scholarly communication and, under the umbrella term altmetrics, are starting to be utilized in research evaluation. Fueled by technological possibilities and an increasing demand to demonstrate impact beyond the scientific community, altmetrics have received great attention as potential democratizers of the scientific reward system and indicators of societal impact. This paper focuses on the current challenges for altmetrics. Heterogeneity, data quality and particular dependencies are identified as the three major issues and discussed in detail with an emphasis on past developments in bibliometrics. The heterogeneity of altmetrics reflects the diversity of the acts and online events, most of which take place on social media platforms. This heterogeneity has made it difficult to establish a common definition or conceptual framework. Data quality issues become apparent in the lack of accuracy, consistency and replicability of various altmetrics, which is largely affected by the dynamic nature of social media events. Furthermore altmetrics are shaped by technical possibilities and are particularly dependent on the availability of APIs and DOIs, strongly dependent on data providers and aggregators, and potentially influenced by the technical affordances of underlying platforms.


Big data Data integration Research and innovation policy Data quality Comparability Standardization Concordance tables Modularization Interoperability Research assessment 



The author acknowledges funding from the Alfred P. Sloan Foundation Grant # 2014-3-25. She would also like to thank Vincent Larivière for stimulating discussions and helpful suggestions on the manuscript, as well as Sam Work for proofreading.


  1. Alperin, J. P. (2015). Geographic variation in social media metrics: An analysis of Latin American journal articles. Aslib Journal of Information Management, 67(3), 289–304. doi: 10.1108/AJIM-12-2014-0176.MathSciNetCrossRefGoogle Scholar
  2. Bar-Ilan, J. (2014). JASIST@ Mendeley revisited. In altmetrics14: expanding impacts and metrics, workshop at web science conference 2014. Retrieved from
  3. Björneborn, L., & Ingwersen, P. (2004). Toward a basic framework for webometrics. Journal of the American Society for Information Science and Technology, 55(14), 1216–1227. doi: 10.1002/asi.20077.CrossRefGoogle Scholar
  4. Bornmann, L., & Leydesdorff, L. (2013). The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000. Journal of Informetrics, 7(2), 286–291. doi: 10.1016/j.joi.2012.12.003.CrossRefGoogle Scholar
  5. Costas, R., Zahedi, Z., & Wouters, P. (2015). Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Association for Information Science and Technology, 66(10), 2003–2019. doi: 10.1002/asi.23309.CrossRefGoogle Scholar
  6. Cronin, B. (2016). The Incessant chattering of texts. In C. R. Sugimoto (Ed.), Theories of informetrics and scholarly communication. A Festschrift in Honor of Blaise Cronin (pp. 13–19). Berlin: De Gruyter.Google Scholar
  7. Cronin, B., Snyder, H. W., Rosenbaum, H., Martinson, A., & Callahan, E. (1998). Invoked on the web. Journal of the American Society for Information Science, 49(14), 1319–1328. doi: 10.1002/(SICI)1097-4571(1998)49:14<1319:AID-ASI9>3.0.CO;2-W.CrossRefGoogle Scholar
  8. Dahler-Larsen, P. (2012). The evaluation society. Stanford, CA: Stanford Business Books, an imprint of Stanford University Press.Google Scholar
  9. Dinsmore, A., Allen, L., & Dolby, K. (2014). Alternative perspectives on impact: The potential of ALMs and altmetrics to inform funders about research impact. PLoS Biology, 12(11), e1002003.CrossRefGoogle Scholar
  10. Garfield, E. (1955). Citation indexes for science. A new dimension in documentation through association of ideas. Science, 122, 108–111.CrossRefGoogle Scholar
  11. Gilbert, G. N. (1977). Referencing as persuasion. Social Studies of Science, 7(1), 113–122.CrossRefGoogle Scholar
  12. Glänzel, W., & Gorraiz, J. (2015). Usage metrics versus altmetrics: Confusing terminology? Scientometrics, 102(3), 2161–2164. doi: 10.1007/s11192-014-1472-7.CrossRefGoogle Scholar
  13. Gross, P. L. K., & Gross, E. M. (1927). College libraries and chemical education. Science, 66(1713), 385–389.CrossRefGoogle Scholar
  14. Haustein, S., Bowman, T. D., & Costas, R. (2015a). When is an article actually published? An analysis of online availability, publication, and indexation dates. In Proceedings of the 15th international society of scientometrics and informetrics conference (pp. 1170–1179). Istanbul, Turkey.Google Scholar
  15. Haustein, S., Bowman, T. D., & Costas, R. (2016a). Interpreting “altmetrics”: viewing acts on social media through the lens of citation and social theories. In C. R. Sugimoto (Ed.), Theories of informetrics and scholarly communication. A Festschrift in Honor of Blaise Cronin (pp. 372–405). Berlin: De Gruyter. Retrieved from
  16. Haustein, S., Bowman, T. D., Holmberg, K., Tsou, A., Sugimoto, C. R., & Larivière, V. (2016b). Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter. Journal of the Association for Information Science and Technology, 67(1), 232–238. doi: 10.1002/asi.23456.CrossRefGoogle Scholar
  17. Haustein, S., Costas, R., & Larivière, V. (2015b). Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(3), e0120495. doi: 10.1371/journal.pone.0120495.CrossRefGoogle Scholar
  18. Haustein, S., & Larivière, V. (2015). The use of bibliometrics for assessing research: possibilities, limitations and adverse effects. In I. M. Welpe, J. Wollersheim, S. Ringelhan, & M. Osterloh (Eds.), Incentives and performance: Governance of knowledge-intensive organizations (pp. 121–139). Springer International Publishing. Retrieved from
  19. Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (2014). Tweets vs. Mendeley readers: How do these two social media metrics differ? It—Information Technology, 56(5), 207–215. doi: 10.1515/itit-2014-1048.Google Scholar
  20. Haustein, S., Sugimoto, C., & Larivière, V. (2015c). Guest editorial: Social media in scholarly communication. Aslib Journal of Information Management,. doi: 10.1108/AJIM-03-2015-0047.Google Scholar
  21. Higher Education Funding Council for England. (2011). Decisions on assessing research impact. Research Excellent Framework (REF) 2014 (Research Excellence Framework No. 01.2011). Retrieved from
  22. Jobmann, A., Hoffmann, C. P., Künne, S., Peters, I., Schmitz, J., & Wollnik-Korn, G. (2014). Altmetrics for large, multidisciplinary research groups: Comparison of current tools. Bibliometrie-Praxis Und Forschung, 3. Retrieved from
  23. Lazarsfeld, P. F. (1993). On social research and its language. Chicago: University of Chicago Press.Google Scholar
  24. Li, X., Thelwall, M., & Giustini, D. (2012). Validating online reference managers for scholarly impact measurement. Scientometrics, 91(2), 461–471. doi: 10.1007/s11192-011-0580-x.CrossRefGoogle Scholar
  25. Lin, J., & Fenner, M. (2013). Altmetrics in evolution: Defining and redefining the ontology of article-level metrics. Information Standards Quarterly, 25(2), 20–26.CrossRefGoogle Scholar
  26. Mas-Bleda, A., Thelwall, M., Kousha, K., & Aguillo, I. F. (2014). Do highly cited researchers successfully use the social web? Scientometrics, 101(1), 337–356. doi: 10.1007/s11192-014-1345-0.CrossRefGoogle Scholar
  27. Moed, H. F. (2016). Altmetrics as traces of the computerization of the research process. In C. R. Sugimoto (Ed.), Theories of informetrics and scholarly communication. A Festschrift in Honor of Blaise Cronin (pp. 360–371). Berlin: De Gruyter.Google Scholar
  28. Moed, H. F., Burger, W. J. M., Frankfort, J. G., & Van Raan, A. F. J. (1985). The use of bibliometric data for the measurement of university research performance. Research Policy, 14(3), 131–149. doi: 10.1016/0048-7333(85)90012-5.CrossRefGoogle Scholar
  29. Piwowar, H. (2013). Value all research products. Nature, 493, 159. doi: 10.1038/493159a.Google Scholar
  30. Priem, J. (2014). Altmetrics. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: harnessing multidimensional indicators of performance (pp. 263–287). Cambridge, MA: MIT Press.Google Scholar
  31. Priem, J., & Hemminger, B. M. (2010). Scientometrics 2.0: Toward new metrics of scholarly impact on the social Web. First Monday, 15(7). Retrieved from
  32. Priem, J., Piwowar, H. A., & Hemminger, B. M. (2012). Altmetrics in the wild: Using social media to explore scholarly impact. arXiv Print, 1–17.Google Scholar
  33. Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2010, October 26). Altmetrics: A manifesto. Retrieved from
  34. Rousseau, R., & Ye, F. Y. (2013). A multi-metric approach for research evaluation. Chinese Science Bulletin, 58(26), 3288–3290. doi: 10.1007/s11434-013-5939-3.CrossRefGoogle Scholar
  35. Rowlands, I., Nicholas, D., Russell, B., Canty, N., & Watkinson, A. (2011). Social media use in the research workflow. Learned Publishing, 24(3), 183–195. doi: 10.1087/20110306.CrossRefGoogle Scholar
  36. Sugimoto, C. R. (Ed.). (2016). Theories of informetrics and scholarly communication. Berlin: De Gruyter.Google Scholar
  37. Taylor, M. (2013). Towards a common model of citation: Some thoughts on merging altmetrics and bibliometrics. Research Trends, 35, 1–6.Google Scholar
  38. Thelwall, M., & Wilson, P. (2015). Mendeley readership altmetrics for medical articles: An analysis of 45 fields. Journal of the Association for Information Science and Technology,. doi: 10.1002/asi.23501.Google Scholar
  39. Van Noorden, R. (2014). Online collaboration: Scientists and the social network. Nature, 512(7513), 126–129. doi: 10.1038/512126a.CrossRefGoogle Scholar
  40. Waltman, L., & Costas, R. (2014). F1000 recommendations as a potential new data source for research evaluation: A comparison with citations. Journal of the Association for Information Science and Technology, 65(3), 433–445. doi: 10.1002/asi.23040.CrossRefGoogle Scholar
  41. Wilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S., Hill, S., et al. (2015). The metric tide: report of the independent review of the role of metrics in research assessment and management.,. doi: 10.13140/RG.2.1.4929.1363.Google Scholar
  42. Zahedi, Z., Bowman, T. D., & Haustein, S. (2014a). Exploring data quality and retrieval strategies for Mendeley reader counts. Presented at the SIG/MET workshop, ASIS&T 2014 annual meeting, Seattle. Retrieved from
  43. Zahedi, Z., Costas, R., & Wouters, P. (2014b). Assessing the impact of publications saved by Mendeley users: Is there any different pattern among users? In IATUL conference, Espoo, Finland, June 2–5 2014. Retrieved from
  44. Zahedi, Z., Fenner, M., & Costas, R. (2014c). How consistent are altmetrics providers? Study of 1000 PLOS ONE publications using the PLOS ALM, Mendeley and Altmetric. com APIs. In altmetrics 14. Workshop at the web science conference, Bloomington, USA. Retrieved from
  45. Zahedi, Z., Fenner, M., & Costas, R. (2015). Consistency among altmetrics data provider/aggregators: What are the challenges? In altmetrics15: 5 years in, what do we know? Amsterdam, The Netherlands. Retrieved from

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2016

Authors and Affiliations

  1. 1.EBSIUniversité de MontréalMontrealCanada

Personalised recommendations