Skip to main content

Evaluating altmetrics

Scientometrics volume 98, pages 1131–1143 (2014)Cite this article

Abstract

The rise of the social web and its uptake by scholars has led to the creation of altmetrics, which are social web metrics for academic publications. These new metrics can, in theory, be used in an evaluative role, to give early estimates of the impact of publications or to give estimates of non-traditional types of impact. They can also be used as an information seeking aid: to help draw a digital library user’s attention to papers that have attracted social web mentions. If altmetrics are to be trusted then they must be evaluated to see if the claims made about them are reasonable. Drawing upon previous citation analysis debates and web citation analysis research, this article discusses altmetric evaluation strategies, including correlation tests, content analyses, interviews and pragmatic analyses. It recommends that a range of methods are needed for altmetric evaluations, that the methods should focus on identifying the relative strengths of influences on altmetric creation, and that such evaluations should be prioritised in a logical order.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1

Notes

  1. http://www.arc.gov.au/era/faq.htm.

  2. http://www.ref.ac.uk/pubs/2011-02/#d.en.69578.

References

  • Bar-Ilan, J., Haustein, S., Peters, I., Priem, J., Shema, H., & Terliesner, J. (2012). Beyond citations: Scholars’ visibility on the social Web, 17th International Conference on Science and Technology Indicators (STI2012) (pp. 98–109). Montreal: Science-Metrix and OST.

  • Blackburn, J. L., & Hakel, M. D. (2006). An examination of sources of peer-review bias. Psychological Science, 17(5), 378–382. doi:10.1111/j.1467-9280.2006.01715.x.

    Article  Google Scholar 

  • Bornmann, L., Nast, I., & Daniel, H.-D. (2008). Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication. Scientometrics, 77(3), 415–432.

    Article  Google Scholar 

  • Brooks, T. A. (1986). Evidence of complex citer motivations. Journal of the American Society for Information Science, 37, 34–36.

    Google Scholar 

  • Case, D. O., & Higgins, G. M. (2000). How can we investigate citation behaviour? A study of reasons for citing literature in communication. Journal of the American Society for Information Science, 51(7), 635–645.

    Article  Google Scholar 

  • Ceci, S. J., & Williams, W. M. (2011). Understanding current causes of women’s underrepresentation in science. Proceedings of the National Academy of Sciences of the United States of America, 108(8), 3157–3162.

    Article  Google Scholar 

  • Desai, T., Shariff, A., Shariff, A., Kats, M., Fang, X., et al. (2012). Tweeting the meeting: An in-depth analysis of Twitter activity at Kidney Week 2011. PLoS One, 7(7), e40253. doi:10.1371/journal.pone.0040253.

    Article  Google Scholar 

  • Helic, H., Strohmaier, M., Trattner, C., Muhr, M., & Lerman, K. (2011). Pragmatic evaluation of folksonomies. Proceedings of the 20th international conference on world wide web (WWW2011) (pp. 417–426). New York, NY: ACM.

  • Horrobin, D. F. (1990). The philosophical basis of peer review and the suppression of innovation. Journal of the American Medical Association, 263(10), 1438–1441.

    Article  Google Scholar 

  • Kousha, K., & Thelwall, M. (2007). Google scholar citations and Google Web/URL citations: A multi-discipline exploratory analysis. Journal of the American Society for Information Science and Technology, 58(7), 1055–1065.

    Article  Google Scholar 

  • Kousha, K., & Thelwall, M. (2008). Assessing the impact of disciplinary research on teaching: An automatic analysis of online syllabuses. Journal of the American Society for Information Science and Technology, 59(13), 2060–2069.

    Article  Google Scholar 

  • Kousha, K., & Thelwall, M. (2009). Google book search: Citation analysis for social science and the humanities. Journal of the American Society for Information Science and Technology, 60(8), 1537–1549.

    Article  Google Scholar 

  • Kousha, K., Thelwall, M., & Rezaie, S. (2010). Using the web for research evaluation: The integrated online impact indicator. Journal of Informetrics, 4(1), 124–135.

    Article  Google Scholar 

  • Lee, C., Sugimoto, C. R., & Zhang, G. (2013). Bias in peer review. Journal of American Society for Information Science and Technology, 64(1), 2–17.

    Article  Google Scholar 

  • Levitt, J., & Thelwall, M. (2011). A combined bibliometric indicator to predict article impact. Information Processing and Management, 47(2), 300–3008.

    Article  Google Scholar 

  • Li, X., Thelwall, M., & Giustini, D. (2012). Validating online reference managers for scholarly impact measurement. Scientometrics, 91(2), 461–471.

    Article  Google Scholar 

  • MacRoberts, M. H., & MacRoberts, B. R. (1996). Problems of citation analysis. Scientometrics, 36(3), 435–444.

    Article  Google Scholar 

  • Mahoney, M. J. (1977). Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1(2), 161–175.

    Article  Google Scholar 

  • Marcus, A., & Oransky, I. (2011). Science publishing: The paper is not sacred. Nature, 480, 449–450.

    Article  Google Scholar 

  • Merton, R. K. (1973). The sociology of science. Theoretical and empirical investigations. Chicago: University of Chicago Press.

    Google Scholar 

  • Moed, H. F. (2005). Citation analysis in research evaluation. New York: Springer.

    Google Scholar 

  • Mohammadi, E., & Thelwall, M. (2013). Assessing non-standard article impact using F1000 labels. Scientometrics. doi:10.1007/s11192-013-0993-9.

  • Neuendorf, K. (2002). The content analysis guidebook. London: Sage.

    Google Scholar 

  • Oppenheim, C. (2000). Do patent citations count? In B. Cronin & H. B. Atkins (Eds.), The web of knowledge: A festschrift in honor of Eugene Garfield (pp. 405–432). Metford, NJ: Information Today Inc. ASIS Monograph Series.

    Google Scholar 

  • Peters, D. P., & Cecia, S. J. (1982). Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences, 5, 187–195.

    Article  Google Scholar 

  • Priem, J., & Costello, K. L. (2010). How and why scholars cite on twitter. Proceedings of the American society for information science and technology (ASIST 2010) (pp. 1–4). doi:10.1002/meet.14504701201.

  • Priem, J., Piwowar, H.A., & Hemminger, B.M. (2012). Altmetrics in the wild: using social media to explore scholarly impact. Retrieved from http://arxiv.org/abs/1203.4745v1. Accessed 23 Aug 2013.

  • Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2011). Altmetrics: A manifesto. Retrieved from http://altmetrics.org/manifesto/. Accessed 23 Aug 2013.

  • Procter, R., Williams, R., Stewart, J., Poschen, M., Snee, H., Voss, A., et al. (2010). Adoption and use of Web 2.0 in scholarly communications. Philosophical Transactions of the Royal Society A, 368(1926), 4039–4056.

    Article  Google Scholar 

  • Seglen, P. O. (1998). Citation rates and journal impact factors are not suitable for evaluation of research. Acta Orthopaedica Scandinavica, 69(3), 224–229.

    Google Scholar 

  • Shema, H., Bar-Ilan, J., & Thelwall, M. (2012). Research blogs and the discussion of scholarly information. PLoS One, 7(5), e35869. doi:10.1371/journal.pone.0035869.

    Article  Google Scholar 

  • Thelwall, M., Haustein, S., Larivière, V., & Sugimoto, C. R. (2013). Do altmetrics work? Twitter and ten other social web services. PLOS One, 8(5), e64841. doi:10.1371/journal.pone.0064841.

    Article  Google Scholar 

  • Thelwall, M., & Kousha, K. (2008). Online presentations as a source of scientific impact? An analysis of PowerPoint files citing academic journals. Journal of the American Society for Information Science and Technology, 59(5), 805–815.

    Article  Google Scholar 

  • van Raan, A. F. J. (1998). In matters of quantitative studies of science the fault of theorists is offering too little and asking too much. Scientometrics, 43(1), 129–148.

    Article  Google Scholar 

  • Vaughan, L., & Huysen, K. (2002). Relationship between links to journal Web sites and impact factors. ASLIB Proceedings, 54(6), 356–361.

    Article  Google Scholar 

  • Vaughan, L., & Shaw, D. (2003). Bibliographic and web citations: What is the difference? Journal of the American Society for Information Science and Technology, 54(14), 1313–1322.

    Article  Google Scholar 

  • Vaughan, L., & Shaw, D. (2005). Web citation data for impact assessment: A comparison of four science disciplines. Journal of the American Society for Information Science and Technology, 56(10), 1075–1087.

    Article  Google Scholar 

  • Weller, K., Dornstädter, R., Freimanis, R., Klein, R. N., & Perez, M. (2010). Social software in academia: Three studies on users’ acceptance of web 2.0 services. Proceedings of the 2nd Web Science Conference (WebSci10), Retrieved May 29, 2013 from http://www.phil-fak.uni-duesseldorf.de/fileadmin/Redaktion/Institute/Informationswissenschaft/weller/websci10_submission_62.pdf.

  • Wennerås, C., & Wold, A. (1997). Nepotism and sexism in peer-review. Nature, 387, 341–343.

    Article  Google Scholar 

  • Whitley, R. (2000). The intellectual and social organization of the sciences (2nd ed.). Oxford: Oxford University Press.

    Google Scholar 

Download references

Acknowledgments

This study is part of the FP7 EU-funded project ACUMEN on assessing Web indicators in research evaluation.

Author information

Authors and Affiliations

  1. Statistical Cybermetrics Research Group, School of Technology, University of Wolverhampton, Wulfruna Street, Wolverhampton, WV1 1SB, UK

    Pardeep Sud & Mike Thelwall

Authors
  1. Pardeep Sud
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mike Thelwall
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mike Thelwall.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Sud, P., Thelwall, M. Evaluating altmetrics. Scientometrics 98, 1131–1143 (2014). https://doi.org/10.1007/s11192-013-1117-2

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-013-1117-2

Keywords

  • Altmetrics
  • Indicators
  • Webometrics