The rise of the social web and its uptake by scholars has led to the creation of altmetrics, which are social web metrics for academic publications. These new metrics can, in theory, be used in an evaluative role, to give early estimates of the impact of publications or to give estimates of non-traditional types of impact. They can also be used as an information seeking aid: to help draw a digital library user’s attention to papers that have attracted social web mentions. If altmetrics are to be trusted then they must be evaluated to see if the claims made about them are reasonable. Drawing upon previous citation analysis debates and web citation analysis research, this article discusses altmetric evaluation strategies, including correlation tests, content analyses, interviews and pragmatic analyses. It recommends that a range of methods are needed for altmetric evaluations, that the methods should focus on identifying the relative strengths of influences on altmetric creation, and that such evaluations should be prioritised in a logical order.
KeywordsAltmetrics Indicators Webometrics
This study is part of the FP7 EU-funded project ACUMEN on assessing Web indicators in research evaluation.
- Bar-Ilan, J., Haustein, S., Peters, I., Priem, J., Shema, H., & Terliesner, J. (2012). Beyond citations: Scholars’ visibility on the social Web, 17th International Conference on Science and Technology Indicators (STI2012) (pp. 98–109). Montreal: Science-Metrix and OST.Google Scholar
- Bornmann, L., Nast, I., & Daniel, H.-D. (2008). Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication. Scientometrics, 77(3), 415–432.CrossRefGoogle Scholar
- Brooks, T. A. (1986). Evidence of complex citer motivations. Journal of the American Society for Information Science, 37, 34–36.Google Scholar
- Helic, H., Strohmaier, M., Trattner, C., Muhr, M., & Lerman, K. (2011). Pragmatic evaluation of folksonomies. Proceedings of the 20th international conference on world wide web (WWW2011) (pp. 417–426). New York, NY: ACM.Google Scholar
- Merton, R. K. (1973). The sociology of science. Theoretical and empirical investigations. Chicago: University of Chicago Press.Google Scholar
- Moed, H. F. (2005). Citation analysis in research evaluation. New York: Springer.Google Scholar
- Mohammadi, E., & Thelwall, M. (2013). Assessing non-standard article impact using F1000 labels. Scientometrics. doi: 10.1007/s11192-013-0993-9.
- Neuendorf, K. (2002). The content analysis guidebook. London: Sage.Google Scholar
- Oppenheim, C. (2000). Do patent citations count? In B. Cronin & H. B. Atkins (Eds.), The web of knowledge: A festschrift in honor of Eugene Garfield (pp. 405–432). Metford, NJ: Information Today Inc. ASIS Monograph Series.Google Scholar
- Priem, J., & Costello, K. L. (2010). How and why scholars cite on twitter. Proceedings of the American society for information science and technology (ASIST 2010) (pp. 1–4). doi: 10.1002/meet.14504701201.
- Priem, J., Piwowar, H.A., & Hemminger, B.M. (2012). Altmetrics in the wild: using social media to explore scholarly impact. Retrieved from http://arxiv.org/abs/1203.4745v1. Accessed 23 Aug 2013.
- Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2011). Altmetrics: A manifesto. Retrieved from http://altmetrics.org/manifesto/. Accessed 23 Aug 2013.
- Seglen, P. O. (1998). Citation rates and journal impact factors are not suitable for evaluation of research. Acta Orthopaedica Scandinavica, 69(3), 224–229.Google Scholar
- Weller, K., Dornstädter, R., Freimanis, R., Klein, R. N., & Perez, M. (2010). Social software in academia: Three studies on users’ acceptance of web 2.0 services. Proceedings of the 2nd Web Science Conference (WebSci10), Retrieved May 29, 2013 from http://www.phil-fak.uni-duesseldorf.de/fileadmin/Redaktion/Institute/Informationswissenschaft/weller/websci10_submission_62.pdf.
- Whitley, R. (2000). The intellectual and social organization of the sciences (2nd ed.). Oxford: Oxford University Press.Google Scholar