And now for something completely different: the congruence of the Altmetric Attention Score’s structure between different article groups
- 859 Downloads
Altmetric Attention Score (AAS) is an increasingly popular composite altmetric measure, which is being criticized for an inappropriate and arbitrary aggregation of different altmetric sources into a single measure. We examined this issue empirically, by testing unidimensionality and the component structure congruence of the five ‘key’ AAS components: News, Blogs, Twitter, Facebook, and Google+. As a reference point, these tests were also done on different citation data: WoS, Scopus, and Google Scholar. All tests were done for groups of articles with: (1) high citations, but lower AAS (HCGs), and (2) high AAS, but lower citations (HAGs). Changes in component structures over time (from 2016 to 2017) were also considered. Citation data consistently formed congruent unidimensional structures for all groups and over time. Altmetric data formed congruent unidimensional structures only for the HCGs, with much inconsistency for the HAGs (including change over time). The relationship between Twitter and News counts was shown to be curvilinear. It was not possible to obtain a satisfactory congruent and reliable linear unidimensional altmetric structure between the groups for any variable combination, even after Mendeley and CiteULike altmetric counts were included. Correlations of altmetric aggregates and citations were fairly inconsistent between the groups. We advise against the usage of composite altmetric measures (including the AAS) for any group comparison purposes, until the measurement invariance issues are dealt with. The underlying pattern of associations between individual altmetrics is likely too complex and inconsistent across conditions to justify them being simply aggregated into a single score.
KeywordsAltmetrics Altmetric Attention Score (AAS) Citations Measurement invariance Measurement congruence
- Development Core Team, R. (2005). R: A language and environment for statistical computing [Computer Software]. Austria: R Foundation for Statistical Computing.Google Scholar
- Kline, P. (2010). Handbook of psychological testing (2nd ed.). London: Routledge.Google Scholar
- McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Erlbaum.Google Scholar
- Nuredini, K., & Peters, I. (2016). Enriching the knowledge of altmetrics studies by exploring social media metrics for Economic and Business Studies journals. EconStor Conference Papers, ZBW—German National Library of Economics. Retrieved from http://EconPapers.repec.org/RePEc:zbw:esconf:146879.
- O’Reilly, T. (2005). What is web 2.0? Design patterns and business models for the next generation of software. Retrieved from http://www.oreilly.com/pub/a/web2/archive/what-is-web-20.html?page=1.
- Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2010). Altmetrics: A manifesto. Retrieved from http://altmetrics.org/manifesto/.
- Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6th ed.). Boston, MA: Pearson.Google Scholar
- Tucker, L. R. (1951). A method for synthesis of factor analysis studies (Personnel Research Section Report No. 984). Washington, DC: Department of the Army.Google Scholar
- Wouters, P., & Costas, R. (2012). Users, narcissism and control—Tracking the impact of scholarly publications in the 21st century. Utrecht: SURF foundation.Google Scholar
- Wu, A. D., Li, Z., & Zumbo, B. D. (2007). Decoding the meaning of factorial invariance and updating the practice of multi-group confirmatory factor analysis: A demonstration with TIMSS data. Practical Assessment Research and Evaluation, 12(3), 1–26.Google Scholar