Interpreting correlations between citation counts and other indicators
Altmetrics or other indicators for the impact of academic outputs are often correlated with citation counts in order to help assess their value. Nevertheless, there are no guidelines about how to assess the strengths of the correlations found. This is a problem because the correlation strength affects the conclusions that should be drawn. In response, this article uses experimental simulations to assess the correlation strengths to be expected under various different conditions. The results show that the correlation strength reflects not only the underlying degree of association but also the average magnitude of the numbers involved. Overall, the results suggest that due to the number of assumptions that must be made, in practice it will rarely be possible to make a realistic interpretation of the strength of a correlation coefficient.
KeywordsCitation analysis Correlation Altmetrics Indicators Discretised lognormal Simulation
- Else, H. (2015). Research funding formula tweaked after REF 2014 results. https://www.timeshighereducation.com/news/research-funding-formula-tweaked-after-ref-2014-results/2018685.article.
- Garanina, O. S., & Romanovsky, M. Y. (2015). Citation distribution of individual scientist: Approximations of stretch exponential distribution with power law tails. In A. A. Salah, Y. Tonta, A. A. Akdag Salah, C. Sugimoto, & U. Al (Eds.), Proceedings of ISSI 2015 (pp. 272–277). Turkey: Bogaziçi University Printhouse.Google Scholar
- Gillespie, C.S. (2015). Fitting heavy tailed distributions: the poweRlaw package. Journal of Statistical Software, 64(2), 1–16. http://www.jstatsoft.org/v64/i02/paper.
- HEFCE. (2015). The metric tide: Correlation analysis of REF2014 scores and metrics. Supplementary Report II to the Independent review of the role of metrics in research assessment and management. Bristol: Hefce. http://www.hefce.ac.uk/pubs/rereports/Year/2015/metrictide/Title,104463,en.html.
- Lipsey, M.W., Puzio, K., Yun, C., Hebert, M.A., Steinka-Fry, K., Cole, M.W., et al. (2012). Translating the statistical representation of the effects of education interventions into more readily interpretable forms. Washington, DC: US Dept of Education, National Center for Special Education Research, Institute of Education Sciences, NCSER 2013-3000.Google Scholar
- Low, W. J., Thelwall, M., & Wilson, P. (2015). Stopped sum models for citation data. In A. A. Salah, Y. Tonta, A. A. AkdagSalah, C. Sugimoto, & U. Al (Eds.), Proceedings of ISSI 2015 Istanbul: 15th international society of scientometrics and informetrics conference (pp. 184–194). Istanbul: Bogaziçi University Printhouse.Google Scholar
- Oppenheim, C. (2000). Do patent citations count? In B. Cronin & H. B. Atkins (Eds.), The web of knowledge: A festschrift in honor of Eugene Garfield (pp. 405–432). Metford: Information Today Inc. ASIS Monograph Series.Google Scholar
- Thelwall, M., & Wilson, P. (in press). Mendeley readership altmetrics for medical articles: An analysis of 45 fields. Journal of the Association for Information Science and Technology. doi:10.1002/asi.23501.
- Wilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S., Hill, S., et al. (2015). The metric tide: Report of the independent review of the role of metrics in research assessment and management. http://www.hefce.ac.uk/pubs/rereports/Year/2015/metrictide/Title,104463,en.html.