Advertisement

Scientometrics

, Volume 85, Issue 1, pp 81–93 | Cite as

Correlation between the Journal Impact Factor and three other journal citation indices

  • Mark R. ElkinsEmail author
  • Christopher G. Maher
  • Robert D. Herbert
  • Anne M. Moseley
  • Catherine Sherrington
Article

Abstract

To determine the degree of correlation among journal citation indices that reflect the average number of citations per article, the most recent journal ratings were downloaded from the websites publishing four journal citation indices: the Institute of Scientific Information’s journal impact factor index, Eigenfactor’s article influence index, SCImago’s journal rank index and Scopus’ trend line index. Correlations were determined for each pair of indices, using ratings from all journals that could be identified as having been rated on both indices. Correlations between the six possible pairings of the four indices were tested with Spearman’s rho. Within each of the six possible pairings, the prevalence of identifiable errors was examined in a random selection of 10 journals and among the 10 most discordantly ranked journals on the two indices. The number of journals that could be matched within each pair of indices ranged from 1,857 to 6,508. Paired ratings for all journals showed strong to very strong correlations, with Spearman’s rho values ranging from 0.61 to 0.89, all p < 0.001. Identifiable errors were more common among scores for journals that had very discordant ranks on a pair of indices. These four journal citation indices were significantly correlated, providing evidence of convergent validity (i.e. they reflect the same underlying construct of average citability per article in a journal). Discordance in the ranking of a journal on two indices was in some cases due to an error in one index.

Keywords

Bibliometric analysis Citation analysis Impact factor 

Notes

Acknowledgments

We thank Jessica Patton and Narelle Cox for their assistance with retrieval of the STL scores and checking for errors.

References

  1. Adam, D. (2002). The counting house. Nature, 415(6873), 726–729.Google Scholar
  2. Banks, M. A., & Dellavalle, R. (2008). Emerging alternatives to the impact factor. OCLC Systems & Services: International Digital Library Perspectives, 24(3), 167–173.Google Scholar
  3. Bollen, J., et al. (2009). A principal component analysis of 39 scientific impact measures. PLoS One, 4(6), e6022.CrossRefGoogle Scholar
  4. Cheung, C. K. (2008). Audience matters: A study of how authors select educational journals. The Asia-Pacific Education Researcher, 17(2), 191–201.Google Scholar
  5. Cozby, P. C. (2008). Methods in Behavioral Research (10 ed.). New York: McGraw Hill.Google Scholar
  6. Davis, P. M. (2008). Eigenfactor: Does the principle of repeated improvement result in better estimates than raw citation counts? Journal of the American Society for Information Science and Technology, 59(13), 2186–2189.CrossRefGoogle Scholar
  7. Davis, G., & Royle, P. (1996). A comparison of Australian university output using journal impact factors. Scientometrics, 35(1), 45–58.CrossRefGoogle Scholar
  8. Duy, J., & Vaughan, L. (2006). Can electronic journal usage data replace citation data as a measure of journal use? An empirical examination. The Journal of Academic Librarianship, 32(5), 512–517.CrossRefGoogle Scholar
  9. Falagas, M. E., et al. (2008). Comparison of SCImago journal rank indicator with journal impact factor. The FASEB Journal, 22, 2623–2628.CrossRefGoogle Scholar
  10. Fassoulaki, A., et al. (2001). Academic anesthesiologists’ views on the importance of the impact factor of scientific journals: A North American and European survey. Canadian Journal of Anesthesia, 48(10), 953–957.CrossRefGoogle Scholar
  11. Franceschet, M. (2010). Journal influence factors. Journal of Informetrics. doi: 10.1016/j.joi.2009.12.002.
  12. Fuyuno, I., & Cyranoski, D. (2006). Cash for papers: Putting a premium on publication. Nature, 441(7095), 792.CrossRefGoogle Scholar
  13. Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science, 178(4060), 471–479.CrossRefGoogle Scholar
  14. Garfield, E. (2006). The history and meaning of the journal impact factor. JAMA, 295(1), 90–93.CrossRefGoogle Scholar
  15. Gavel, Y., & Iselid, L. (2008). Web of Science and Scopus: a journal title overlap study. Online Information Review, 32(1), 8.CrossRefGoogle Scholar
  16. Gordon, M. D. (1982). Citation ranking versus subjective evaluation in the determination of the journal hierarchies in the social sciences. Journal of the American Society for Information Science, 33(1), 55–57.CrossRefGoogle Scholar
  17. Hecht, F., Hecht, B. K., & Sandberg, A. A. (1998). The journal “impact factor”: A misnamed, misleading, misused measure. Cancer Genetics and Cytogenetics, 104(2), 77–81.CrossRefGoogle Scholar
  18. Hernán, M. A. (2008). Epidemiologists (of all people) should question journal impact factors. Epidemiology, 19, 366–368.Google Scholar
  19. Kelland, J., & Young, A. P. (1994). Citation as a form of library use. Collection Management, 19(1/2), 81–100.CrossRefGoogle Scholar
  20. Kurmis, A. P. (2003). Understanding the limitations of the journal impact factor. The Journal of Bone and Joint Surgery, 85, 2449–2454.Google Scholar
  21. Leydesdorff, L. (2009). How are new citation-based journal indicators adding to the bibliometric toolbox? Journal of the American Society for Information Science and Technology, 60(7), 1327–1336.CrossRefGoogle Scholar
  22. Midorikawa, N., et al. (1984). The relationships among the citation measures and the factors influence on them. Information Services and Use, 4(6), 417–424.Google Scholar
  23. Monastersky, R. (2005). The number that’s devouring science. The Chronicle of Higher Education, 52(8), A12–A22.Google Scholar
  24. Opthof, T. (1997). Sense and nonsense about the impact factor. Cadiovascular Research, 33(1), 1–7.CrossRefGoogle Scholar
  25. PLoS Medicine Editors. (2006). The impact factor game. PLoS Medicine, 3(6), p. e291, 1–2.Google Scholar
  26. Reedijk, J. (1998). Sense and nonsense of science citation analyses: Comments on the monopoly position of ISI and citation inaccuracies. Risks of possible misuse and biased citation and impact data. New Journal of Chemistry, 22, 767–770.Google Scholar
  27. Rey-Rocha, J., et al. (2001). Some misuses of journal impact factor in research evaluation. Cortex, 37(4), 595–597.CrossRefGoogle Scholar
  28. Rousseau, R., & STIMULATE 8 Group. (2009). On the relation between the WoS impact factor, the eigenfactor, the SCImago journal rank, the article influence score and the journal h-index. http://eprints.rclis.org/16448/. Accessed 4 July 2009.
  29. Schein, M. (2000). What American surgeons read: A survey of a thousand Fellows of the American College of Surgeons. Current Surgery, 57(3), 252–258.CrossRefGoogle Scholar
  30. Smart, J. C., & Elton, C. F. (1982). Consumption factor scores of psychology journals. Scientometrics, 4(5), 349–360.CrossRefGoogle Scholar
  31. van Leeuwen, T., & Moed, H. F. (2005). Characteristics of journal impact factors: The effect of uncitedness and citation distribution on the understanding of journal impact factors. Scientometrics, 63(2), 357–371.CrossRefGoogle Scholar
  32. Wade, N. (1975). Citation analysis: A new tool for science administrators. Science, 188(4187), 429–432.CrossRefGoogle Scholar
  33. Yue, W., Wilson, C. S., & Rousseau, R. (2004). The immediacy index and the journal impact factor: Two highly correlated derived measures. The Canadian Journal of Infomation and Library Science, 28(1), 33–48.Google Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2010

Authors and Affiliations

  • Mark R. Elkins
    • 1
    Email author
  • Christopher G. Maher
    • 2
  • Robert D. Herbert
    • 2
  • Anne M. Moseley
    • 2
  • Catherine Sherrington
    • 2
  1. 1.Department of Respiratory MedicineRoyal Prince Alfred HospitalCamperdownAustralia
  2. 2.The George Institute for International HealthThe University of SydneySydneyAustralia

Personalised recommendations