Research on impact evaluation of open access journals

  • Mingkun WeiEmail author


Impact evaluation of open access journals represent how citation indicator and open metric indicators may be affected by open access. Citation indicators and open metric indicators are used in integrated assessments of impact evaluation. Here we test, systematically across many indicators, how well such impact indicators capture the impacts of OA journals. Using the Scopus as data sources for impact evaluation of OA journals, we find that the majority of indicators underestimate the value of freedom in impact evaluation such as citation indicators, while impacts on citation impact are overestimated in impact evaluation. This has important implications for impact evaluation of academic exchange that rely on citation frequency. It also means that citation indicators reflect part of impact of OA journal. This research evaluates the impact of OA journals from citation indicators and open metric indicators, which plays an important role in the development of OA journals.


OA journal Impact evaluation Open metric impact Citation impact 



The author would like to thank the anonymous reviewers for the valuable comments and suggestions. This paper is supported by Social Science Foundation of Hebei Province in China (Grant No. HB19TQ014).


  1. Abramo, G., & D’Angelo, C. A. (2011). Evaluating research: From informed peer review to bibliometrics. Scientometrics,87(3), 499–514.CrossRefGoogle Scholar
  2. Adler, N. J., & Harzing, A. W. (2009). When knowledge wins: Transcending the sense and nonsense of academic rankings. Academy of Management Learning & Education,8(1), 72–95.CrossRefGoogle Scholar
  3. Berhidi, A., Szluka, P., et al. (2009). New bibliometric indicators. Is this the end of the impact factor era? Magyar Onkologia,53(2), 115.CrossRefGoogle Scholar
  4. Bornmann, L., & Daniel, H. D. (2005). Does the h-index for ranking of scientists really work? Scientometrics,65(3), 391–392.CrossRefGoogle Scholar
  5. Bornmann, L., & Daniel, H. D. (2007). What do we know about the h index? Journal of the Association for Information Science and Technology,58(9), 1381–1385.Google Scholar
  6. Bornmann, L., & Daniel, H. D. (2008). Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine. New York: Wiley.CrossRefGoogle Scholar
  7. Braun, T. and W. Nzel, et al. (2005). A Hirsch-type index for journals the scientist magazine of the life sciences. Scientist 19. Google Scholar
  8. Braun, T., Glänzel, W., et al. (2006). A Hirsch-type index for journals. Scientometrics,69(1), 169–173.CrossRefGoogle Scholar
  9. Butler, D. (2008). Free journal-ranking tool enters citation market. Nature,451(7174), 6.CrossRefGoogle Scholar
  10. Colledge, L., Moya Anegón, F. D., et al. (2010). SJR and SNIP: Two new journal metrics in Elsevier’s Scopus. Serials the Journal for the Serials Community,23(3), 215–221.CrossRefGoogle Scholar
  11. Costas, R., & Bordons, M. (2007). The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level. Journal of Informetrics,1(3), 193–203.CrossRefGoogle Scholar
  12. Cronin, B., & Meho, L. (2014). Using the h-index to rank influential information scientists. Journal of the Association for Information Science & Technology,57(9), 1275–1278.Google Scholar
  13. Da Silva, J. A. T., & Memon, A. R. (2017). CiteScore: A cite for sore eyes, or a valuable, transparent metric? Scientometrics,111, 1–4.CrossRefGoogle Scholar
  14. Efremenkova, V. M., & Gonnova, S. M. (2016). A comparison of Scopus and WoS database subject classifiers in mathematical disciplines. Scientific & Technical Information Processing,43(2), 115–122.CrossRefGoogle Scholar
  15. Eger, T., Scheufen, M., et al. (2016). The determinants of open access publishing: Survey evidence from countries in the Mediterranean Open Access Network (MedOANet). Economia E Politica Industriale forthcoming,4, 1–27.Google Scholar
  16. Egghe, L. (2000). Theory and practice of the g-Index. Scientometrics,69(1), 131–152.CrossRefGoogle Scholar
  17. Falagas, M. E., Kouranos, V. D., et al. (2008). Comparison of SCImago journal rank indicator with journal impact factor. Faseb Journal Official Publication of the Federation of American Societies for Experimental Biology,22(8), 2623.CrossRefGoogle Scholar
  18. Franceschet, M. (2010). A comparison of bibliometric indicators for computer science scholars and journals on Web of Science and Google Scholar. Scientometrics,83(1), 243–258.CrossRefGoogle Scholar
  19. Garfield, E. (1955). Citation indexes for science; A new dimension in documentation through association of ideas. International Journal of Epidemiology,122(3159), 108.Google Scholar
  20. Garfield, E. (1964). “Science Citation Index”—a new dimension in indexing. Science,144(3619), 649.CrossRefGoogle Scholar
  21. Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science,178(4060), 471–479.CrossRefGoogle Scholar
  22. Garfield, E., & Sher, I. H. (1963). New factors in the evaluation of scientific literature through citation indexing. Journal of the Association for Information Science and Technology,14(3), 195–201.Google Scholar
  23. Glänzel, W. (2006). On the h-index—a mathematical approach to a new measure of publication activity and citation impact. Scientometrics,67(2), 315–321.CrossRefGoogle Scholar
  24. Glänzel, W., & Moed, H. F. (2002). Journal impact measures in bibliometric research. Scientometrics,53(2), 171–193.CrossRefGoogle Scholar
  25. González-Pereira, B., Guerrero-Bote, V. P., et al. (2010). A new approach to the metric of journals’ scientific prestige: The SJR indicator. Journal of Informetrics,4(3), 379–391.CrossRefGoogle Scholar
  26. Guerrero-Bote, V. P., & Moya-Anegón, F. (2012). A further step forward in measuring journals’scientific prestige: The SJR2 indicator. Journal of Informetrics,6(4), 674–688.CrossRefGoogle Scholar
  27. Harzing, A. W., & Ron, V. D. W. (2009). A Google scholar h-index for journals: An alternative metric to measure journal impact in economics and business? Journal of the American Society for Information Science and Technology,60(1), 41–46.CrossRefGoogle Scholar
  28. Harzing, A. W., & Wal, R. V. D. (2009). A Google scholar h-index for journals: An alternative metric to measure journal impact in economics and business. Journal of the Association for Information Science and Technology,60(1), 41–46.Google Scholar
  29. Heneberg, P. (2019). The troubles of high-profile open access megajournals. Scientometrics,120(2), 733–746.CrossRefGoogle Scholar
  30. Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America,102(46), 16569–16572.zbMATHCrossRefGoogle Scholar
  31. Hodge, D. R., & Lacasse, J. R. (2011). Evaluating Journal Quality: Is the H-Index a Better Measure than Impact Factors? Research on Social Work Practice,20(2), 222–230.CrossRefGoogle Scholar
  32. Jin, B. H. (2007). The R- and AR-indices: Complementing the -index. Chinese Science Bulletin,52(6), 855–863.CrossRefGoogle Scholar
  33. Laakso, M., & Björk, B. (2016). Hybrid open access—A longitudinal study. Journal of Informetrics,10, 919–932.CrossRefGoogle Scholar
  34. Lancho-Barrantes, B. S., Guerrero-Bote, V. P., et al. (2010). What lies behind the averages and significance of citation indicators in different disciplines? Journal of Information Science,36(3), 371–382.CrossRefGoogle Scholar
  35. Lehmann, S., Jackson, A. D., et al. (2006). Measures for measures. Nature,444(7122), 1003–1004.CrossRefGoogle Scholar
  36. Leydesdorff, L., & Milojević, S. (2015). The citation impact of german sociology journals: Some problems with the use of scientometric indicators in journal and research evaluations. Zeitschrift Für Sozialwissenschaftliche Forschung Und Praxis,66(2), 193–204.Google Scholar
  37. Leydesdorff, L., & Opthof, T. (2010). Scopus’s Source Normalized Impact per Paper (SNIP) versus a journal impact factor based on fractional counting of citations. Journal of the Association for Information Science and Technology,61(11), 2365–2369.Google Scholar
  38. Mingers, J. (2009). Measuring the research contribution of management academics using the Hirsch-index. Journal of the Operational Research Society,60(9), 1143–1153.CrossRefGoogle Scholar
  39. Mingers, J., & Leydesdorff, L. (2015). Identifying research fields within business and management: A journal cross-citation analysis. Journal of the Operational Research Society,66(8), 1370–1384.CrossRefGoogle Scholar
  40. Mingers, J., Macri, F., et al. (2012). Using the h-index to measure the quality of journals in the field of business and management. Information Processing and Management,48(2), 234–241.CrossRefGoogle Scholar
  41. Moed, H. F. (2010). Measuring contextual citation impact of scientific journals. Journal of Informetrics,4(3), 265–277.CrossRefGoogle Scholar
  42. Moed, H. F. (2011). The source normalized impact per paper is a valid and sophisticated indicator of journal citation impact. Journal of the Association for Information Science and Technology,62(1), 211–213.Google Scholar
  43. Moed, H. F. (2012). The future of research evaluation rests with an intelligent combination of advanced metrics and transparent peer review. Science & Public Policy,34(8), 575–583.CrossRefGoogle Scholar
  44. Moussa, S., & Touzani, M. (2010). Ranking marketing journals using the Google scholar-based hg -index. Journal of Informetrics,4(1), 107–117.CrossRefGoogle Scholar
  45. Norris, M., & Oppenheim, C. (2010). The h-index: a broad review of a new bibliometric indicator. Journal of Documentation,66(5), 681–705.CrossRefGoogle Scholar
  46. Oppenheim, C. (2007). Using the h-index to rank influential British researchers in information science and librarianship: Brief communication. New York: Wiley.CrossRefGoogle Scholar
  47. Oppenheim, C. (2014). Using the h-index to rank influential British researchers in information science and librarianship. Journal of the American Society for Information Science and Technology,58(2), 297–301.CrossRefGoogle Scholar
  48. Priem, J. & H. Piwowar (2012). The launch of impact story: Using altmetrics to tell data-driven stories.Google Scholar
  49. Raan, A. F. J. V. (2006). Comparisonof the Hirsch-index with standard bibliometric indicators and with peerjudgment. Scientometrics,67(3), 491–502.CrossRefGoogle Scholar
  50. Ramin, S., & Sarraf, S. A. (2012). Comparison between Impact factor, SCImago journal rank indicator and Eigenfactor score of nuclear medicine journals. Nuclear Medicine Review Central & Eastern Europe,15(2), 132–136.Google Scholar
  51. Saad, G. (2006). Exploring the h-index at the author and journal levels using bibliometric data of productive consumer scholars and business-related journals respectively. Scientometrics,69(1), 117–120.CrossRefGoogle Scholar
  52. Schubert, W. (2007). The use of bibliometrics to measure research quality in UK higher education institutions Glänzel. Evidence Ltd for Universities Uk,57(1), 19–32.Google Scholar
  53. Schubert, A., & Glänzel, W. (2007). A systematic analysis of Hirsch-type indices for journals. Journal of Informetrics,1(3), 179–184.CrossRefGoogle Scholar
  54. Truex, D. P. and M. J. Cuellar, et al. (2009). Assessing scholarly influence: Using the hirsch indices to reframe the discourse.Google Scholar
  55. Van Noorden, N. (2016). Controversial impact factor gets a heavyweight rival. Nature News,540(7633), 325–326.CrossRefGoogle Scholar
  56. Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of Informetrics,10(2), 365–391.CrossRefGoogle Scholar
  57. Wang, X., Chen, L., et al. (2015). The open access advantage considering citation, article usage and social media attention. Scientometrics,103(2), 555–564.CrossRefGoogle Scholar
  58. Wilsdon, J., Allen, L., et al. (2015). The metric tide: Report of the independent review of the role of metrics in research assessment and management. London: HEFCE. Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2019

Authors and Affiliations

  1. 1.School of ManagementHebei UniversityBaodingChina

Personalised recommendations