, Volume 120, Issue 3, pp 1091–1110 | Cite as

Evaluation and comparison of influence in international Open Access journals between China and USA

  • Rongying Zhao
  • Xu WangEmail author


This paper makes an evaluation and comparison of influence on international Open Access (OA) between China and USA. First, Chinese and American OA journals are taken as an example. Second, combining traditional citation indicators with altmetrics indicators, a process of influence evaluation and comparison model on Chinese and American OA journals is constructed. Third, an evaluation indicator system of OA journals from two dimensions of academic influence and societal influence is constructed too. Lastly, through normalization, correlation analysis, reliability analysis, validity analysis, factor analysis and two-dimensional analysis, the academic and societal influence of OA journals are evaluated and compared between China and USA. It is found that in terms of academic influence and societal influence, the evaluation scores of international OA journals are quite different between China and USA. The influence of the USA is far greater than that of China. The results show a strong positive correlation among the 7 traditional citation indicators based on JCR, and a high positive correlation among the 8 altmetrics indicators based on Altmetrics Explorer, and the indicators have a significant consistency both overall and internal. There is a highly positive correlation between academic influence and societal influence scores for the Chinese and American OA journals. Overall, the results confirm that the evaluation of societal influence of OA journals is used as supplement to the evaluation of academic influence. Altmetrics is also a powerful tool to measure the societal influence of OA journals, but it cannot replace the traditional citation indicators at present.


International OA journal Citation indicator Altmetrics indicator Academic influence Societal influence Evaluation and comparison 



Thank for providing the dataset and anonymous reviewers for their useful comments. This research is funded by the National Social Science Found Major Project of China (18ZDA325), the National Social Science Found Project of China (16BTQ055) and the Fundamental Research Funds for the Central Universities (Independent Research Projects of Wuhan University (Humanities and Social Sciences)).


  1. Akers, K. G. (2017). Introducing altmetrics to the Journal of the Medical Library Association. Journal of the Medical Library Association Jmla, 105(3), 213–215. Scholar
  2. Amath, A., Ambacher, K., Leddy, J. J., Wood, T. J., & Ramnanan, C. J. (2017). Comparing alternative and traditional dissemination metrics in medical education. Medical Education, 51(9), 935.Google Scholar
  3. Bai, X., Xia, F., Ivan, L., Zhang, J., & Ning, Z. (2016). Identifying anomalous citations for objective evaluation of scholarly article impact. PLoS ONE, 11(9), e162364.Google Scholar
  4. Bornmann, L. (2014). Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of Informetrics, 8(4), 895–903.Google Scholar
  5. Bornmann, L. (2015). Usefulness of altmetrics for measuring the broader impact of research: A case study using data from PLOS and F1000Prime. Aslib Journal of Information Management, 67(3), 305–319.Google Scholar
  6. Bornmann, L., & Haunschild, R. (2018). Do altmetrics correlate with the quality of papers? A large-scale empirical study based on F1000Prime data. PLoS ONE, 13(5), e0197133.Google Scholar
  7. Chang, Y. (2017). Comparative study of characteristics of authors between open access and non-open access journals in library and information science. Library & Information Science Research, 39(1), 8–15. Scholar
  8. Costas, R., Zahedi, Z., & Wouters, P. (2014). Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Association for Information Science & Technology, 66(10), 2003–2019.Google Scholar
  9. Davis, P. M. (2011). Open access, readership, citations: A randomized controlled trial of scientific journal publishing. FASEB Journal, 25(7), 2129–2134. Scholar
  10. Ezema, I. J., & Onyancha, O. B. (2017). Citation impact of health and medical journals in Africa: Does open accessibility matter? Electronic Library, 35(5), 934–952. Scholar
  11. Garfield, E. (1955). Citation indexes for science; a new dimension in documentation through association of ideas. Science, 122(3159), 108–111.Google Scholar
  12. Garfield, E. (1970). Citation indexing for studying science. Nature, 227(5260), 870.Google Scholar
  13. Guo, F., Xue, J., & Li, R. (2014). Open access in China: A study of social science journals. Journal of Scholarly Publishing, 45(4), 336–352. Scholar
  14. Hallo, M., Lujan-Mora, S., & Mate, A. (2017). Evaluating open access journals using Semantic Web technologies and scorecards. Journal of Information Science, 43(1), 3–16. Scholar
  15. Harnad, S. (2009). Open access scientometrics and the UK Research Assessment Exercise. Scientometrics, 79(1), 147–156. Scholar
  16. Holmberg, K., & Park, H. W. (2018). An altmetric investigation of the online visibility of South Korea-based scientific journals. Scientometrics, 117(1), 603–613. Scholar
  17. Hu, D., Huang, B., & Zhou, W. (2012). Open access journals in China: The current situation and development strategies. Serials Review, 38(2), 86–92. Scholar
  18. Hua, F., Sun, H., Walsh, T., Glenny, A. M., & Worthington, H. (2017). Open access to journal articles in oncology: Current situation and citation impact. Annals of Oncology, 28(10), 2612–2617. Scholar
  19. Huang, W., Wang, P., & Wu, Q. (2018). A correlation comparison between Altmetric Attention Scores and citations for six PLOS journals. PLoS ONE, 13(4), e194962.Google Scholar
  20. Jabaley, C. S., Groff, R. F., Stentz, M. J., Moll, V., Lynde, G. C., Blum, J. M., et al. (2018). Highly visible sepsis publications from 2012 to 2017: Analysis and comparison of altmetrics and bibliometrics. Journal of Critical Care, 48, 357–371. Scholar
  21. Jamali, H. R., & Alimohammadi, D. (2015). Blog citations as indicators of the societal impact of research: Content analysis of social sciences blogs. International Journal of Knowledge Content Development & Technology, 5(1), 15–32.Google Scholar
  22. Jan, V. (2003). Should scholarly societies embrace open access (or is it the kiss of death)? Learned Publishing, 16(3), 167–169.Google Scholar
  23. Jokic, M., Mervar, A., & Mateljan, S. (2018). Scientific potential of European fully open access journals. Scientometrics, 114(3), 1373–1394. Scholar
  24. Koler-Povh, T., Juznic, P., & Turk, G. (2014). Impact of open access on citation of scholarly publications in the field of civil engineering. Scientometrics, 98(2), 1033–1045. Scholar
  25. Lawrence, S. (2001). Free online availability substantially increases a paper’s impact. Nature, 411(6837), 521.Google Scholar
  26. Liu, C. L., Xu, Y. Q., Wu, H., Chen, S. S., & Guo, J. J. (2013). Correlation and interaction visualization of altmetric indicators extracted from scholarly social network activities: Dimensions and structure. Journal of Medical Internet Research, 15(11), e259. Scholar
  27. Liu, W., & Li, Y. (2018). Open access publications in sciences and social sciences: A comparative analysis. Learned Publishing, 31(2), 107–119. Scholar
  28. Luis Ortega, J. (2017). The presence of academic journals on Twitter and its relationship with dissemination (tweets) and research impact (citations). Aslib Journal of Information Management, 69(6), 674–687. Scholar
  29. Maflahi, N., & Thelwall, M. (2018). How quickly do publications get read? The evolution of Mendeley reader counts for new articles. Journal of the Association for Information Science and Technology, 69(1), 158–167. Scholar
  30. Maggio, L. A., Leroux, T. C., Meyer, H. S., & Artino, A. R. (2018). #meded: Exploring the relationship between altmetrics and traditional measures of dissemination in health professions education. Perspectives on Medical Education, 7(4), 239–247.Google Scholar
  31. Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations/Robert K. Merton; edited and with an introd. by Norman W. Storer. Contemporary Sociology, 5(5), 52.Google Scholar
  32. Merton, R. K. (1988). The Matthew effect in science, II: Cumulative Advantage and the symbolism of intellectual property. ISIS, 79(4), 606–623.Google Scholar
  33. Moed, H. F. (2009). New developments in the use of citation analysis in research evaluation. Archivum Immunologiae et Therapiae Experimentalis, 57(1), 13–18. Scholar
  34. Moed, H. F., & Halevi, G. (2015). Multidimensional assessment of scholarly research impact. Journal of the Association for Information Science & Technology, 66(10), 1988–2002.Google Scholar
  35. Moksness, L., & Olsen, S. O. (2017). Understanding researchers’ intention to publish in open access journals. Journal of Documentation, 73(6), 1149–1166. Scholar
  36. Pisoschi, A. M., & Pisoschi, C. G. (2016). Is open access the solution to increase the impact of scientific journals? Scientometrics, 109(2), 1075–1095. Scholar
  37. Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2010). altmetrics: A manifesto. Retrieved March 1, 2019, from
  38. Shen, C. (2017). Open access scholarly journal publishing in Chinese. Publications. Scholar
  39. Sotudeh, H., & Estakhr, Z. (2018). Sustainability of open access citation advantage: The case of Elsevier’s author-pays hybrid open access journals. Scientometrics, 115(1), 563–576. Scholar
  40. Spearman, C. (1987). The proof and measurement of association between two things. American Journal of Psychology, 100(3/4), 441–471.Google Scholar
  41. Sugimoto, C. R., Work, S., Larivière, V., & Haustein, S. (2017). Scholarly use of social media and altmetrics: A review of the literature. Journal of the Association for Information Science & Technology, 68(9), 2037–2062.Google Scholar
  42. Thelwall, M. (2017). Are Mendeley reader counts high enough for research evaluations when articles are published? Aslib Journal of Information Management, 69(2), 174–183. Scholar
  43. Thelwall, M., & Nevill, T. (2018). Could scientists use scores to predict longer term citation counts? Journal of Informetrics, 12(1), 237–248.Google Scholar
  44. Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of Informetrics, 10(2), 365–391.MathSciNetGoogle Scholar
  45. Werner, R. (2015). The focus on bibliometrics makes papers less useful. Nature, 517(7534), 245.Google Scholar
  46. Xia, J. (2010). A longitudinal study of scholars attitudes and behaviors toward open-access journal publishing. Journal of the Association for Information Science & Technology, 61(3), 615–624. Scholar
  47. Yan, E., & Li, K. (2018). Which domains do open-access journals do best in? A 5-year longitudinal study. Journal of the Association for Information Science and Technology, 69(6), 844–856. Scholar
  48. Yang, S., Xing, X., & Wolfram, D. (2018). Difference in the impact of open-access papers published by China and the USA. Scientometrics, 115(2), 1017–1037. Scholar
  49. Yu, H. (2017). Context of altmetrics data matters: An investigation of count type and user category. Scientometrics, 111(1), 1–17.Google Scholar
  50. Yu, L., Pan, Y., & Wu, Y. (2009). Research on data normalization methods in multi-attribute evaluation. Library and Information Service, 53(12), 136–139.Google Scholar
  51. Yuan, S., & Hua, W. (2011). Scholarly impact measurements of LIS open access journals: Based on citations and links. Electronic Library, 29(5), 682–697. Scholar
  52. Zhang, D. X. (2014). Development of open access in China: Strategies, practices, challenges. Insights, 27(1), 45–50.Google Scholar
  53. Zhang, L., & Watson, E. M. (2017). Measuring the impact of gold and green open access. Journal of Academic Librarianship, 43(4), 337–345. Scholar
  54. Zygmont, C., & Smith, M. R. (2014). Robust factor analysis in the presence of normality violations, missing data, and outliers: Empirical questions and possible solutions. The Quantitative Methods for Psychology, 10(1), 40–55.Google Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2019

Authors and Affiliations

  1. 1.Research Center for Chinese Science EvaluationWuhan UniversityWuhanChina
  2. 2.Center for Studies of Information ResourcesWuhan UniversityWuhanChina
  3. 3.School of Information ManagementWuhan UniversityWuhanChina

Personalised recommendations