Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Building journal’s long-term impact: using indicators detected from the sustained active articles

Abstract

The Journal’s Impact Factor is an appropriate measure of recent concern rather than an effective measure of long-term impact of journals. This paper is mainly to find indicators that can effectively quantify the long-term impact of journal, with the aim to provide more useful supplementary information for journal evaluation. By examining the correlation between articles’ past citations and their future citations in different time windows, we found that the articles which were referenced in the past years will yield useful information also in the future. The age characteristics of these sustained active articles in journals provide clues for establishing long-term impact metrics for journals. A new indicator: h1-index was proposed to extract the active articles with at least the same number of citations as the h1-index in the statistical year. On this basis, four indicators describing the age characteristics of active articles were proposed to quantify the long-term impact of journals. The experimental results show that these indicators have a high correlation with the journal’s total citations, indicating that it is appropriate for these indicators to express the impact of the journal. Combining the average age of the active articles with the impact factors of journals, we found that some journals with short-term attraction strategies can also build long-term impact. The indicators presented in this paper that describe the long-term impact of journals will be a useful complement to journal quality assessment.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

References

  1. Andrei, T., Teodorescu, D., & Mirica, A. (2016). Beyond the impact factor: Measuring the international visibility of Romanian social sciences journals. Scientometrics, 108(1), 1–20.

  2. Antonoyiannakis, M. (2018). Impact factors and the central limit theorem: Why citation average are scale dependent. Journal of Informetrics, 12(4), 1072–1088.

  3. Archambault, E., & Larivière, V. (2009). History of the journal impact factor: Contingencies and consequences. Scientometrics, 79(3), 635–649.

  4. Asnafi, S., Gunderson, T., McDonald, R. J., & Kallmes, D. F. (2017). Association of h-index of editorial board members and impact factor among radiology journals. Academic Radiology, 24(2), 119–123.

  5. Bador, P., & Lafouge, T. (2010). Comparative analysis between impact factor and h-index for pharmacology and psychiatry journals. Scientometrics, 84(1), 65–69.

  6. Barker, K. L., & Thyer, B. A. (2005). An empirical evaluation of the editorial practices of social work journals: Voices of authors pubished in 2000. Journal of Social Service Research, 32(1), 17–31.

  7. Bertoli-Barsotti, L., & Lando, T. (2017a). The h-index as an almost-exact function of some basic statistics. Scientometrics, 113(2), 1209–1228.

  8. Bertoli-Barsotti, L., & Lando, T. (2017b). A theoretical model of the relationship between the h-index and other simple citation indicators. Scientometrics, 111(3), 1415–1448.

  9. Bogocz, J., Bak, A., & Polanski, J. (2014). No free lunches in nature? An analysis of the regional distribution of the affiliations of nature publications. Scientometrics, 101(1), 547–568.

  10. Bornmann, L. (2017). Confidence intervals for Journal Impact Factors. Scientometrics, 111(3), 1869–1871.

  11. Bornmann, L., & Williams, R. (2017). Can the journal impact factor be used as a criterion for the selection of junior researchers? A large-scale empirical study based on researcher ID data. Journal of Informetrics, 11(3), 788–799.

  12. Braun, T., Glänzel, W., & Schubert, A. (2005). A Hirsch-type index for journals. The Scientist, 19(22), 8.

  13. Braun, T., Glänzel, W., & Schubert, A. (2006). A Hirsch-type index for journals. Scientometrics, 69(1), 169–173.

  14. Brembs, B., Button, K., & Munafò, M. (2013). Deep impact: Unintended consequences of journal rank. Frontiers in Human Neuroscience, 7, 291.

  15. Cameron, B. F. (2005). Trends in the usage of ISI bibliometric data: Uses, abuses, and implications. Portal: Libraries and the Academy, 5(1), 105–125.

  16. Campanario, J. M. (2018). Are leaders really leading? Journals that are first in web of science subject categories in the context of their groups. Scientometrics, 115(1), 111–130.

  17. Chi, P. S., & Glänzel, W. (2018). Comparison of citation and usage indicators in research assessment in scientific disciplines and journals. Scientometrics, 116(1), 537–554.

  18. Crookes, P. A., Reis, S. L., & Jones, S. C. (2010). The development of a ranking tool for refereed journals in which nursing and midwifery researchers publish their work. Nurse Education Today, 30(5), 420–427.

  19. da Silva, J. A. T. (2017). Does China need to rethink its metrics-and citation-based research rewards policies? Scientometrics, 112(3), 1853–1857.

  20. de Araujo, C. G. S., & Sardinha, A. (2011). H-index of the citing articles: A contribution to the evaluation of scientific production of experienced researchers. Revista Brasileira de Medicina do Esporte, 17(5), 358–362.

  21. Dempsey, J. A. (2009). Impact factor and its role in academic promotion: A statement adopted by the international Respiratory Journal Editors Roundtable. Journal of Applied Physiology, 107(4), 1005.

  22. Dorta-González, P., & Dorta-González, M. I. (2013). Impact maturity times and citation time windows: The 2-year maximum journal impact factor. Journal of Informetrics, 7(3), 593–602.

  23. Editors. (2006). The impact factor game. It is time to find a better way to assess the scientific literature. PLoS Medicine, 3(6), 707–708.

  24. Epstein, W. M. (2004). Confirmational response bias and the quality of the editorial processes among American social work journals. Research on Social Work Practice, 14(6), 450–458.

  25. Eyre-Walker, A., & Stoletzki, N. (2013). The assessment of science: The relative merits of post-publication review, the impact factor, and the number of citations. PLoS Biology, 11(10), e1001675.

  26. Falagas, M. E., Kouranos, V. D., Arencibia-Jorge, R., & Karageorgopoulos, D. E. (2008). Comparison of SCImago journal rank indicator with journal impact factor. Faseb Journal, 22(8), 2623–2628.

  27. Fiala, J., Mares, J. J., & Sestak, J. (2017). Reflections on how to evaluate the professional value of scientific papers and their corresponding citations. Scientometrics, 112(1), 697–709.

  28. Fischer, I., & Steiger, H. J. (2018). Dynamics of journal impact factors and limits to their inflation. Journal of Scholarly Publishing, 50(1), 26–36.

  29. Fuyono, I., & Cyranoski, D. (2006). Cash for papers: Putting a premium on publication. Nature, 441(7095), 792.

  30. Garfield, E. (1955). Citation indexes to science: A new dimension in documentation through association of ideas. Science, 122, 108–111.

  31. Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science, 178(4060), 471–479.

  32. Garfield, E. (1996). How can impact factors be improved? British Medical Journal, 313(7054), 411–413.

  33. Garfield, E. (2001). Impact factors, and why they won’t go away. Nature, 411(6837), 522.

  34. Glänzel, W. (2008). On some new bibliometric applications of statistics related to the h-index. Scientometrics, 77(1), 187–196.

  35. Glänzel, W. (2009). The multi-dimensionality of journal impact. Scientometrics, 78(2), 355–374.

  36. Glänzel, W. (2010a). The role of the h-index and the characteristic scores and scales in testing the tail properties of scientometric distributions. Scientometrics, 83(3), 697–709.

  37. Glänzel, W. (2010b). On reliability and robustness of scientometrics indicators based on stochastic models. An evidence-based opinion paper. Journal of Informetrics, 4(3), 313–319.

  38. Glänzel, W. (2011). The application of characteristic scores and scales to the evaluation and ranking of scientific journals. Journal of Information Science, 37(1), 40–48.

  39. Glänzel, W., & Moed, H. F. (2002). Journal impact measures in bibliometric research. Scientometrics, 53(2), 171–193.

  40. Glänzel, W., & Schubert, A. (2010). Hirsch-type characteristics of the tail of distributions. The genralised h-index. Journal of Informetrics, 4(1), 118–123.

  41. Glänzel, W., & Thijs, B. (2018). The role of baseline granularity for benchmarking citation impact. The case of CSS profiles. Scientometrics, 116(1), 521–536.

  42. Goncalves, R. R., Kieling, C., Bressan, R. A., Mari, J. J., & Rohde, L. A. (2009). The evaluation of scientific productivity in Brazil: An assessment of the mental health field. Scientometrics, 80(2), 529–537.

  43. Ha, T. C., Tan, S. B., & Soo, K. C. (2006). The journal impact factor: Too much of an impact? Annals of the Academy of Medicine, Singapore, 35(12), 911–916.

  44. Haghdoost, A., Zare, M., & Bazrafshan, A. (2014). How variable are the journal impact measures? Online Information Review, 38(6), 723–737.

  45. Hartley, J. (2012). To cite or not to cite: Author self-citations and the impact factor. Scientometrics, 92(2), 313–317.

  46. Harzing, A. W., & van der Wal, R. (2009). A Google Scholar h-index for journals: An alternative metric to measure journal impact in economics and business. Journal of the American Society for Information Science and Technology, 60(1), 41–46.

  47. Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16569–16572.

  48. Hsu, W. C., Tasi, C. F., & Li, J. H. (2015). A hybrid indicator for journal ranking: An example from the field of health care sciences and services. Online Information Review, 39(7), 858–869.

  49. Hunt, G. E., Cleary, M., & Walter, G. (2010). Psychiatry and the Hirsch h-index: The relationship between journal impact factors and accrued citations. Harvard Review of Psychiatry, 18(4), 207–219.

  50. Iglesias, J. E., & Pecharroman, C. (2007). Scaling the h-index for different scientific ISI fields. Scientometrics, 73(3), 303–320.

  51. Jacob, J. H., Lehrl, S., & Henkel, A. W. (2007). Early recognition of high quality researchers of the German psychiatry by worldwide accessible bibliometric indicators. Scientometrics, 73(2), 117–130.

  52. Jung, H., Seo, I., Kim, J., & Kim, B. K. (2017). Factors affecting government-funded research quality. Asian Journal of Technology Innovation, 25(3), 447–467.

  53. Kumaran, M., & Ha, C. (2017). Knowledge of journal impact factors among nursing faculty: A cross-sectional study. Journal of the Medical Library Association, 105(2), 140–144.

  54. Leydesdorff, L. (2008). Caveats for the use of citation indicators in research and journal evaluations. Journal of the American Society for Information Science and Technology, 59(2), 278–287.

  55. Leydesdorff, L. (2012). Alternatives to the journal impact factor: 13 and the top-10% (or top-25%) of the most-highly cited papers. Scientometrics, 92(2), 355–365.

  56. Leydesdorff, L. (2013). An evaluation of impacts in “Nanoscience and nanotechnology”: Steps towards standards for citation analysis. Scientometrics, 94(1), 35–55.

  57. Lindner, M. D., Torralba, K. D., & Khan, N. A. (2018). Scientific productivity: An exploratory study of metrics and incentives. PLoS ONE, 13(4), e0195321.

  58. Malesios, C. (2016). Measuring the robustness of the journal h-index with respect to publication and citation values: A Bayesian sensitivity analysis. Journal of Informetrics, 10(3), 719–731.

  59. McGarty, C. (2000). The citation impact factor in social psychology: A bad statistic that encourages bad science? Current Research in Social Psychology, 5(1), 1–16.

  60. McVeigh, M. E., & Mann, S. J. (2009). The journal impact factor denominator: Defining citable (counted) items. JAMA-Journal of the American Medical Association, 302(10), 1107–1109.

  61. Mimouni, M., Ratmansky, M., Sacher, Y., Aharoni, S., & Mimouni-Bloch, A. (2016). Self-citation rate and impact factor in pediatrics. Scientometrics, 108(3), 1455–1460.

  62. Mingers, J., Macri, F., & Petrovici, D. (2012). Using the h-index to measure the quality of journals in the field of business and management. Information Processing and Management, 48(2), 234–241.

  63. Mirnezami, S. R., Beaudry, C., & Lariviere, V. (2016). What determines researchers’ scientific impact? A case study of Quebec researchers. Science and Publicpolicy, 43(2), 262–274.

  64. Olden, J. D. (2007). How do ecological journals stack-up? Ranking of scientific quality according to the h-index. Ecoscience, 14(3), 370–376.

  65. Opthof, T. (1997). Sense and nonsense about the impact factor. Cardiovascular Research, 33(1), 1–7.

  66. Packalen, M., & Bhattacharya, J. (2017). Neophilia ranking of scientific journals. Scientometrics, 110(1), 43–64.

  67. Polit, D. F., & Northam, S. (2011). Impact factors in nursing journals. Nursing Outlook, 59(1), 18–28.

  68. Prathap, G., Mini, S., & Nishy, P. (2016). Does high impact factor successfully predict future citations? An analysis using Peirce’s measure. Scientometrics, 108(3), 1043–1047.

  69. Prozesky, H., & Boshoff, N. (2012). Bibilometrics as a tool for measuring gender-specific research performance: An example from South African invasion ecology. Scientometrics, 90(2), 383–406.

  70. Racz, A., & Markovic, S. (2018). “Worth(less) papers”-are journal impact factor and number of citations suitable indicators to evaluate quality of scientists? Nova Prisutnost, 16(2), 369–389.

  71. Romero-Torres, M., Acosta-Moreno, L. A., & Tejada-Gomez, M. A. (2013). Use of the h index to rank scientific Latin American journals: Colombia a case study. Revista Espanola de Documentacion Cientifica, 36(1), e003.

  72. Rousseau, R. (2016). Positive correlation between journal production and journal impact factors. Journal of Informetrics, 10(2), 567–568.

  73. Saad, G. (2006). Exploring the h-index at the author and journal levels using bibliometric data of productive consumer scholars and business-related journals respectively. Scientometrics, 69(1), 117–120.

  74. Sangwal, K. (2013). Some citation-related characteristics of scientific journals published in individual countries. Scientometrics, 97(3), 719–741.

  75. Schubert, A., & Glänzel, W. (2007). A systematic analysis of Hirsch-type indices for journals. Journal of Informetrics, 1(3), 179–184.

  76. Sebire, N. J. (2008). H-index and impact factors: Assessing the clinical impact of researchers and specialist journals. Ultrasound in Obstetrics and Gynecology, 32(7), 843–845.

  77. Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314, 498–502.

  78. Shen, Z. S., Yang, L. Y., & Wu, J. S. (2018). Lognormal distribution of citation counts is the reason for the relation between Impact Factors and Citation Success Index. Journal of Informetrics, 12(1), 153–157.

  79. Smith, K. M., Crookes, E., & Crookes, P. A. (2013). Measuring research ‘impact’ for academic promotion: Issues from the literature. Journal of Higher Education Policy and Management, 35(4), 410–420.

  80. Solarino, S. (2012). Impact factor, citation index, h-index: Are researchers still free to choose where and how to publish their results? Annals of Geophysics, 55(3), 473–477.

  81. Sternberg, R. J. (2018). Evaluating merit among scientists. Journal of Applied Research in Memory and Cognition, 7(2), 209–216.

  82. Tahamtan, I., Afshar, A. S., & Ahamdzadeh, K. (2016). Factors affecting number of citations: A comprehensive review of the literature. Scientometrics, 107(3), 1195–1225.

  83. Valderrama, P., Escabias, M., Jimenez-Contreras, E., Rodriguez-Archilla, A., & Valderrama, M. J. (2018). Proposal of a stochastic model to determine the bibliometric variables influencing the quality of a journal: Application to the field of Dentistry. Scientometrics, 115(2), 1087–1095.

  84. Van Raan, A. F. J. (2012). Properties of journal impact in relation to bibliometric research group performance indicators. Scientometrics, 92(2), 457–469.

  85. Vanclay, J. K. (2008). Ranking forestry journals using the h-index. Journal of Informetrics, 2(4), 326–334.

  86. Vinkler, P. (2001). An attempt for defining some basic categories of scientometrics and classifying the indicators of evaluative scientometrics. Scientometrics, 50(3), 539–544.

  87. Vinkler, P. (2002). The institutionalization of scientific information. The scientometric model (ISI-S MODEL). Library Trends, 50(3), 553–569.

  88. Vinkler, P. (2004). Characterization of the impact of sets of scientific papers: The garfield (impact) factor. Journal of the American Society for Information Science and Technology, 55(5), 431–435.

  89. Vinkler, P. (2009a). Introducing the Current Contribution Index for characterizing the recent, relevant impact of journals. Scientometrics, 79(2), 409–420.

  90. Vinkler, P. (2009b). The π-index: A new indicator for assessing scientific impact. Journal of Information Science, 35(5), 602–612.

  91. Vinkler, P. (2010). The π v-index: A new indicator to characterize the impact of journals. Scientometrics, 82(3), 461–475.

  92. Vinkler, P. (2012). The Garfield impact factor, one of the fundamental indicators in scientometrics. Scientometrics, 92(2), 471–483.

  93. Vinkler, P. (2017). The size and impact of the elite set of publications in scientometric assessments. Scientometrics, 110(1), 163–177.

  94. Vinluan, L. R. (2012). Research productivity in education and psychology in the Philippines and comparison with ASEAN countries. Scientometrics, 91(1), 277–294.

  95. Xia, J. F., & Smith, M. P. (2018). Alternative journal impact factors in open access publishing. Learned Publishing, 31(4), 403–411.

  96. Yan, E. J., Wu, C. J., & Song, M. (2018). The funding factor: A cross-disciplinary examination of the association between research funding and citation impact. Scientometrics, 115(1), 369–384.

  97. Yang, D. H., Li, X., Sun, X. X., & Wan, J. (2016). Detecting impact factor manipulation with data mining techniques. Scientometrics, 109(3), 1989–2005.

  98. Yeung, A. W. K. (2019). Higher impact factor of neuroimaging journals is associated with larger number of articles published and smaller percentage of uncited articles. Scientometrics, 12, 523.

  99. Yu, G., & Li, Y. J. (2007). Parameter identification of the observed citation distribution. Scientometrics, 71(2), 339–348.

  100. Yu, G., & Li, Y. J. (2010). Identification of referencing and citation processes of scientific journals based on the citation distribution model. Scientometrics, 82(2), 249–261.

  101. Yu, G., Wang, X. H., & Yu, D. R. (2005). The influence of publication delays on impact factors. Scientometrics, 64(2), 235–246.

  102. Yuen, J. (2018). Comparision of impact factor, eigenfactor metrics, and scimago journal rank indicator and h-index for neurosurgical and spinal surgical journals. World Neurosurgery, 119, E328–E337.

  103. Yuret, T. (2018). Author-weighted impact factor and reference return ratio: Can we attain more equality among fields? Scientometrics, 116(3), 2097–2111.

  104. Zhang, L., Thijs, B., & Glänzel, W. (2011). The diffusion of H-related literature. Journal of Informetrics, 5(4), 583–593.

  105. Zou, Y. W., & Laubichler, M. D. (2017). Measuring the contributions of Chinese scholars to the research field of systems biology from 2005 to 2013. Scientometrics, 110(3), 1615–1631.

Download references

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Grant No. 71473034), the Heilongjiang Provincial Natural Science Foundation of China (Grant No. LH2019G001), and the financial assistance from Postdoctoral Scientific Research Developmental Fund of Heilongjiang Province (Grant No. LBH-Q16003).

Author information

Correspondence to Guangsheng Chen.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 773 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Wang, M., Jiao, S., Chai, K. et al. Building journal’s long-term impact: using indicators detected from the sustained active articles. Scientometrics 121, 261–283 (2019). https://doi.org/10.1007/s11192-019-03196-8

Download citation

Keywords

  • Impact evaluation
  • Long-term impact
  • Active articles
  • h 1-Index
  • Impact factor