Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Source normalized indicators of citation impact: an overview of different approaches and an empirical comparison

Abstract

Different scientific fields have different citation practices. Citation-based bibliometric indicators need to normalize for such differences between fields in order to allow for meaningful between-field comparisons of citation impact. Traditionally, normalization for field differences has usually been done based on a field classification system. In this approach, each publication belongs to one or more fields and the citation impact of a publication is calculated relative to the other publications in the same field. Recently, the idea of source normalization was introduced, which offers an alternative approach to normalize for field differences. In this approach, normalization is done by looking at the referencing behavior of citing publications or citing journals. In this paper, we provide an overview of a number of source normalization approaches and we empirically compare these approaches with a traditional normalization approach based on a field classification system. We also pay attention to the issue of the selection of the journals to be included in a normalization for field differences. Our analysis indicates a number of problems of the traditional classification-system-based normalization approach, suggesting that source normalization approaches may yield more accurate results.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Notes

  1. 1.

    Differences in citation density between fields may also be caused by unidirectional citation flows between fields (e.g., from applied fields to more basic fields) and by differences between fields in the growth rate of the literature. The source normalization approach does not correct for these effects (e.g., Zitt and Small 2008).

  2. 2.

    WoS covers a substantial number of trade magazines. Examples of some of the larger ones are Genetic Engineering & Biotechnology News, Naval Architect, and Professional Engineering. Popular magazines covered by WoS include, among others, the scientific magazines American Scientist, New Scientist, and Scientific American and the business magazines Forbes and Fortune.

  3. 3.

    In the case of the Netherlands, WoS for instance covers the Dutch language journals Psychologie & Gezondheid, Tijdschrift voor Communicatiewetenschap, and Tijdschrift voor Diergeneeskunde as well as the English language journals Economist-Netherlands, Netherlands Heart Journal, and Netherlands Journal of Medicine.

  4. 4.

    In the case of a journal that is assigned to multiple fields in a field classification system, e i is calculated as the harmonic average of the expected numbers of citations obtained for the different fields. For a justification of this approach, we refer to Waltman et al. (2011).

  5. 5.

    In a somewhat different context, the idea of fractional citation counting was already suggested by Small and Sweeney (1985).

  6. 6.

    Counting all references in a citing publication instead of only active references disadvantages fields with a relatively large number of references to older publications and to publications in journals not covered by one’s database.

  7. 7.

    The full results of our analysis are available online at www.ludowaltman.nl/normalization/.

  8. 8.

    A similar conclusion is reached by Radicchi and Castellano (2012a). However, there is a fundamental difference between our analysis and the one by Radicchi and Castellano. Radicchi and Castellano apply fractional citation counting in the way it was originally proposed by Leydesdorff and Opthof (2010), which means that fractioning is done based on the total number of references in a citing publication. Instead of the total number of references, we look at the number of active references in a citing publication (cf. Leydesdorff et al. in press). Our analysis makes clear that taking into account only active references does not solve the problems of the fractional citation counting approach.

  9. 9.

    This problem is also discussed by Glänzel et al. (1999). As a solution, these authors propose to treat journals with a broad scope in a special way. In their proposal, publications in journals with a broad scope are assigned to fields based on their references.

  10. 10.

    This the following publication: Sheldrick, G.M. (2008). A short history of SHELX. Acta Crystallographica Section A, 64(1), 112–122. By the end of 2011, this publication had been cited almost 25,000 times.

  11. 11.

    Recent studies on classification-system-based normalization approaches focus on identifying general patterns in the citation distributions of scientific fields (e.g., Crespo et al. 2012; Radicchi and Castellano 2012b; Radicchi et al. 2008). These studies usually do not exclude any journals. It seems likely that the results of these studies depend quite significantly on whether trade journals, popular magazines, and other special journals are included or excluded.

References

  1. Adams, J., Gurney, K., & Jackson, L. (2008). Calibrating the zoom—A test of Zitt’s hypothesis. Scientometrics, 75(1), 81–95.

  2. Braun, T., & Glänzel, W. (1990). United Germany: the new scientific superpower? Scientometrics, 19(5–6), 513–521.

  3. Buela-Casal, G., Perakakis, P., Taylor, M., & Checa, P. (2006). Measuring internationality: reflections and perspectives on academic journals. Scientometrics, 67(1), 45–65.

  4. Glänzel, W., Schubert, A., & Czerwon, H.-J. (1999). An item-by-item subject classification of papers published in multidisciplinary and general journals using reference analysis. Scientometrics, 44(3), 427–439.

  5. Glänzel, W., Schubert, A., Thijs, B., & Debackere, K. (2011). A priori vs. a posteriori normalisation of citation indicators. The case of journal ranking. Scientometrics, 87(2), 415–424.

  6. Glänzel, W., Thijs, B., Schubert, A., & Debackere, K. (2009). Subfield-specific normalized relative indicators and a new generation of relational charts: methodological foundations illustrated on the assessment of institutional research performance. Scientometrics, 78(1), 165–188.

  7. Leydesdorff, L., & Bornmann, L. (2011). How fractional counting of citations affects the impact factor: normalization in terms of differences in citation potentials among fields of science. J Am Soc Inf Sci Technol, 62(2), 217–229.

  8. Leydesdorff, L., & Opthof, T. (2010). Scopus’s source normalized impact per paper (SNIP) versus a journal impact factor based on fractional counting of citations. J Am Soc Inf Sci Technol, 61(11), 2365–2369.

  9. Leydesdorff, L., Zhou, P., & Bornmann, L. (in press). How can journal impact factors be normalized across fields of science? An assessment in terms of percentile ranks and fractional counts. J Am Soc Inf Sci Technol.

  10. Crespo, J. A., Li, Y., & Ruiz-Castillo, J. (2012). Differences in citation impact across scientific fields (Working Paper Economic Series 12-06). Departamento de Economía, Universidad Carlos III of Madrid.

  11. Lundberg, J. (2007). Lifting the crown—citation z-score. J Informetr, 1(2), 145–154.

  12. Moed, H. F. (2010). Measuring contextual citation impact of scientific journals. J Informetr, 4(3), 265–277.

  13. Moed, H. F., De Bruin, R. E., & Van Leeuwen, T. N. (1995). New bibliometric tools for the assessment of national research performance: database description, overview of indicators and first applications. Scientometrics, 33(3), 381–422.

  14. Neuhaus, C., & Daniel, H.-D. (2009). A new reference standard for citation analysis in chemistry and related fields based on the sections of chemical abstracts. Scientometrics, 78(2), 219–229.

  15. Radicchi, F., & Castellano, C. (2012a). Testing the fairness of citation indicators for comparison across scientific domains: the case of fractional citation counts. J Informetr, 6(1), 121–130.

  16. Radicchi, F., & Castellano, C. (2012b). A reverse engineering approach to the suppression of citation biases reveals universal properties of citation distributions. PLoS One, 7(3), e33833.

  17. Radicchi, F., Fortunato, S., & Castellano, C. (2008). Universality of citation distributions: toward an objective measure of scientific impact. Proceedings of the National Academy of Sciences, 105(45), 17268–17272.

  18. Schubert, A., & Braun, T. (1996). Cross-field normalization of scientometric indicators. Scientometrics, 36(3), 311–324.

  19. Small, H., & Sweeney, E. (1985). Clustering the science citation index using co-citations. I. A comparison of methods. Scientometrics, 7(3–6), 391–409.

  20. Van Eck, N. J., Waltman, L., Van Raan, A. F. J., Klautz, R. J. M., & Peul, W. C. (2012). Citation analysis may severely underestimate the impact of clinical research as compared to basic research. arXiv:1210.0442.

  21. Van Leeuwen, T. N., & Calero Medina, C. (2012). Redefining the field of economics: improving field normalization for the application of bibliometric techniques in the field of economics. Res Eval, 21(1), 61–70.

  22. Van Leeuwen, T. N., Moed, H. F., Tijssen, R. J. W., Visser, M. S., & Van Raan, A. F. J. (2001). Language biases in the coverage of the Science Citation Index and its consequences for international comparisons of national research performance. Scientometrics, 51(1), 335–346.

  23. Van Raan, A. F. J., Van Leeuwen, T. N., & Visser, M. S. (2011a). Severe language effect in university rankings: particularly Germany and France are wronged in citation-based rankings. Scientometrics, 88(2), 495–498.

  24. Van Raan, T., Van Leeuwen, T., & Visser, M. (2011b). Non-English papers decrease rankings. Nature, 469, 34.

  25. Waltman, L., & Van Eck, N. J. (2010a). A general source normalized approach to bibliometric research performance assessment. In Book of Abstracts of the Eleventh International Conference on Science and Technology Indicators (pp. 298–299).

  26. Waltman, L., & Van Eck, N. J. (2010b). The relation between Eigenfactor, audience factor, and influence weight. J Am Soc Inf Sci Technol, 61(7), 1476–1486.

  27. Waltman, L., & Van Eck, N. J. (in press). A new methodology for constructing a publication-level classification system of science. J Am Soc Inf Sci Technol.

  28. Waltman, L., Van Eck, N. J., Van Leeuwen, T. N., & Visser, M. S. (2012). Some modifications to the SNIP journal impact indicator. arXiv:1209.0785.

  29. Waltman, L., Van Eck, N. J., Van Leeuwen, T. N., Visser, M. S., & Van Raan, A. F. J. (2011a). Towards a new crown indicator: some theoretical considerations. J Informetr, 5(1), 37–47.

  30. Waltman, L., Yan, E., & Van Eck, N. J. (2011b). A recursive field-normalized bibliometric performance indicator: an application to the field of library and information science. Scientometrics, 89(1), 301–314.

  31. Zhou, P., & Leydesdorff, L. (2011). Fractional counting of citations in research evaluation: a cross- and interdisciplinary assessment of the Tsinghua University in Beijing. J Informetr, 5(3), 360–368.

  32. Zitt, M. (2010). Citing-side normalization of journal impact: a robust variant of the audience factor. J Informetr, 4(3), 392–406.

  33. Zitt, M. (2011). Behind citing-side normalization of citations: some properties of the journal impact factor. Scientometrics, 89(1), 329–344.

  34. Zitt, M., & Bassecoulard, E. (1998). Internationalization of scientific journals: a measurement based on publication and citation scope. Scientometrics, 41(1–2), 255–271.

  35. Zitt, M., Ramanana-Rahary, S., & Bassecoulard, E. (2003). Correcting glasses help fair comparisons in international science landscape: country indicators as a function of ISI database delineation. Scientometrics, 56(2), 259–282.

  36. Zitt, M., Ramanana-Rahary, S., & Bassecoulard, E. (2005). Relativity of citation performance and excellence measures: from cross-field to cross-scale effects of field-normalisation. Scientometrics, 63(2), 373–401.

  37. Zitt, M., & Small, H. (2008). Modifying the journal impact factor by fractional citation weighting: the audience factor. J Am Soc Inf Sci Technol, 59(11), 1856–1860.

Download references

Acknowledgments

We would like to thank Javier Ruiz Castillo for his comments on an earlier draft of this paper. We are also grateful to an anonymous referee for various useful comments.

Author information

Correspondence to Ludo Waltman.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Waltman, L., van Eck, N.J. Source normalized indicators of citation impact: an overview of different approaches and an empirical comparison. Scientometrics 96, 699–716 (2013). https://doi.org/10.1007/s11192-012-0913-4

Download citation

Keywords

  • Bibliometric indicator
  • Citation analysis
  • Field normalization
  • Source normalization