Skip to main content
SpringerLink
Log in

Source normalized indicators of citation impact: an overview of different approaches and an empirical comparison

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Different scientific fields have different citation practices. Citation-based bibliometric indicators need to normalize for such differences between fields in order to allow for meaningful between-field comparisons of citation impact. Traditionally, normalization for field differences has usually been done based on a field classification system. In this approach, each publication belongs to one or more fields and the citation impact of a publication is calculated relative to the other publications in the same field. Recently, the idea of source normalization was introduced, which offers an alternative approach to normalize for field differences. In this approach, normalization is done by looking at the referencing behavior of citing publications or citing journals. In this paper, we provide an overview of a number of source normalization approaches and we empirically compare these approaches with a traditional normalization approach based on a field classification system. We also pay attention to the issue of the selection of the journals to be included in a normalization for field differences. Our analysis indicates a number of problems of the traditional classification-system-based normalization approach, suggesting that source normalization approaches may yield more accurate results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. Differences in citation density between fields may also be caused by unidirectional citation flows between fields (e.g., from applied fields to more basic fields) and by differences between fields in the growth rate of the literature. The source normalization approach does not correct for these effects (e.g., Zitt and Small 2008).

  2. WoS covers a substantial number of trade magazines. Examples of some of the larger ones are Genetic Engineering & Biotechnology News, Naval Architect, and Professional Engineering. Popular magazines covered by WoS include, among others, the scientific magazines American Scientist, New Scientist, and Scientific American and the business magazines Forbes and Fortune.

  3. In the case of the Netherlands, WoS for instance covers the Dutch language journals Psychologie & Gezondheid, Tijdschrift voor Communicatiewetenschap, and Tijdschrift voor Diergeneeskunde as well as the English language journals Economist-Netherlands, Netherlands Heart Journal, and Netherlands Journal of Medicine.

  4. In the case of a journal that is assigned to multiple fields in a field classification system, e i is calculated as the harmonic average of the expected numbers of citations obtained for the different fields. For a justification of this approach, we refer to Waltman et al. (2011).

  5. In a somewhat different context, the idea of fractional citation counting was already suggested by Small and Sweeney (1985).

  6. Counting all references in a citing publication instead of only active references disadvantages fields with a relatively large number of references to older publications and to publications in journals not covered by one’s database.

  7. The full results of our analysis are available online at www.ludowaltman.nl/normalization/.

  8. A similar conclusion is reached by Radicchi and Castellano (2012a). However, there is a fundamental difference between our analysis and the one by Radicchi and Castellano. Radicchi and Castellano apply fractional citation counting in the way it was originally proposed by Leydesdorff and Opthof (2010), which means that fractioning is done based on the total number of references in a citing publication. Instead of the total number of references, we look at the number of active references in a citing publication (cf. Leydesdorff et al. in press). Our analysis makes clear that taking into account only active references does not solve the problems of the fractional citation counting approach.

  9. This problem is also discussed by Glänzel et al. (1999). As a solution, these authors propose to treat journals with a broad scope in a special way. In their proposal, publications in journals with a broad scope are assigned to fields based on their references.

  10. This the following publication: Sheldrick, G.M. (2008). A short history of SHELX. Acta Crystallographica Section A, 64(1), 112–122. By the end of 2011, this publication had been cited almost 25,000 times.

  11. Recent studies on classification-system-based normalization approaches focus on identifying general patterns in the citation distributions of scientific fields (e.g., Crespo et al. 2012; Radicchi and Castellano 2012b; Radicchi et al. 2008). These studies usually do not exclude any journals. It seems likely that the results of these studies depend quite significantly on whether trade journals, popular magazines, and other special journals are included or excluded.

References

  • Adams, J., Gurney, K., & Jackson, L. (2008). Calibrating the zoom—A test of Zitt’s hypothesis. Scientometrics, 75(1), 81–95.

    Article  Google Scholar 

  • Braun, T., & Glänzel, W. (1990). United Germany: the new scientific superpower? Scientometrics, 19(5–6), 513–521.

    Article  Google Scholar 

  • Buela-Casal, G., Perakakis, P., Taylor, M., & Checa, P. (2006). Measuring internationality: reflections and perspectives on academic journals. Scientometrics, 67(1), 45–65.

    Article  Google Scholar 

  • Glänzel, W., Schubert, A., & Czerwon, H.-J. (1999). An item-by-item subject classification of papers published in multidisciplinary and general journals using reference analysis. Scientometrics, 44(3), 427–439.

    Article  Google Scholar 

  • Glänzel, W., Schubert, A., Thijs, B., & Debackere, K. (2011). A priori vs. a posteriori normalisation of citation indicators. The case of journal ranking. Scientometrics, 87(2), 415–424.

    Article  Google Scholar 

  • Glänzel, W., Thijs, B., Schubert, A., & Debackere, K. (2009). Subfield-specific normalized relative indicators and a new generation of relational charts: methodological foundations illustrated on the assessment of institutional research performance. Scientometrics, 78(1), 165–188.

    Article  Google Scholar 

  • Leydesdorff, L., & Bornmann, L. (2011). How fractional counting of citations affects the impact factor: normalization in terms of differences in citation potentials among fields of science. J Am Soc Inf Sci Technol, 62(2), 217–229.

    Article  Google Scholar 

  • Leydesdorff, L., & Opthof, T. (2010). Scopus’s source normalized impact per paper (SNIP) versus a journal impact factor based on fractional counting of citations. J Am Soc Inf Sci Technol, 61(11), 2365–2369.

    Article  Google Scholar 

  • Leydesdorff, L., Zhou, P., & Bornmann, L. (in press). How can journal impact factors be normalized across fields of science? An assessment in terms of percentile ranks and fractional counts. J Am Soc Inf Sci Technol.

  • Crespo, J. A., Li, Y., & Ruiz-Castillo, J. (2012). Differences in citation impact across scientific fields (Working Paper Economic Series 12-06). Departamento de Economía, Universidad Carlos III of Madrid.

  • Lundberg, J. (2007). Lifting the crown—citation z-score. J Informetr, 1(2), 145–154.

    Article  Google Scholar 

  • Moed, H. F. (2010). Measuring contextual citation impact of scientific journals. J Informetr, 4(3), 265–277.

    Article  Google Scholar 

  • Moed, H. F., De Bruin, R. E., & Van Leeuwen, T. N. (1995). New bibliometric tools for the assessment of national research performance: database description, overview of indicators and first applications. Scientometrics, 33(3), 381–422.

    Article  Google Scholar 

  • Neuhaus, C., & Daniel, H.-D. (2009). A new reference standard for citation analysis in chemistry and related fields based on the sections of chemical abstracts. Scientometrics, 78(2), 219–229.

    Article  Google Scholar 

  • Radicchi, F., & Castellano, C. (2012a). Testing the fairness of citation indicators for comparison across scientific domains: the case of fractional citation counts. J Informetr, 6(1), 121–130.

    Article  Google Scholar 

  • Radicchi, F., & Castellano, C. (2012b). A reverse engineering approach to the suppression of citation biases reveals universal properties of citation distributions. PLoS One, 7(3), e33833.

    Article  Google Scholar 

  • Radicchi, F., Fortunato, S., & Castellano, C. (2008). Universality of citation distributions: toward an objective measure of scientific impact. Proceedings of the National Academy of Sciences, 105(45), 17268–17272.

    Article  Google Scholar 

  • Schubert, A., & Braun, T. (1996). Cross-field normalization of scientometric indicators. Scientometrics, 36(3), 311–324.

    Article  Google Scholar 

  • Small, H., & Sweeney, E. (1985). Clustering the science citation index using co-citations. I. A comparison of methods. Scientometrics, 7(3–6), 391–409.

    Article  Google Scholar 

  • Van Eck, N. J., Waltman, L., Van Raan, A. F. J., Klautz, R. J. M., & Peul, W. C. (2012). Citation analysis may severely underestimate the impact of clinical research as compared to basic research. arXiv:1210.0442.

  • Van Leeuwen, T. N., & Calero Medina, C. (2012). Redefining the field of economics: improving field normalization for the application of bibliometric techniques in the field of economics. Res Eval, 21(1), 61–70.

    Article  Google Scholar 

  • Van Leeuwen, T. N., Moed, H. F., Tijssen, R. J. W., Visser, M. S., & Van Raan, A. F. J. (2001). Language biases in the coverage of the Science Citation Index and its consequences for international comparisons of national research performance. Scientometrics, 51(1), 335–346.

    Article  Google Scholar 

  • Van Raan, A. F. J., Van Leeuwen, T. N., & Visser, M. S. (2011a). Severe language effect in university rankings: particularly Germany and France are wronged in citation-based rankings. Scientometrics, 88(2), 495–498.

    Article  Google Scholar 

  • Van Raan, T., Van Leeuwen, T., & Visser, M. (2011b). Non-English papers decrease rankings. Nature, 469, 34.

    Article  Google Scholar 

  • Waltman, L., & Van Eck, N. J. (2010a). A general source normalized approach to bibliometric research performance assessment. In Book of Abstracts of the Eleventh International Conference on Science and Technology Indicators (pp. 298–299).

  • Waltman, L., & Van Eck, N. J. (2010b). The relation between Eigenfactor, audience factor, and influence weight. J Am Soc Inf Sci Technol, 61(7), 1476–1486.

    Article  Google Scholar 

  • Waltman, L., & Van Eck, N. J. (in press). A new methodology for constructing a publication-level classification system of science. J Am Soc Inf Sci Technol.

  • Waltman, L., Van Eck, N. J., Van Leeuwen, T. N., & Visser, M. S. (2012). Some modifications to the SNIP journal impact indicator. arXiv:1209.0785.

  • Waltman, L., Van Eck, N. J., Van Leeuwen, T. N., Visser, M. S., & Van Raan, A. F. J. (2011a). Towards a new crown indicator: some theoretical considerations. J Informetr, 5(1), 37–47.

    Article  Google Scholar 

  • Waltman, L., Yan, E., & Van Eck, N. J. (2011b). A recursive field-normalized bibliometric performance indicator: an application to the field of library and information science. Scientometrics, 89(1), 301–314.

    Article  Google Scholar 

  • Zhou, P., & Leydesdorff, L. (2011). Fractional counting of citations in research evaluation: a cross- and interdisciplinary assessment of the Tsinghua University in Beijing. J Informetr, 5(3), 360–368.

    Article  Google Scholar 

  • Zitt, M. (2010). Citing-side normalization of journal impact: a robust variant of the audience factor. J Informetr, 4(3), 392–406.

    Article  Google Scholar 

  • Zitt, M. (2011). Behind citing-side normalization of citations: some properties of the journal impact factor. Scientometrics, 89(1), 329–344.

    Article  MathSciNet  Google Scholar 

  • Zitt, M., & Bassecoulard, E. (1998). Internationalization of scientific journals: a measurement based on publication and citation scope. Scientometrics, 41(1–2), 255–271.

    Article  Google Scholar 

  • Zitt, M., Ramanana-Rahary, S., & Bassecoulard, E. (2003). Correcting glasses help fair comparisons in international science landscape: country indicators as a function of ISI database delineation. Scientometrics, 56(2), 259–282.

    Article  Google Scholar 

  • Zitt, M., Ramanana-Rahary, S., & Bassecoulard, E. (2005). Relativity of citation performance and excellence measures: from cross-field to cross-scale effects of field-normalisation. Scientometrics, 63(2), 373–401.

    Article  Google Scholar 

  • Zitt, M., & Small, H. (2008). Modifying the journal impact factor by fractional citation weighting: the audience factor. J Am Soc Inf Sci Technol, 59(11), 1856–1860.

    Article  Google Scholar 

Download references

Acknowledgments

We would like to thank Javier Ruiz Castillo for his comments on an earlier draft of this paper. We are also grateful to an anonymous referee for various useful comments.

Author information

Authors and Affiliations

  1. Centre for Science and Technology Studies, Leiden University, Leiden, The Netherlands

    Ludo Waltman & Nees Jan van Eck

Authors
  1. Ludo Waltman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nees Jan van Eck
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ludo Waltman.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Waltman, L., van Eck, N.J. Source normalized indicators of citation impact: an overview of different approaches and an empirical comparison. Scientometrics 96, 699–716 (2013). https://doi.org/10.1007/s11192-012-0913-4

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-012-0913-4

Keywords

Search

Navigation