Abstract
Field normalization is a necessary step in a fair cross-field comparison of citation impact. In practice, mean-based method (m-score) is the most popular method for field normalization. However, considering that mean-based method only utilizes the central tendency of citation distribution in the normalization procedure and dispersion is also a significant characteristic, an open and important issue is whether alternative normalization methods which take both central tendency and variability into account perform better than mean-based method. With the aim of collapsing citation distributions of different fields into a universal distribution, this study compares the normalization effect of m-score and z-score based on 236 Web of Science (WoS) subject categories. The results show that both m-score and z-score have remarkable normalization effect as compared with raw citations, but neither of them can realize the ideal goal of “universality of citation distributions”. The results also suggest that m-score is generally preferable to z-score. The essential cause that m-score has an edge over z-score as a whole has a direct relationship with the characteristics of skewed citation distributions in which case m-score is more applicable than z-score.
Similar content being viewed by others
Notes
The values of mean-based indicator are also called m-score in this paper. The specific meaning of m-score can be clearly inferred from the context.
Mean with zero cited (or uncited) publications refers to average citation rates of publications in the case of including uncited publications. Median with zero cited publication, mean/median without zero cited publications have similar meanings in this paper.
In a small field, especially when z takes small values, even the nearest observed global top z % percentage will deviate from the theoretical z % significantly. This deviation introduces a significant error for small fields which makes top z % method less robust.
Though the MDs of z-score are much smaller than those of raw citations at the first seven top z % sections, MD of z-score at top 80 % is slightly greater than that of raw citations.
In the commonly used mean-based normalization, the reference standard for the citation count of a publication is the average citation rates of the reference set to which the publication under consideration belongs. The operation of averaging implies that the citation counts of publications within a reference set are comparable and additive.
References
Abramo, G., Cicero, T., & D’Angelo, C. A. (2012a). How important is choice of the scaling factor in standardizing citations? Journal of Informetrics, 6(4), 645–654.
Abramo, G., Cicero, T., & D’Angelo, C. A. (2012b). Revisiting the scaling of citations for research evaluation. Journal of Informetrics, 6(4), 470–479.
Albarrán, P., Crespo, J. A., Ortuño, I., & Ruiz-Castillo, J. (2011). The skewness of science in 219 sub-fields and a number of aggregates. Scientometrics, 88(2), 385–397.
Bornmann, L., & Daniel, H.-D. (2009). Universality of citation distributions—A validation of Radicchi et al.’s relative indicator c f = c/c 0 at the micro level using data from chemistry. Journal of the American Society for Information Science and Technology, 60(8), 1664–1670.
Bornmann, L., Mutz, R., Neuhaus, C., & Daniel, H.-D. (2008). Citation counts for research evaluation: Standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics in Science and Enviromental Politics, 8, 93–102.
Braun, T., & Glänzel, W. (1990). United Germany: The new scientific superpower? Scientometrics, 19(5–6), 513–521.
Castellano, C., & Radicchi, F. (2009). On the fairness of using relative indicators for comparing citation performance in different disciplines. Archivum Immunologiae et Therapiae Experimentalis, 57(2), 85–90.
Crespo, J. A., Li, Y., & Ruiz-Castillo, J. (2013). The measurement of the effect on citation inequality of differences in citation practices across scientific fields. PLoS One, 8(3), e58727.
De Bruin, R. E., Kint, A., Luwel, M., & Moed, H. F. (1993). A study of research evaluation and planning: The University of Ghent. Research Evaluation, 3(1), 25–41.
Garfield, E. (1979a). Citation indexing—Its theory and application in Science, Technology and Humanities. New York: Wiley & Sons.
Garfield, E. (1979b). Is citation analysis a legitimate evaluation tool? Scientometrics, 1(4), 359–375.
Glänzel, W., Thijs, B., Schubert, A., & Debackere, K. (2009). Subfield-specific normalized relative indicators and a new generation of relational charts: Methodological foundations illustrated on the assessment of institutional research performance. Scientometrics, 78(1), 165–188.
Katz, J. S. (1999). Bibliometric indicators and the social sciences. Brighton, UK: SPRU, University of Sussex.
Katz, J. S. (2000). Scale-independent indicators and research evaluation. Science and Public Policy, 27(1), 23–36.
Larivière, V., & Gingras, Y. (2011). Average of ratios vs. ratio of averages: An empirical analysis of four levels of aggregation. Journal of Informetrics, 5(3), 392–399.
Lee, G. J. (2010). Assessing publication performance of research units: Extensions through operational research and economic techniques. Scientometrics, 84(3), 717–734.
Li, Y., Radicchi, F., Castellano, C., & Ruiz-Castillo, J. (2013). Quantitative evaluation of alternative field normalization procedures. Journal of Informetrics, 7(3), 746–755.
Lillquist, E., & Green, S. (2010). The discipline dependence of citation statistics. Scientometrics, 84(3), 749–762.
Lundberg, J. (2007). Lifting the crown—citation z-score. Journal of Informetrics, 1(2), 145–154.
McAllister, P. R., Narin, F., & Corrigan, J. G. (1983). Programmatic evaluation and comparison based on standardized citation scores. IEEE Transactions on Engineering Management, 30(4), 205–211.
Moed, H. F. (2010). CWTS crown indicator measures citation impact of a research groups’s publication oeuvre. Journal of Informetrics, 4(3), 436–438.
Moed, H. F., De Bruin, R. E., & van Leeuwen, T. N. (1995). New bibliometric tools for the assessment of national research performance: Database description, overview of indicators and first applications. Scientometrics, 33(3), 381–422.
Narin, F. (1976). Evaluative bibliometrics: The use of publication and citation analysis in the evaluation of scientific activity. Washington DC: National Science Foundation.
Nederhof, A. J. (2006). Bibliometric monitoring of performance in the social sciences and the humanities: A review. Scientometrics, 66(1), 81–100.
Opthof, T., & Leydesdorff, L. (2010). Caveats for the journal and field normalizations in the CWTS (“Leiden”) evaluations of research performance. Journal of Informetrics, 4(3), 423–430.
Radicchi, F., & Castellano, C. (2011). Rescaling citations of publications in physics. Pysical Review E, 83(4), 046116.
Radicchi, F., & Castellano, C. (2012). A reverse engineering approach to the suppression of citation biases reveals universal properties of citation distributions. PLoS One, 7(3), e33833.
Radicchi, F., Fortunato, S., & Castellano, C. (2008). Universality of citation distributions: Toward an objective measure of scientific impact. Proceedings of the National Academy of Sciences of the United States of America, 105(45), 17268–17272.
Schubert, A., & Braun, T. (1986). Relative indicators and relational charts for comparative assessment of publication output and citation impact. Scientometrics, 9(5–6), 281–291.
Schubert, A., & Braun, T. (1996). Cross-field normalization of scientometric indicators. Scientometrics, 36(3), 311–324.
Seglen, P. O. (1992). The skewness of science. Journal of the American Society for Information Science, 43(9), 628–638.
Spaan, J. A. E. (2010). The danger of pseudoscience in informetrics. Journal of Informetrics, 4(3), 439–440.
Thompson, B. (1993). GRE percentile ranks cannot be added or averaged: A position paper exploring the scaling characteristics of percentile ranks, and the ethical and legal culpabilities created by adding percentile ranks in making “high-stakes” admission decisions. Paper presented at the Annual Meeting of the Mid-South Educational Research Association, New Orleans, LA.
van Raan, A. F. J., van Eck, N. J., van Leeuwen, T. N., Visser, M. S., & Waltman, L. (2010a). The new set of bibliometric indicators of CWTS. Paper presented at the Eleventh International Conference on Science and Technology Indicators, Leiden, The Netherlands.
van Raan, A. F. J., van Leeuwen, T. N., Visser, M. S., van Eck, N. J., & Waltman, L. (2010b). Rivals for the crown: Reply to Opthof and Leydesdorff. Journal of Informetrics, 4(3), 431–435.
Vinkler, P. (2012). The case of scientometricians with the “absolute relative” impact indicator. Journal of Informetrics, 6(2), 254–264.
Waltman, L., van Eck, N. J., van Leeuwen, T. N., Visser, M. S., & van Raan, A. F. J. (2011). Towards a new crown indicator: Some theoretical considerations. Journal of Informetrics, 5(1), 37–47.
Waltman, L., van Eck, N. J., & van Raan, A. F. J. (2012). Universality of citation distributions revisited. Journal of the American Society for Information Science and Technology, 63(1), 72–77.
Acknowledgments
This study is supported by a grant from the National Education Sciences Planning program during the 12th Five-Year period (No. CIA110141).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Zhang, Z., Cheng, Y. & Liu, N.C. Comparison of the effect of mean-based method and z-score for field normalization of citations at the level of Web of Science subject categories. Scientometrics 101, 1679–1693 (2014). https://doi.org/10.1007/s11192-014-1294-7
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-014-1294-7