Abstract
Scientific production has been evaluated from very different perspectives, the best known of which are essentially based on the impact factors of the journals included in the Journal Citation Reports (JCR). This has been no impediment to the simultaneous issuing of warnings regarding the dangers of their indiscriminate use when making comparisons. This is because the biases incorporated in the elaboration of these impact factors produce significant distortions, which may invalidate the results obtained. Notable among such biases are those generated by the differences in the propensity to cite of the different areas, journals and/or authors, by variations in the period of materialisation of the impact and by the varying presence of knowledge areas in the sample of reviews contained in the JCR. While the traditional evaluation method consists of standardisation by subject categories, recent studies have criticised this approach and offered new possibilities for making inter-area comparisons. In view of such developments, the present study proposes a novel approach to the measurement of scientific activity, in an attempt to lessen the aforementioned biases. This approach consists of combining the employment of a new impact factor, calculated for each journal, with the grouping of the institutions under evaluation into homogeneous groups. An empirical application is undertaken to evaluate the scientific production of Spanish public universities in the year 2000. This application considers both the articles published in the multidisciplinary databases of the Web of Science (WoS) and the data concerning the journals contained in the Sciences and Social Sciences Editions of the Journal Citation Report (JCR). All this information is provided by the Institute of Scientific Information (ISI), via its Web of Knowledge (WoK).
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
In theory, the normalisation required to homogenise field representativity in the sample should be performed by weighting the millions of citations handled every year by the following factor: 1/total citations of articles published in JCR journals.
Carnegie Foundation’s classification distinguishes between universities which offer the complete range of higher education, including doctoral degrees (Doctorate-granting Institutions or Doctoral/Research Universities) and those which teach up to the Master’s level (Master’s Colleges and Universities) or those which focus principally on Bachelor’s degrees (Baccalaureate Colleges). Maclean’s University Rankings distinguishes between Primarily Undergraduate Universities (i.e. those which are largely dedicated to undergraduates), Comprehensive Universities (which offer an extensive range of qualifications for undergraduates and graduates and receive significant revenue from their research activity) and, lastly, Medical-Doctoral Universities (which offer a wide range of doctoral and research programmes and include medical faculties).
Scientific reviews are the chosen medium for 85% of all works published in the Scientific-Technical areas. In Humanistic and Social Sciences they only represent 40% (books are the chosen medium for 48% of publications in Humanities). Moreover, national reviews are the natural vehicle for contributions to Social Sciences in Spain (Gobierno de Aragón 2004).
These areas are: Mathematics and Physics; Chemistry; Cellular and Molecular Biology; Biomedical Sciences; Natural Sciences; Engineering and Architecture; Social, Political and Behavioural Sciences; Economic and Business Sciences; Law, History and Art; Philosophy, Philology and Linguistics.
The intention here is to relativise universities’ production with regard to their size. Although the disadvantage of this relativisation may reside in how widely teaching/research loads vary among universities, we believe that in the case of the SPUs it is plausible to assume that they all have a similar distribution of teaching and research, since this is established by the relevant legislation.
For the elaboration of these two blocks it was necessary to establish a criterion which permitted the determination of which area an article belonged to. A decision was taken to follow the classification of the JCR and not the Citation Index (SCI and SSCI). The fact that some reviews are included in both subsets was also taken into account. To overcome this difficulty, it was decided to divide the impacts proportionally between two macro-areas.
References
Aksnes, D. W. (2006). Citation rates and perceptions of scientific contribution. Journal of the American Society for Information Science and Technology, 57(2), 169–185.
Archambault, E., & Larivière, V. (2009). History of the journal impact factor: Contingencies and consequences. Scientometrics, 79(3), 635–649.
Braun, T. (1999). Bibliometric indicators for the evaluation of universities—intelligence from the quantitation of the scientific literature. Scientometrics, 45(3), 425–432.
Buchanan, R. A. (2006). Accuracy of cited references—the role of citation databases. College and Research Libraries, 67(4), 292–303.
Garfield, E. (1996). How can impact factors be improved?. British Medical Journal, 313(7054), 411–413.
Garfield, E. (1998). Long-term vs. short-term journal impact: part ii cumulative impact factors. The Scientist, 12(14), 12–13.
Glänzel, W., & Moed, H. F. (2002). Journal impact measures in bibliometric research. Scientometrics, 53(20), 171–193.
Gobierno de Aragón, D. d. C. y. T. (2004). II Plan Autonómico de Investigación, Desarrollo y Transferencia de Conocimientos de Aragón: [II PAID 2005–2008]. Zaragoza: Gobierno de Aragón, Departamento de Ciencia y Tecnología.
Gómez-Sancho, J. M., & Mancebón-Torrubia, M. J. (2009). The evaluation of scientific production: Towards a neutral impact factor. Scientometrics, 81(2), 435–458.
Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate data analysis (7th ed.). New York: Prentice-Hall.
Hernández Armenteros, J. (2002). La universidad española en cifras (2002) Información académica, productiva y financiera de las universidades públicas españolas Indicadores universitarios. Madrid: Conferencia de Rectores de las Universidades Españolas (CRUE).
Jacsó, P. (2006). Deflated, inflated and phantom citation counts. Online Information Review, 30(3), 297–309.
Kostoff, R. N. (2002). Citation analysis of research performer quality. Scientometrics, 53(1), 49–71.
Leydesdorff, L. (2008). Caveats for the use of citation indicators in research and journal evaluations. Journal of the American Society for Information Science and Technology, 59(2), 278–287.
Moed, H. F. (2006). Bibliometric rankings of world universities. CWTS Report 2006-01. The Netherlands: Centre for Science and Technology Studies (CWTS), Leiden University.
Moed, H. F., & Van Leeuwen, T. N. (1995). Improving the accuracy of institute for scientific information’s journal impact factors. Journal of the American Society for Information Science and Technology, 46(6), 461–467.
Moed, H. F., & Van Leeuwen, T. N. (1996). Impact factors can mislead. Nature, 381, 381–386.
Moed, H. F., & Vriens, M. (1989). Possible inaccuracies occurring in citation analysis. Journal of Information Science, 15, 95–107.
Moed, H. F., De Bruin, R. E., & Van Leeuwen, T. N. (1995). New bibliometric tools for the assessment of national research performance—database description, overview of indicators and first applications. Scientometrics, 33(3), 381–422.
Moed, H. F., Van Leeuwen, T. N., & Reedijk, J. (1999). Towards appropriate indicators of journal impact. Scientometrics, 46(3), 575–589.
Moya Anegon, F., Chinchilla Rodríguez, Z., Corera Alvarez, E., Gómez Crisóstomo, M. R., González Molina, A., Muñoz Fernández, F. J., et al. (2007). Indicadores bibliométricos de la actividad científica española (1990–2004). Madrid: FECYT.
Mueller, P. S., Murali, N. S., Cha, S. S., Erwin, P. J. Y., & Ghosh, A. K. (2006). The association between impact factors and language of general internal medicine journals. Swiss Medical Weekly, 136(27/28), 441–443.
Opthof, T., & Leydesdorff, L. (2010). Caveats for the journal and field normalizations in the CWTS (“Leiden”) evaluations of research performance. Journal of Informetrics. doi:10.1016/j.joi.2010.02.003
Rinia, E. J., van Leeuwen, T. N., Bruins, E. E. W., Van Vuren, H. G., & Van Raan, A. F. J. (2001). Citation delay in interdisciplinary knowledge exchange. Scientometrics, 51(1), 293–309.
Rousseau, R. (2005). Median and percentile impact factors: A set of new indicators. Scientometrics, 63(3), 431–441.
Schubert, A., & Braun, T. (1996). Cross-field normalization of scientometric indicators. Scientometrics, 36(1), 311–324.
Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314(7079), 498–502.
Sombatsompop, N., Markpin, T., & Premkamolnetr, N. (2004). A modified method for calculating the impact factors of journals in ISI journal citation reports—polymer science category in 1997–2001. Scientometrics, 60(2), 235–271.
Wallin, J. A. (2005). Bibliometric methods: Pitfalls and possibilities. Basic and Clinical Pharmacology and Toxicology, 97(5), 261–275.
Zitt, M., & Small, H. (2008). Modifying the journal impact factor by fractional citation weighting: The audience factor. Journal of the American Society for Information Science and Technology, 59(11), 1856–1860.
Zitt, M., Ramana-Rahary, S., & Bassecoulard, E. (2005). Relativity of citation performance and excellence measures: From cross-field to cross-scale effects of field-normalisation. Scientometrics, 63(2), 373–401.
Acknowledgments
The authors would like to thank the two anonymous reviewers for their useful and constructive comments. Any errors in the article are responsibility of its authors.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Gómez-Sancho, J.M., Mancebón-Torrubia, M.J. A new approach to measuring scientific production in JCR journals and its application to Spanish public universities. Scientometrics 85, 271–293 (2010). https://doi.org/10.1007/s11192-010-0217-5
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-010-0217-5