Abstract
Due to its quantitative nature, bibliometrics is becoming increasingly popular among policy makers for academic hiring and career promotions. In this article, we quantitatively assess the impact that the granularity level in the classification of scientific areas would entail on research evaluation based on bibliometric indicators. We use as a case study the Italian national habilitation system (ASN), which classifies faculty members according to their academic discipline and relies on journal counts, citations, and h-indices as a basis for promoting tenure track researchers to associate professors and associate to full professors. The assessment checks whether the individual indicators of a researcher are above a certain threshold, e.g., the median over the population of researchers working in the same discipline. Our investigation focuses on two related, rather broad disciplines: computer science and computer engineering. We show that the ASN practice of using the same thresholds for all members of a scientific discipline can favor certain sub-communities that are characterized by higher bibliometric indicators, and disfavor others. We report evidence that up to 30% of Italian faculty members of certain sub-communities would see their indicators drop below the threshold, thus becoming not eligible for promotion, if the ASN were conducted on a more accurate, fine-grained classification. Conversely, in the same scenario, up to 11% of faculty members, in different sub-communities, would see their indicators rise above the threshold, granting them eligibility. Our data set includes 1685 authors, 89,185 distinct publications, and 262,286 author-publication pairs.
Similar content being viewed by others
Notes
We refer to the Web page https://www.anvur.it/attivita/asn/asn-2012-2013/indicatori-e-relative-mediane/ and to the document https://www.anvur.it/wp-content/uploads/2012/08/documento_accompagnamento_all_2.pdf (both in Italian) for details.
References
Abramo, G., & D’Angelo, C. A. (2014). How do you define and measure research productivity? Scientometrics, 101(2), 1129–1144. https://doi.org/10.1007/s11192-014-1269-8.
Abramo, G., & D’Angelo, C. A. (2015). An assessment of the first “scientific habilitation” for university appointments in Italy. Economia Politica, 32(3), 329–357. https://doi.org/10.1007/s40888-015-0016-9.
Abramo, G., & D’Angelo, C. A. (2015). The determinants of academic career advancement: Evidence from Italy. Science and Public Policy, 42(6), 761–774. https://doi.org/10.1093/scipol/scu086.
Abramo, G., & D’Angelo, A. C. (2016). A farewell to the MNCS and like size-independent indicators. Journal of Informetrics, 10(2), 646–651. https://doi.org/10.1016/j.joi.2016.04.006.
Abramo, G., D’Angelo, C. A., & Rosati, F. (2014). Career advancement and scientific performance in universities. Scientometrics, 98(2), 891–907. https://doi.org/10.1007/s11192-013-1075-8.
Becher, T. (1989). Academic tribes and territories: Intellectual enquiry and the cultures of disciplines. Buckingham: Open University Press.
Becher, T., & Trowler, P. R. (2001). Academic tribes and territories: Intellectual enquiry and the cultures of disciplines (2nd ed.). Buckingham: Open University Press.
Bornmann, L., Mutz, R., & Daniel, H.-D. (2008a). Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine. Journal of the Association for Information Science and Technology, 59(5), 830–837.
Bornmann, L., Mutz, R., & Daniel, H.-D. (2008b). Citation counts for research evaluation: standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics in Science and Environmental Politics, 8, 93–102.
Braithwaite, J., Herkes, J., Churruca, K., Long, J. C., Pomare, C., Boyling, C., et al. (2019). Comprehensive researcher achievement model (cram): A framework for measuring researcher achievement, impact and influence derived from a systematic literature review of metrics and models. BMJ Open,. https://doi.org/10.1136/bmjopen-2018-025320.
Demetrescu, C., Ribichini, A., & Schaerf, M. (2018). Accuracy of author names in bibliographic data sources: An italian case study. Scientometrics, 117(3), 1777–1791.
Demetrescu, C., Lupia, F., Mendicelli, A., Ribichini, A., Scarcello, F., & Schaerf, M. (2019). On the Shapley value and its application to the Italian VQR research assessment exercise. Journal of Informetrics, 13(1), 87–104.
Egghe, L. (2006). An improvement of the h-index: The g-index. ISSI Newsletter, 2, 05.
Garner, R., Hirsch, J., Albuquerque, F., & Fargen, K. (2017). Bibliometric indices: Defining academic productivity and citation rates of researchers, departments and journals. Journal of Neurointerventional Surgery, 10, 08. https://doi.org/10.1136/neurintsurg-2017-013265.
Gorraiz, J., Gumpenberger, C., & Glade, T. (2016). On the bibliometric coordinates of four different research fields in geography. Scientometrics, 107(2), 873–897.
Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature, 520(7548), 429–431. https://doi.org/10.1038/520429a.
Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences, 102(46), 16569–16572. https://doi.org/10.1073/pnas.0507655102.
Kelly, C. D., & Jennions, M. D. (2006). The H index and career assessment by numbers. Trends in Ecology and Evolution, 21(4), 167–170. https://doi.org/10.1016/j.tree.2006.01.005.
Kreiman, G., & Maunsell, J. (2011). Nine criteria for a measure of scientific output. Frontiers in Computational Neuroscience, 5, 48. https://doi.org/10.3389/fncom.2011.00048.
Marini, G. (2017). New promotion patterns in italian universities: Less seniority and more productivity? Data from ASN. Higher Education, 73(2), 189–205. https://doi.org/10.1007/s10734-016-0008-x. ISSN 1573-174X.
Marini, G. (2018). Tools of individual evaluation and prestige recognition in spain: How sexenio ‘mints the golden coin of authority’. European Journal of Higher Education, 8(2), 201–214. https://doi.org/10.1080/21568235.2018.1428649.
Marini, G., & Meschitti, V. (2018). The trench warfare of gender discrimination: Evidence from academic promotions to full professor in italy. Scientometrics, 115(2), 9891006. https://doi.org/10.1007/s11192-018-2696-8.
Marzolla, M. (2015). Quantitative analysis of the Italian national scientific qualification. Journal of Informetrics, 9(2), 285–316. https://doi.org/10.1016/j.joi.2015.02.006.
Marzolla, M. (2016). Assessing evaluation procedures for individual researchers: The case of the Italian National Scientific Qualification. Journal of Informetrics, 10(2), 408–438. https://doi.org/10.1016/j.joi.2016.01.009.
Pautasso, M. (2015). The Italian University habilitation and the challenge of increasing the representation of women in academia. Challenges, 6(1), 26–41. https://doi.org/10.3390/challe6010026.
Sidiropoulos, A., Katsaros, D., & Manolopoulos, Y. (2007). Generalized hirsch h-index for disclosing latent facts in citation networks. Scientometrics, 72(253–280), 08. https://doi.org/10.1007/s11192-007-1722-z.
Simko, I. (2015). Analysis of bibliometric indicators to determine citation bias. Palgrave Communications, 1, 15011.
Todeschini, R., & Baccini, A. (2016). Handbook of bibliometric indicators: Quantitative tools for studying and evaluating research. Hoboken: Wiley.
Trowler, P. (2014). Academic tribes and territories: The theoretical trajectory. Osterreichische Zeitschrift fur Geschichtswissenschaften, 25, 17–26.
Trowler, P., Saunders, M., & Bamber, V. (2012). Tribes and territories in the 21st century: Rethinking the significance of disciplines in higher education. New York: Routledge.
Waltman, L., van Eck, N. J., van Leeuwen, T. N., Visser, M. S., & van Raan, A. J. (2011). Towards a new crown indicator: Some theoretical considerations. Journal of Informetrics, 5(1), 37–47. https://doi.org/10.1016/j.joi.2010.08.001.
Zhang, C.-T. (2009). The e-index, complementing the h-index for excess citations. PLOS One, 4(5), 1–4. https://doi.org/10.1371/journal.pone.0005429.
Acknowledgements
Work partially supported by MIUR, the Italian Ministry of Education, University and Research, under PRIN Project n. 20174LF3T8 AHeAD (Efficient Algorithms for HArnessing Networked Data).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Demetrescu, C., Finocchi, I., Ribichini, A. et al. On bibliometrics in academic promotions: a case study in computer science and engineering in Italy. Scientometrics 124, 2207–2228 (2020). https://doi.org/10.1007/s11192-020-03548-9
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-020-03548-9