Abstract
In this paper we investigate the problem of university classification and its relation to ranking practices in the policy context of an official evaluation of Romanian higher education institutions and their study programs. We first discuss the importance of research in the government-endorsed assessment process and analyze the evaluation methodology and the results it produced. Based on official documents and data we show that the Romanian classification of universities was implicitly hierarchical in its conception and therefore also produced hierarchical results due to its close association with the ranking of study programs and its heavy reliance on research outputs. Then, using a distinct dataset on the research performance of 1385 faculty members working in the fields of political science, sociology and marketing we further explore the differences between university categories. We find that our alternative assessment of research productivity—measured with the aid of Hirsch’s (Proc Natl Acad Sci 102(46):16569–16572, 2005) h-index and with Egghe’s (Scientometrics 69(1):131–152, 2006) g-index—only provides empirical support for a dichotomous classification of Romanian institutions.
Similar content being viewed by others
Notes
This provision was later annulled through a Government Emergency Ordinance adopted in December 2013.
For all ranking domains belonging to natural sciences, mathematics, biomedical sciences and engineering—in total 33 out of the 60 domains—the research criterion had a weight of 0.60 whereas for most domains belonging to social sciences (for example political science or administrative science) it had a weight of 0.50 and for those within humanities (for example philosophy or history) it had a weight of 0.40.
According to the methodology the h-index is to be calculated based on the following databases for academics belonging to natural sciences, mathematics, biomedical sciences and engineering: Thompson Reuters’ Web of Science, Elsevier’s Scopus, Google Scholar. Different weights are attached to the h-index from each database: 0.5 for Web of Science, 0.3 for Scopus and 0.2 for Google Scholar. For academics in social sciences and humanities the h-index is to be calculated only from Google Scholar.
There is of course one logical exception: in the case of index values normalized with reference to academic titles the intra-university tests of difference which compare index values between academic titles within each of the three university types are rendered obsolete. In the case of this specific analysis conducted under this particular normalization technique the differences in index values are naturally very small and statistically insignificant.
References
Albarrán, P., Crespo, J. A., Ortuño, I., & Ruiz-Castillo, J. (2011). The skewness of science in 219 sub-fields and a number of aggregates. Scientometrics, 88(2), 385–397.
Andreescu, L., Gheorghiu, R., Irimia, A., & Curaj, A. (2015). Mergers and classifications in Romania: Opportunities and obstacles. In A. Curaj, L. Georghiou, J. C. Harper, & E. Egron-Polak (Eds.), Mergers and alliances in higher education. International practice and emerging opportunities (pp. 33–55). Dordrecht: Springer.
Andreescu, L., Gheorghiu, R., Proteasa, V., & Curaj, A. (2012). Institutional diversification and homogeneity in Romanian higher education: The larger picture. In A. Curaj, P. Scott, L. Vlăsceanu, & L. Wilson (Eds.), European higher education at the crossroads: Between the Bologna process and national reforms (pp. 863–885). Dordrecht: Springer.
Billaut, J.-C., Bouyssou, D., & Vincke, P. (2010). Should you believe in the Shanghai ranking? An MCDM view. Scientometrics, 84(1), 237–263.
Bornmann, L., & Leydesdorff, L. (2014). On the meaningful and non-meaningful use of reference sets in bibliometrics. Journal of Informetrics, 8(1), 273–275.
Bornmann, L., Mutz, R., Hug, S. E., & Daniel, H.-D. (2011). A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants. Journal of Informetrics, 5(3), 346–359.
Buela-Casal, G., Gutiérrez-Martínez, O., Bermúdez-Sánchez, M. P., & Vadillo-Muñoz, O. (2007). Comparative study of international academic rankings of universities. Scientometrics, 71(3), 349–365.
Cheng, Y., & Liu, N. C. (2006). A first approach to the classification of the top 500 world universities by their disciplinary characteristics using scientometrics. Scientometrics, 68(1), 135–150.
Costas, R., & Bordons, M. (2007). The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level. Journal of Informetrics, 1(3), 193–203.
Dill, D., & Soo, M. (2005). Academic quality, league tables, and public policy: A crossnational analysis of university ranking systems. Higher Education, 49(4), 495–533.
Egghe, L. (2006). Theory and practise of the g-index. Scientometrics, 69(1), 131–152.
García, J. A., Rodríguez-Sánchez, R., Fdez-Valdivia, J., Robinson-García, N., & Torres-Salinas, D. (2012). Mapping academic institutions according to their journal publication profile: Spanish universities as a case study. Journal of the American Society for Information Science and Technology, 63(11), 2328–2340.
Geuna, A. (2001). The changing rationale for European university research funding: Are there negative unintended consequences? Journal of Economic Issues, 35(3), 607–632.
Glänzel, W., & Henk, F. M. (2013). Opinion paper: Thoughts and facts on bibliometric indicators. Scientometrics, 96(1), 381–394.
Harzing, A. W. (2007). Publish or Perish. http://www.harzing.com/pop.htm.
Harzing, A.-W., Alakangas, S., & Adams, D. (2014). hIa: An individual annual h-index to accommodate disciplinary and career length differences. Scientometrics, 99(3), 811–821.
Hazelkorn, E. (2013). How rankings are reshaping higher education. In V. Climent, F. Michavila, & M. Ripolles (Eds.), Los rankings univeritarios: Mitos y realidades. Ed. Tecnos.
Hicks, D. (2012). Performance-based university research funding systems. Research Policy, 41(2), 251–261.
Hirsch, J. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences, 102(46), 16569–16572.
Huisman, J., Meek, L., & Wood, F. (2007). Institutional diversity in higher education: A cross-national and longitudinal analysis. Higher Education Quarterly, 61(4), 563–577.
Huisman, J., & van Vught, F. (2009). Diversity in European higher education: Historical trends and current policies. In F. van Vught (Ed.), Mapping the higher education landscape. Towards a European classification of higher education (pp. 17–38). Dordrecht: Springer.
Kaiser, F., Faber, M., & Jongbloed, B. (2012). U-Map, university activity profiles in practice. In A. Curaj, P. Scott, L. Vlăsceanu, & L. Wilson (Eds.), European higher education at the crossroads: Between the Bologna process and national reforms (pp. 887–903). Dordrecht: Springer.
Kaur, J., Radicchi, F., & Menczer, F. (2013). Universality of scholarly impact metrics. Journal of Informetrics, 7(4), 924–932.
Longden, B. (2011). Ranking indicators and weights. In J. C. Shin, R. K. Toutkoushian, & U. Teichler (Eds.), University rankings. Theoretical basis, methodology and impacts on global higher education (pp. 73–104). New York: Springer.
McCormick, A. (2008). The Complex interplay between classification and ranking of colleges and universities: Should the Berlin principles apply equally to classification? Higher Education in Europe, 33(2–3), 209–218.
McCormick, A., & Zhao, C. (2005). Rethinking and reframing the Carnegie Classification. Change, 37(5), 51–57.
Miroiu, A., & Andreescu, L. (2010). Goals and instruments of diversification in higher education. Quality Assurance Review for Higher Education, 2(2), 89–101.
Miroiu, A., Păunescu, M., & Vîiu, G.-A. (2015). Ranking Romanian academic departments in three fields of study using the g-index. Quality in Higher Education, 21(2), 189–212.
Ortega, J. S., López-Romero, E., & Fernández, I. (2011). Multivariate approach to classify research institutes according to their outputs: The case of the CSIC’s institutes. Journal of Informetrics, 5(3), 323–332.
Păunescu, M., Florian, B., & Hâncean, M.-G. (2012). Internalizing quality assurance in higher education: Challenges of transition in enhancing the institutional responsibility for quality. In A. Curaj, P. Scott, L. Vlăsceanu, & L. Wilson (Eds.), European higher education at the crossroads: Between the Bologna process and national reforms (pp. 317–338). Dordrecht: Springer.
Rauhvargers, A. (2011). Global university rankings and their impact. Brussels: European University Association.
Ruiz-Castillo, J., & Costas, R. (2014). The skewness of scientific productivity. Journal of Informetrics, 8(4), 917–934.
Sadlak, J., & Liu, N. C. (2007). The world-class university and ranking: Aiming beyond status. Bucharest: UNESCO-CEPES.
Salmi, J. (2009). The challenge of establishing world-class universities. Washington, DC: World Bank.
Salmi, J., & Saroyan, A. (2007). League tables as policy instruments: Uses and misuses. Higher Education Management and Policy, 19(2), 31–68.
Schreiber, M. (2008). An empirical investigation of the g-index for 26 physicists in comparison with the h-index, the a-index, and the r-index. Journal of the American Society for Information Science and Technology, 59(9), 1513–1522.
Seglen, P. O. (1992). The skewness of science. Journal of the American Society for Information Science, 43(9), 628–638.
Shin, J. C. (2009). Classifying higher education institutions in Korea: A performance-based approach. Higher Education, 57(2), 247–266.
Shin, J. C., & Kehm, B. (Eds.). (2013). Institutionalization of world-class university in global competition. Dordrecht: Springer.
Shulman, L. S. (2005). Classification’s complexities. The Chronicle of Higher Education B, 52, 20.
Stensaker, B., & Gornitzka, A. (2009). The ingredients of trust in European higher education’. In B. M. Kehm, J. Huisman, & B. Stensaker (Eds.), The European higher education area: Perspectives on a moving target (pp. 125–139). Rotterdam: Sense Publishers.
Usher, A., & Medow, J. (2009). A global survey of university rankings and league tables. In B. M. Kehm & B. Stensaker (Eds.), University rankings, diversity, and the new landscape of higher education (pp. 3–18). Rotterdam: Sense Publishers.
van der Wende, M. (2008). Rankings and classifications in higher education: A European perspective. In J. C. Smart (Ed.), Higher education: Handbook of theory and research (Vol. 23, pp. 49–72). Berlin: Springer.
van Raan, A. F. J. (2005a). Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62(1), 133–143.
van Raan, A. F. J. (2005b). Measurement of central aspects of scientific research: Performance, interdisciplinarity, structure. Measurement, 3(1), 1–19.
van Vught, F. (2009). Diversity and differentiation in higher education. In F. van Vught (Ed.), Mapping the higher education landscape. Towards a European classification of higher education (pp. 1–16). Dordrecht: Springer.
van Vught, F., & Ziegele, F. (Eds.). (2011). Design and testing the feasibility of a multidimensional global university ranking. Final report. Consortium for Higher Education and Research Performance Assessment, CHERPA-Network.
Vîiu, G.-A., Vlăsceanu, M., & Miroiu, A. (2012). Ranking political science departments: The case of Romania. Quality Assurance Review for Higher Education, 4(2), 79–97.
Vîiu, G.-A., Păunescu, M., & Miroiu, A. (2015). Research-driven classification and ranking in higher education. An empirical appraisal of a Romanian policy experience. In A. A. Salah, Y. Tonta, A. A. A. Salah, A. Sugimoto, & U. Al (Eds.), Proceedings of ISSI 2015 Istanbul (pp. 622–633). Istanbul: Boğaziçi University Printhouse.
Vlăsceanu, L., Miroiu, A., Păunescu, M., & Hâncean, M.-G. (Eds.). (2011). Barometrul calităţii 2010. Starea calităţii în învăţământul superior din România. Braşov: Editura Universităţii Transilvania din Braşov.
Acknowledgments
Financial support from the National Research Council (Grant Number PN-II-ID-PCE-2011-3-0746) is gratefully acknowledged by Gabriel Vîiu and Adrian Miroiu. The authors would also like to express their gratitude for the helpful comments and suggestions provided to them during the anonymous peer review process, both for the original paper submitted for the 2015 ISSI conference as well as for the subsequent manuscript submitted to Scientometrics.
Author information
Authors and Affiliations
Corresponding author
Additional information
The present paper is a revised and expanded version of the work (Vîiu et al. 2015) previously published in the Proceedings of ISSI 2015 Istanbul.
Rights and permissions
About this article
Cite this article
Vîiu, GA., Păunescu, M. & Miroiu, A. Research-driven classification and ranking in higher education: an empirical appraisal of a Romanian policy experience. Scientometrics 107, 785–805 (2016). https://doi.org/10.1007/s11192-016-1860-2
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-016-1860-2