Advertisement

Scientometrics

, Volume 107, Issue 2, pp 785–805 | Cite as

Research-driven classification and ranking in higher education: an empirical appraisal of a Romanian policy experience

  • Gabriel-Alexandru Vîiu
  • Mihai Păunescu
  • Adrian Miroiu
Article

Abstract

In this paper we investigate the problem of university classification and its relation to ranking practices in the policy context of an official evaluation of Romanian higher education institutions and their study programs. We first discuss the importance of research in the government-endorsed assessment process and analyze the evaluation methodology and the results it produced. Based on official documents and data we show that the Romanian classification of universities was implicitly hierarchical in its conception and therefore also produced hierarchical results due to its close association with the ranking of study programs and its heavy reliance on research outputs. Then, using a distinct dataset on the research performance of 1385 faculty members working in the fields of political science, sociology and marketing we further explore the differences between university categories. We find that our alternative assessment of research productivity—measured with the aid of Hirsch’s (Proc Natl Acad Sci 102(46):16569–16572, 2005) h-index and with Egghe’s (Scientometrics 69(1):131–152, 2006) g-index—only provides empirical support for a dichotomous classification of Romanian institutions.

Keywords

Research evaluation Higher education Institutional classification Institutional ranking h-index g-index 

Notes

Acknowledgments

Financial support from the National Research Council (Grant Number PN-II-ID-PCE-2011-3-0746) is gratefully acknowledged by Gabriel Vîiu and Adrian Miroiu. The authors would also like to express their gratitude for the helpful comments and suggestions provided to them during the anonymous peer review process, both for the original paper submitted for the 2015 ISSI conference as well as for the subsequent manuscript submitted to Scientometrics.

References

  1. Albarrán, P., Crespo, J. A., Ortuño, I., & Ruiz-Castillo, J. (2011). The skewness of science in 219 sub-fields and a number of aggregates. Scientometrics, 88(2), 385–397.CrossRefGoogle Scholar
  2. Andreescu, L., Gheorghiu, R., Irimia, A., & Curaj, A. (2015). Mergers and classifications in Romania: Opportunities and obstacles. In A. Curaj, L. Georghiou, J. C. Harper, & E. Egron-Polak (Eds.), Mergers and alliances in higher education. International practice and emerging opportunities (pp. 33–55). Dordrecht: Springer.CrossRefGoogle Scholar
  3. Andreescu, L., Gheorghiu, R., Proteasa, V., & Curaj, A. (2012). Institutional diversification and homogeneity in Romanian higher education: The larger picture. In A. Curaj, P. Scott, L. Vlăsceanu, & L. Wilson (Eds.), European higher education at the crossroads: Between the Bologna process and national reforms (pp. 863–885). Dordrecht: Springer.CrossRefGoogle Scholar
  4. Billaut, J.-C., Bouyssou, D., & Vincke, P. (2010). Should you believe in the Shanghai ranking? An MCDM view. Scientometrics, 84(1), 237–263.CrossRefGoogle Scholar
  5. Bornmann, L., & Leydesdorff, L. (2014). On the meaningful and non-meaningful use of reference sets in bibliometrics. Journal of Informetrics, 8(1), 273–275.CrossRefGoogle Scholar
  6. Bornmann, L., Mutz, R., Hug, S. E., & Daniel, H.-D. (2011). A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants. Journal of Informetrics, 5(3), 346–359.CrossRefGoogle Scholar
  7. Buela-Casal, G., Gutiérrez-Martínez, O., Bermúdez-Sánchez, M. P., & Vadillo-Muñoz, O. (2007). Comparative study of international academic rankings of universities. Scientometrics, 71(3), 349–365.CrossRefGoogle Scholar
  8. Cheng, Y., & Liu, N. C. (2006). A first approach to the classification of the top 500 world universities by their disciplinary characteristics using scientometrics. Scientometrics, 68(1), 135–150.CrossRefGoogle Scholar
  9. Costas, R., & Bordons, M. (2007). The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level. Journal of Informetrics, 1(3), 193–203.CrossRefGoogle Scholar
  10. Dill, D., & Soo, M. (2005). Academic quality, league tables, and public policy: A crossnational analysis of university ranking systems. Higher Education, 49(4), 495–533.CrossRefGoogle Scholar
  11. Egghe, L. (2006). Theory and practise of the g-index. Scientometrics, 69(1), 131–152.MathSciNetCrossRefGoogle Scholar
  12. García, J. A., Rodríguez-Sánchez, R., Fdez-Valdivia, J., Robinson-García, N., & Torres-Salinas, D. (2012). Mapping academic institutions according to their journal publication profile: Spanish universities as a case study. Journal of the American Society for Information Science and Technology, 63(11), 2328–2340.CrossRefGoogle Scholar
  13. Geuna, A. (2001). The changing rationale for European university research funding: Are there negative unintended consequences? Journal of Economic Issues, 35(3), 607–632.CrossRefGoogle Scholar
  14. Glänzel, W., & Henk, F. M. (2013). Opinion paper: Thoughts and facts on bibliometric indicators. Scientometrics, 96(1), 381–394.CrossRefGoogle Scholar
  15. Harzing, A. W. (2007). Publish or Perish. http://www.harzing.com/pop.htm.
  16. Harzing, A.-W., Alakangas, S., & Adams, D. (2014). hIa: An individual annual h-index to accommodate disciplinary and career length differences. Scientometrics, 99(3), 811–821.CrossRefGoogle Scholar
  17. Hazelkorn, E. (2013). How rankings are reshaping higher education. In V. Climent, F. Michavila, & M. Ripolles (Eds.), Los rankings univeritarios: Mitos y realidades. Ed. Tecnos.Google Scholar
  18. Hicks, D. (2012). Performance-based university research funding systems. Research Policy, 41(2), 251–261.MathSciNetCrossRefGoogle Scholar
  19. Hirsch, J. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences, 102(46), 16569–16572.CrossRefGoogle Scholar
  20. Huisman, J., Meek, L., & Wood, F. (2007). Institutional diversity in higher education: A cross-national and longitudinal analysis. Higher Education Quarterly, 61(4), 563–577.CrossRefGoogle Scholar
  21. Huisman, J., & van Vught, F. (2009). Diversity in European higher education: Historical trends and current policies. In F. van Vught (Ed.), Mapping the higher education landscape. Towards a European classification of higher education (pp. 17–38). Dordrecht: Springer.CrossRefGoogle Scholar
  22. Kaiser, F., Faber, M., & Jongbloed, B. (2012). U-Map, university activity profiles in practice. In A. Curaj, P. Scott, L. Vlăsceanu, & L. Wilson (Eds.), European higher education at the crossroads: Between the Bologna process and national reforms (pp. 887–903). Dordrecht: Springer.CrossRefGoogle Scholar
  23. Kaur, J., Radicchi, F., & Menczer, F. (2013). Universality of scholarly impact metrics. Journal of Informetrics, 7(4), 924–932.CrossRefGoogle Scholar
  24. Longden, B. (2011). Ranking indicators and weights. In J. C. Shin, R. K. Toutkoushian, & U. Teichler (Eds.), University rankings. Theoretical basis, methodology and impacts on global higher education (pp. 73–104). New York: Springer.Google Scholar
  25. McCormick, A. (2008). The Complex interplay between classification and ranking of colleges and universities: Should the Berlin principles apply equally to classification? Higher Education in Europe, 33(2–3), 209–218.CrossRefGoogle Scholar
  26. McCormick, A., & Zhao, C. (2005). Rethinking and reframing the Carnegie Classification. Change, 37(5), 51–57.CrossRefGoogle Scholar
  27. Miroiu, A., & Andreescu, L. (2010). Goals and instruments of diversification in higher education. Quality Assurance Review for Higher Education, 2(2), 89–101.Google Scholar
  28. Miroiu, A., Păunescu, M., & Vîiu, G.-A. (2015). Ranking Romanian academic departments in three fields of study using the g-index. Quality in Higher Education, 21(2), 189–212.CrossRefGoogle Scholar
  29. Ortega, J. S., López-Romero, E., & Fernández, I. (2011). Multivariate approach to classify research institutes according to their outputs: The case of the CSIC’s institutes. Journal of Informetrics, 5(3), 323–332.Google Scholar
  30. Păunescu, M., Florian, B., & Hâncean, M.-G. (2012). Internalizing quality assurance in higher education: Challenges of transition in enhancing the institutional responsibility for quality. In A. Curaj, P. Scott, L. Vlăsceanu, & L. Wilson (Eds.), European higher education at the crossroads: Between the Bologna process and national reforms (pp. 317–338). Dordrecht: Springer.CrossRefGoogle Scholar
  31. Rauhvargers, A. (2011). Global university rankings and their impact. Brussels: European University Association.Google Scholar
  32. Ruiz-Castillo, J., & Costas, R. (2014). The skewness of scientific productivity. Journal of Informetrics, 8(4), 917–934.CrossRefGoogle Scholar
  33. Sadlak, J., & Liu, N. C. (2007). The world-class university and ranking: Aiming beyond status. Bucharest: UNESCO-CEPES.Google Scholar
  34. Salmi, J. (2009). The challenge of establishing world-class universities. Washington, DC: World Bank.CrossRefGoogle Scholar
  35. Salmi, J., & Saroyan, A. (2007). League tables as policy instruments: Uses and misuses. Higher Education Management and Policy, 19(2), 31–68.CrossRefGoogle Scholar
  36. Schreiber, M. (2008). An empirical investigation of the g-index for 26 physicists in comparison with the h-index, the a-index, and the r-index. Journal of the American Society for Information Science and Technology, 59(9), 1513–1522.CrossRefGoogle Scholar
  37. Seglen, P. O. (1992). The skewness of science. Journal of the American Society for Information Science, 43(9), 628–638.CrossRefGoogle Scholar
  38. Shin, J. C. (2009). Classifying higher education institutions in Korea: A performance-based approach. Higher Education, 57(2), 247–266.CrossRefGoogle Scholar
  39. Shin, J. C., & Kehm, B. (Eds.). (2013). Institutionalization of world-class university in global competition. Dordrecht: Springer.Google Scholar
  40. Shulman, L. S. (2005). Classification’s complexities. The Chronicle of Higher Education B, 52, 20.MathSciNetGoogle Scholar
  41. Stensaker, B., & Gornitzka, A. (2009). The ingredients of trust in European higher education’. In B. M. Kehm, J. Huisman, & B. Stensaker (Eds.), The European higher education area: Perspectives on a moving target (pp. 125–139). Rotterdam: Sense Publishers.Google Scholar
  42. Usher, A., & Medow, J. (2009). A global survey of university rankings and league tables. In B. M. Kehm & B. Stensaker (Eds.), University rankings, diversity, and the new landscape of higher education (pp. 3–18). Rotterdam: Sense Publishers.Google Scholar
  43. van der Wende, M. (2008). Rankings and classifications in higher education: A European perspective. In J. C. Smart (Ed.), Higher education: Handbook of theory and research (Vol. 23, pp. 49–72). Berlin: Springer.CrossRefGoogle Scholar
  44. van Raan, A. F. J. (2005a). Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62(1), 133–143.CrossRefGoogle Scholar
  45. van Raan, A. F. J. (2005b). Measurement of central aspects of scientific research: Performance, interdisciplinarity, structure. Measurement, 3(1), 1–19.Google Scholar
  46. van Vught, F. (2009). Diversity and differentiation in higher education. In F. van Vught (Ed.), Mapping the higher education landscape. Towards a European classification of higher education (pp. 1–16). Dordrecht: Springer.CrossRefGoogle Scholar
  47. van Vught, F., & Ziegele, F. (Eds.). (2011). Design and testing the feasibility of a multidimensional global university ranking. Final report. Consortium for Higher Education and Research Performance Assessment, CHERPA-Network.Google Scholar
  48. Vîiu, G.-A., Vlăsceanu, M., & Miroiu, A. (2012). Ranking political science departments: The case of Romania. Quality Assurance Review for Higher Education, 4(2), 79–97.Google Scholar
  49. Vîiu, G.-A., Păunescu, M., & Miroiu, A. (2015). Research-driven classification and ranking in higher education. An empirical appraisal of a Romanian policy experience. In A. A. Salah, Y. Tonta, A. A. A. Salah, A. Sugimoto, & U. Al (Eds.), Proceedings of ISSI 2015 Istanbul (pp. 622–633). Istanbul: Boğaziçi University Printhouse.Google Scholar
  50. Vlăsceanu, L., Miroiu, A., Păunescu, M., & Hâncean, M.-G. (Eds.). (2011). Barometrul calităţii 2010. Starea calităţii în învăţământul superior din România. Braşov: Editura Universităţii Transilvania din Braşov.Google Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2016

Authors and Affiliations

  1. 1.Department of Political ScienceNational School of Political and Administrative StudiesBucharestRomania

Personalised recommendations