Robust Ranking of Universities Evaluated by Hierarchical and Interacting Criteria

  • Salvatore CorrenteEmail author
  • Salvatore Greco
  • Roman Słowiński
Part of the International Series in Operations Research & Management Science book series (ISOR, volume 274)


In this chapter, we present a methodology of decision aiding that helps to build a ranking of a finite set of alternatives evaluated by a family of hierarchically structured criteria. The presentation has a tutorial character, and takes as an example the ranking of universities. Each university is generally evaluated on several aspects, such as quality of faculty and research output. Moreover, their performance on these macro-criteria can be further detailed by evaluation on some subcriteria. To take into account the hierarchical structure of criteria presented as a tree, the multiple criteria hierarchy process will be applied. The aggregation of the university performances will be done by the Choquet integral preference model that is able to take into account the possible negative and positive interactions between the criteria at hand. On the basis of an indirect preference information supplied by the decision maker in terms of pairwise comparisons of some universities, or comparison of some criteria in terms of their importance and their interaction, the robust ordinal regression and the stochastic multicriteria acceptability analysis will be used. They will provide the decision maker some robust recommendations presented in the form of necessary and possible preference relations between universities, and in the form of a distribution of possible rank positions got by each of them, taking into account all preference models compatible with the available preference information. The methodology will be presented step by step on a sample of some European universities.



The authors wish to acknowledge the two anonymous reviewers for their valuable remarks and suggestions on the first version of this chapter. The first two authors wish to acknowledge the funding by the FIR of the University of Catania “BCAEA3, New developments in Multiple Criteria Decision Aiding (MCDA) and their application to territorial competitiveness” and by the research project “Data analytics for entrepreneurial ecosystems, sustainable development and wellbeing indices” of the Department of Economics and Business of the University of Catania. The second author has also benefited of the fund “Chance” of the University of Catania.


  1. Abramo, G., Cicero, T., & D’Angelo, C.A. (2011). A field-standardized application of DEA to national-scale research assessment of universities. Journal of Informetrics, 5(4), 618–628.Google Scholar
  2. Agasisti, T., Dal Bianco, A., Landoni, P., Sala, A., & Salerno, M. (2011). Evaluating the efficiency of research in academic departments: An empirical analysis in an Italian region. Higher Education Quarterly, 65(3), 267–289.Google Scholar
  3. Agasisti, T., Catalano, G., Landoni, P., & Verganti, R. (2012). Evaluating the performance of academic departments: An analysis of research-related output efficiency. Research Evaluation, 21(1), 2–14.Google Scholar
  4. Aigner, D., Lovell, C. A. K., & Schmidt, P. (1977). Formulation and estimation of stochastic frontier production function models. Journal of Econometrics, 6(1), 21–37.Google Scholar
  5. Angilella, S., Corrente, S., Greco, S., & Słowiński, R. (2016). Robust ordinal regression and stochastic multiobjective acceptability analysis in multiple criteria hierarchy process for the Choquet integral preference model. Omega, 63, 154–169.Google Scholar
  6. Angilella, S., Greco, S., & Matarazzo, B. (2010). Non-additive robust ordinal regression: A multiple criteria decision model based on the Choquet integral. European Journal of Operational Research, 201(1), 277–288.Google Scholar
  7. Aoki, S., Inoue, K., & Gejima, R. (2010). Data envelopment analysis for evaluating Japanese universities. Artificial Life and Robotics, 15(2), 165–170.Google Scholar
  8. ARWU. (2018) The rankings of the Shanghai Jiao Tong University.
  9. Bayraktar, E., Tatoglu, E., & Zaim, S. (2013). Measuring the relative efficiency of quality management practices in Turkish public and private universities. Journal of the Operational Research Society, 64(12), 1810–1830.Google Scholar
  10. Behzadian, M., Kazemzadeh, R. B., Albadvi, A., & Aghdasi, M. (2010). PROMETHEE: A comprehensive literature review on methodologies and applications. European Journal of Operational Research, 200(1), 198–215.Google Scholar
  11. Billaut, J.-C., Bouyssou, D., & Vincke, P. (2010). Should you believe in the Shanghai ranking? Scientometrics, 84(1), 237–263.Google Scholar
  12. Blasi, B., Romagnosi, S., & Bonaccorsi, A. (2017). Playing the ranking game: Media coverage of the evaluation of the quality of research in Italy. Higher Education, 73(5), 741–757.Google Scholar
  13. Brans, J. P., Mareschal, B., & Vincke, Ph. (1984). PROMETHEE: A new family of outranking methods in multicriteria analysis. In J. P. Brans (Ed.), Operational Research, IFORS 84 (pp. 477–490). Amsterdam: North Holland.Google Scholar
  14. Brans, J. P., & Vincke, Ph. (1985). A preference ranking organisation method: The PROMETHEE method for MCDM. Management Science, 31(6), 647–656.Google Scholar
  15. Buela-Casal, G., Gutiérrez-Martínez, O., Bermúdez-Sánchez, M. P., & Vadillo-Muñoz, O. (2007). Comparative study of international academic rankings of universities. Scientometrics, 71(3), 349–365.Google Scholar
  16. CEPES. (2006). The Berlin principles on ranking of higher education institutions. Google Scholar
  17. Chavas, J.-P., Barham, B., Foltz, J., & Kim, K. (2012). Analysis and decomposition of scope economies: R&D at US research universities. Applied Economics, 44(11), 1387–1404.Google Scholar
  18. Choquet, G. (1953). Theory of capacities. Annales de l’Institut Fourier, 5(54), 131–295.Google Scholar
  19. Cooper, W. W., Seiford, L. M., & Zhu, J. (2004). Data envelopment analysis. In Handbook on data envelopment analysis (pp. 1–39). Berlin: Springer.Google Scholar
  20. Corrente, S., Figueira, J. R., Greco, S., & Słowiński. (2017). A robust ranking method extending ELECTRE III to hierarchy of interacting criteria, imprecise weights and stochastic analysis. Omega, 73, 1–17.Google Scholar
  21. Corrente, S., Greco, S., & Ishizaka, A. (2016). Combining analytical hierarchy process and Choquet integral within non additive robust ordinal regression. Omega, 61, 2–18.Google Scholar
  22. Corrente, S., Greco, S., Kadziński, M., & Słowiński, R. (2013). Robust ordinal regression in preference learning and ranking. Machine Learning, 93, 381–422.Google Scholar
  23. Corrente, S., Greco, S., & Słowiński, R. (2012). Multiple criteria hierarchy process in robust ordinal regression. Decision Support Systems, 53(3), 660–674.Google Scholar
  24. CWTS. (2018). The ranking of the centre for science and technology studies at Leiden University. Google Scholar
  25. Deb, K., Agrawal, S., Pratap, A., & Meyarivan, T. (2002). A fast and elitist multi-objective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation, 6(2), 182–197.Google Scholar
  26. Figueira, J. R., Greco, S., Roy, B., & Słowiński, R. (2013). An overview of ELECTRE methods and their recent extensions. Journal of Multicriteria Decision Analysis, 20, 61–85.Google Scholar
  27. Florian, R. (2007). Irreproducibility of the results of the Shanghai academic ranking of world universities. Scientometrics, 72(1), 25–32.Google Scholar
  28. Fürnkranz, J., & Hüllermeier, E. (eds.) (2010). Preference Learning. Berlin: Springer.Google Scholar
  29. Giannoulis, C., & Ishizaka, A. (2010). A Web-based decision support system with ELECTRE III for a personalised ranking of British universities. Decision Support Systems, 48(3), 488–497.Google Scholar
  30. Giarlotta, A., & Greco, S. (2013). Necessary and possible preference structures. Journal of Mathematical Economics, 49(2), 163–172.Google Scholar
  31. Govindan, K., & Jepsen, M. B. (2016). ELECTRE: A comprehensive literature review on methodologies and applications. European Journal of Operational Research, 250(1), 1–29.Google Scholar
  32. Grabisch, M. (1996). The application of fuzzy integrals in multicriteria decision making. European Journal of Operational Research, 89(3), 445–456.Google Scholar
  33. Grabisch, M. (1997). k-order additive discrete fuzzy measures and their representation. Fuzzy Sets and Systems, 92(2), 167–189.Google Scholar
  34. Grabisch, M., & Labreuche, C. (2010). A decade of application of the Choquet and Sugeno integrals in multi-criteria decision aid. Annals of Operations Research, 175(1), 247–290.Google Scholar
  35. Greco, S., Figueira, J. R., & Ehrgott, M. (2016). Multiple criteria decision analysis: State of the art surveys. Berlin: Springer.Google Scholar
  36. Greco, S., Matarazzo, B., & Słowiński, R. (2001). Rough sets theory for multicriteria decision analysis. European Journal of Operational Research, 129(1), 1–47.Google Scholar
  37. Greco, S., Mousseau, V., & Słowiński, R. (2008). Ordinal regression revisited: Multiple criteria ranking using a set of additive value functions. European Journal of Operational Research, 191(2), 416–436.Google Scholar
  38. Greco, S., Mousseau, V., & Słowiński, R. (2014). Robust ordinal regression for value functions handling interacting criteria. European Journal of Operational Research, 239(3), 711–730.Google Scholar
  39. Ishizaka, A., Resce, G., & Mareschal, B. (2018). Visual management of performance with PROMETHEE productivity analysis. Soft Computing, 22, 7325–7338.Google Scholar
  40. Jacquet-Lagrèze, E., & Siskos, Y. (2001). Preference disaggregation: 20 years of MCDA experience. European Journal of Operational Research, 130(2), 233–245.Google Scholar
  41. Jeremic, V., Bulajic, M., Martic, M., & Radojicic, Z. (2011). A fresh approach to evaluating the academic ranking of world universities. Scientometrics, 87(3), 587–596.Google Scholar
  42. Johnes, G. (2013). Efficiency in English higher education institutions revisited: A network approach. Economics Bulletin, 33(4), 2698–2706.Google Scholar
  43. Jovanovic, M., Jeremic, V., Savic, G., Bulajic, M., Martic, M. (2012). How does the normalization of data affect the ARWU ranking? Scientometrics, 93(2), 319–327.Google Scholar
  44. Katharaki, M., & Katharakis, G. (2010). A comparative assessment of Greek universities’ efficiency using quantitative analysis. International Journal of Educational Research, 49(4–5), 115–128.Google Scholar
  45. Keeney, R. L., & Raiffa, H. (1976). Decisions with multiple objectives: Preferences and value tradeoffs. New York: Wiley.Google Scholar
  46. Kempkes, G., & Pohl, C. (2010). The efficiency of German universities - some evidence from nonparametric and parametric methods. Applied Economics, 42(16), 2063–2079.Google Scholar
  47. Kounetas, A., Anastasiou, K., Mitropoulos, P., & Mitropoulos, I. (2011). Departmental efficiency differences within a Greek university: An application of a DEA and Tobit analysis. International Transactions in Operational Research, 18(5), 545–559.Google Scholar
  48. Lahdelma, R., Hokkanen, J., & Salminen, P. (1998). SMAA - stochastic multiobjective acceptability analysis. European Journal of Operational Research, 106(1), 137–143.Google Scholar
  49. Leskinen, P., Viitanen, J., Kangas, A., & Kangas, J. (2006). Alternatives to incorporate uncertainty and risk attitude in multicriteria evaluation of forest plans. Forest Science, 52(3), 304–312.Google Scholar
  50. Liu, N. C., & Cheng, Y. (2005). The academic ranking of world universities. Higher Education in Europe, 30(2), 127–136.Google Scholar
  51. March, J. G. (1978). Bounded rationality, ambiguity, and the engineering of choice. The Bell Journal of Economics, 9, 587–608.Google Scholar
  52. Marginson, S. (2014). University rankings and social science. European Journal of Education, 49(1), 45–59.Google Scholar
  53. Marichal, J. L., & Roubens, M. (2000). Determination of weights of interacting criteria from a reference set. European Journal of Operational Research, 124(3), 641–650.Google Scholar
  54. Meredith, M. (2004). Why do universities compete in the ratings game? An empirical analysis of the effects of the US news and world report college rankings. Research in Higher Education, 45(5), 43–461.Google Scholar
  55. Moed, H. F. (2017). A critical comparative analysis of five world university rankings. Scientometrics, 110(2), 967–990.Google Scholar
  56. Mousseau, V., Figueira, J. R., Dias, L., Gomes da Silva, C., & Climaco, J. (2003). Resolving inconsistencies among constraints on the parameters of an MCDA model. European Journal of Operational Research, 147(1), 72–93.Google Scholar
  57. Murofushi, S., & Soneda, T. (1993). Techniques for reading fuzzy measures (III): Interaction index. In: 9th Fuzzy Systems Symposium, Sapporo, Japan (pp. 693–696).Google Scholar
  58. Nazarko, J., & Šaparauskas, J. (2014). Application of DEA method in efficiency evaluation of public higher education institutions. Technological and Economic Development of Economy, 20(1), 25–44.Google Scholar
  59. Olcay, G. A., & Bulu, M. (2017). Is measuring the knowledge creation of universities possible? A review of university rankings. Technological Forecasting and Social Change, 123, 153–160.Google Scholar
  60. Pietrucha, J. (2018). Country-specific determinants of world university rankings. Scientometrics, 114(3), 1129–1139.Google Scholar
  61. QS. (2018). World universities rankings.
  62. Rosenmayer, T. (2014). Using data envelopment analysis: A case of universities. Review of Economic Perspectives, 14(1), 34–54.Google Scholar
  63. Rota, G. C. (1964). On the foundations of combinatorial theory I. Theory of Möbius functions. Wahrscheinlichkeitstheorie und Verwandte Gebiete, 2, 340–368.Google Scholar
  64. Roy, B., & Słowiński, R. (2013). Questions guiding the choice of a multicriteria decision aiding method. EURO Journal on Decision Processes, 1(1), 1–29.Google Scholar
  65. Saisana, M., d’Hombres, B., & Saltelli, A. (2011). Rickety numbers: Volatility of university rankings and policy implications. Research Policy, 40(1), 165–177.Google Scholar
  66. Serow, R. C. (2000). Research and teaching at a research university. Higher Education, 40(4), 449–463.Google Scholar
  67. Sexton, T. R., Comunale, C. L., & Gara, S. C. (2012). Efficiency-based funding for public four-year colleges and universities. Education Finance and Policy, 7(3), 331–359.Google Scholar
  68. Shafer, G. (1976). A Mathematical Theory of Evidence. Princeton: Princeton University Press.Google Scholar
  69. Shapley, L. S. (1935). A value for n-person games. In H. W. Kuhn, & A. W. Tucker (Eds.) Contributions to the Theory of Games II (pp. 307–317). Princeton: Princeton University Press.Google Scholar
  70. Smith, R.L. (1984). Efficient Monte Carlo procedures for generating points uniformly distributed over bounded regions. Operations Research, 32, 1296–1308.Google Scholar
  71. Soh, K. (2015). What the overall doesn’t tell about world university rankings: Examples from ARWU, QSWUR, and THEWUR in 2013. Journal of Higher Education Policy and Management, 37(3), 295–307.Google Scholar
  72. Soh, K. (2017). The seven deadly sins of world university ranking: A summary from several papers. Journal of Higher Education Policy and Management, 39(1), 104–115.Google Scholar
  73. Tervonen, T., & Figueira, J. R. (2008). A survey on stochastic multicriteria acceptability analysis methods. Journal of Multi-Criteria Decision Analysis, 15(1–2), 1–14.Google Scholar
  74. Tervonen, T., Van Valkenhoef, G., Bastürk, N., & Postmus, D. (2014). Hit-and-run enables efficient weight generation for simulation-based multiple criteria decision analysis. European Journal of Operational Research, 224, 552–559.Google Scholar
  75. THE. (2018). The rankings of the times higher education supplement.
  76. Tochkov, K., Nenovsky, N., & Tochkov, K. (2012). University efficiency and public funding for higher education in Bulgaria. Post-Communist Economies, 24(4), 517–534.Google Scholar
  77. U-Multirank. (2018). Universities compared. Your way.
  78. Usher, A., & Medow, J. (2009). A global survey of university rankings and league tables. In B. M. Kehm & B. Stensaker (Eds.) University rankings, diversity, and the new landscape of higher education (pp. 3–18). Rotterdam: Sense Publishers.Google Scholar
  79. Van Valkenhoef, G., Tervonen, T., & Postmus, D. (2014). Notes on “Hit-And-Run enables efficient weight generation for simulation-based multiple criteria decision analysis”. European Journal of Operational Research, 239, 865–867.Google Scholar
  80. Wakker, P. P. (1989). Additive representations of preferences: A new foundation of decision analysis. Berlin: Springer.Google Scholar
  81. Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E., Tijssen, R. J. W., Eck, N. J., Leeuwen, T. N., Raan, A. F. J., Visser, M. S., & Wouters, P. (2012). The Leiden ranking 2011/2012: Data collection, indicators, and interpretation. Journal of the Association for Information Science and Technology, 63(12), 2419–2432.Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Salvatore Corrente
    • 1
    Email author
  • Salvatore Greco
    • 1
    • 2
  • Roman Słowiński
    • 3
    • 4
  1. 1.Department of Economics and BusinessUniversity of CataniaCataniaItaly
  2. 2.University of Portsmouth, Portsmouth Business SchoolCentre of Operations Research and Logistics (CORL)PortsmouthUK
  3. 3.Institute of Computing SciencePoznań University of TechnologyPoznańPoland
  4. 4.Systems Research InstitutePolish Academy of SciencesWarsawPoland

Personalised recommendations