Robust Ranking of Universities Evaluated by Hierarchical and Interacting Criteria

Abstract

In this chapter, we present a methodology of decision aiding that helps to build a ranking of a finite set of alternatives evaluated by a family of hierarchically structured criteria. The presentation has a tutorial character, and takes as an example the ranking of universities. Each university is generally evaluated on several aspects, such as quality of faculty and research output. Moreover, their performance on these macro-criteria can be further detailed by evaluation on some subcriteria. To take into account the hierarchical structure of criteria presented as a tree, the multiple criteria hierarchy process will be applied. The aggregation of the university performances will be done by the Choquet integral preference model that is able to take into account the possible negative and positive interactions between the criteria at hand. On the basis of an indirect preference information supplied by the decision maker in terms of pairwise comparisons of some universities, or comparison of some criteria in terms of their importance and their interaction, the robust ordinal regression and the stochastic multicriteria acceptability analysis will be used. They will provide the decision maker some robust recommendations presented in the form of necessary and possible preference relations between universities, and in the form of a distribution of possible rank positions got by each of them, taking into account all preference models compatible with the available preference information. The methodology will be presented step by step on a sample of some European universities.

Appendix: An Example of the Application of the Hit-And-Run Method

In this section we shall show how the HAR method works by means of an easily replicable didactic example. Let us suppose that we have to sample from a 2D region delimited by the linear constraints x = −2, x = 2, y = −2, and y = 2. In this way, the considered region is delimited by the four vertices A, B, C, and D shown in Fig. 4a. At first, a point is taken randomly inside the region. We denoted it by P in the figure. Then, a random direction d is sampled, by sampling a point from the 2D-sphere¹¹ (in this case therefore a circle). The line passing for P and having the direction d, denoted by d1 in the figure, intersects the boundaries of the region, in particular the lines y = 2 and y = −2 in the points Q and R, respectively. Therefore, a random point is selected inside the segment which extreme points are Q and R previously obtained. The point P1 is therefore the second point of our sampling procedure. To perform the second iteration, we proceed in the same way. We sample a random direction d and we consider the line, d2 in Fig. 4b, passing for P1 and having direction d. The line intersects the boundaries of the region, in particular the lines y = −2 and x = 2 in the points S and T, respectively. Then, a point inside the segment which extremes are S and T is sampled in a random way. This second sampled point is denoted as P2 in Fig. 4b. The iterative procedure continues step by step until the maximum number of iterations has been reached. In Fig. 4c, all sampled points (10,000) of our didactic example are shown.

Fig. 4

Didactic explanation of the Hit-And-Run method. (a) First iteration. (b) Second iteration. (c) Final sampling