Post factum analysis for robust multiple criteria ranking and sorting
 1.8k Downloads
 10 Citations
Abstract
Providing partial preference information for multiple criteria ranking or sorting problems results in the indetermination of the preference model. Investigating the influence of this indetermination on the suggested recommendation, we may obtain the necessary, possible and extreme results confirmed by, respectively, all, at least one, or the most and least advantageous preference model instances compatible with the input preference information. We propose a framework for answering questions regarding stability of these results. In particular, we are investigating the minimal improvement that warrants feasibility of some currently impossible outcome as well as the maximal deterioration by which some already attainable result still holds. Taking into account the setting of multiple criteria ranking and sorting problems, we consider such questions in view of pairwise preference relations, or attaining some rank, or assignment. The improvement or deterioration of the sort of an alternative is quantified with the change of its performances on particular criteria and/or its comprehensive score. The proposed framework is useful in terms of design, planning, formulating the guidelines, or defining the future performance targets. It is also important for robustness concern because it finds which parts of the recommendation are robust or sensitive with respect to the modification of the alternatives’ performance values or scores. Application of the proposed approach is demonstrated on the problem of assessing environmental impact of main European cities.
Keywords
Multiple criteria decision aiding Additive value function Sensitivity analysis Linear programming Distancebased uncertainty analysis Performance values1 Introduction
The concept of criterion plays a fundamental role in decision aiding. Serving as a tool for evaluating and comparing alternatives, a criterion represents a specific point of view on the impact and quality of alternative decisions. As noted by Hyde and Maier [17], the performance values that are assigned to each alternative for every criterion are obtained from models, available data, or by expert judgment based on previous knowledge or experience.
In the presence of multiple conflicting criteria, decision aiding is performed with the use of some explicit usually formalized models and methods. A crucial element of these methods concerns dealing with preferences of the Decision Maker (DM) in a way that ensures that a recommended decision is as consistent with the DM’s objectives and value system as possible.
Indirect and imprecise preference information The majority of the recently proposed Multiple Criteria Decision Aiding (MCDA) methods admit partial preference information provided by the DM at the input. In particular, one may elicit holistic judgments, such as pairwise comparisons of alternatives or criteria (see, e.g., [12, 29]), assignmentbased pairwise comparisons [19], assignment examples (see, e.g., [9, 33]), rankrelated requirements [22], or desired classcardinalities (see, e.g., [23, 42]). Furthermore, one may also specify some imprecise statements, like lower and upper bounds for comprehensive scores (see, e.g., [40]) or preference ratios (see, e.g., [28]). In any case, elicitation of partial preferences requires less cognitive effort on the part of the DM than providing precise values for the preference model parameters.
Robustness analysis in multiple criteria decision aiding When using partial preference information, the DM’s preference model is defined imprecisely. Depending on the selected compatible model instance, the recommendation for the set of alternatives may vary significantly. To investigate its robustness, several approaches have been proposed. In the context of ranking problems, they indicate the range of ranks attained by each alternative (see [21]), verify the possibility and necessity of the pairwise preference relations by checking if they hold for at least one or all compatible preference model instances, respectively (see, e.g., [12, 16, 28]), or indicate various intensity measures of the dominance relation (see, e.g., [2]). For multiple criteria sorting, analysis in terms of the possible and the necessary may be referred to class assignments (see [13, 27, 33]) or assignmentbased preference relations (see [26]). Methods that employ the whole set of compatible preference model instances usually admit incremental specification of preference information in the course of an interactive procedure. Indeed, analysis of the robust results provokes reaction of the DM who may add a new or revise the old preference information. As noted by Corrente et al. [8], such an interactive process ends when the yielded necessary, possible, or extreme recommendation is decisive and convincing for the DM.
The core of post factum analysis In this paper, we propose a methodology of post factum analysis that is designed for use once recommendation has been worked out. Without loss of generality, we focus on an additive value function preference model which is of particular interest in MCDA for an intuitive interpretation of numerical scores of alternatives and a straightforward impact of pieces of preference information on the final result. Thus, in our case, a single preference model instance is a value function composed of marginal value functions with a precisely established course (shape).

in case a is not even possibly preferred to b, what is the minimal improvement of performances of a that guarantees the truth of the possible preference;

in case a is possibly assigned to classes \(C_2\)–\(C_4\), what is the minimal improvement of the comprehensive value (score) of a that warrants that it is assigned to class at least as good as \(C_3\) for all compatible value functions;

in case a is ranked first for the most advantageous value function, what is the maximal deterioration of its performances on criteria \(g_1\) and \(g_2\), such that a is still possibly ranked at the very top;

in case a is assigned to a class at least as good as b for at least one compatible value function, what is the maximal deterioration of the comprehensive value (score) of a for which this possible assignmentbased preference relation is still satisfied.
We perceive the proposed post factum analysis as a complementary methodology to MCDA which, until now, concentrated mainly on providing a recommendation based on a preference model compatible with preference information provided by the DM. While robustness analysis of the obtained recommendations concerned validity of conclusions with respect to allowed changes of either performances or preference model parameters, the post factum analysis extends the concept of such analysis to other useful questions, like “what improvement on all or some performances of a given alternative should be made, so that it achieves a better result in the recommendation obtained with a set of compatible preference models?”, or “what is the margin of safety in some or all performances of a given alternative, within which it can maintain the same rank or class assignment as in the obtained robust recommendation?”. Answering this type of questions is very useful for the DM who wants to assess the opportunities and threats for particular alternatives. Such a preoccupation is typical for environmental management, engineering design, business consult, marketing, and public sector institutions.
Relation with data envelopment analysis Investigation of the improvement and deterioration of the alternative’s performances and/or comprehensive value that grants, respectively, achievement or maintenance of some target result has received a limited attention in MCDA (see [5, 17, 39, 41], and a review in Sect. 4). Moreover, among these approaches there is none that would investigate the impact of the change of alternatives’ performances or scores on the robustness of delivered recommendation.
Nevertheless, this type of analysis is at the core of Data Envelopment Analysis (DEA) (see [6]). DEA is a technique for measuring the relative efficiency of Decision Making Units (DMUs) that use similar inputs to produce similar outputs where the multiple inputs and outputs are incommensurate in nature. On one hand, the basic algorithms for checking if DMU is efficient, measure a “radial distance” of DMU from an efficient frontier. This distance can be interpreted as the coefficient by which DMU’s outputs should be multiplied (or DMU’s inputs need to be divided) in order to make it efficient (see [3]). In this way, DEA provides a measure of efficiency for each DMU, at the same time indicating for the nonefficient ones their “efficient peers”, i.e., the best practices to follow. Recently, DEA has been generalized to Ratiobased Efficiency Analysis (REA) which derives its results from all feasible output and input weights (see [34]). REA extends conventional efficiency scores by computing efficiency bounds, pairwise dominance relations, ranking intervals, and specification of performance targets. The latter is closely related to the necessary and possible improvements that we consider in view of multiple criteria ranking. On the other hand, if some DMU is efficient, neither an increase of any output nor the decrease of any input can change its efficiency status. Nevertheless, it can afford a limited increase in input or decrease in output, still remaining efficient after the change. Similar concerns have been raised with respect to uncertain coefficients with computation of the stability intervals which guarantee that the results do not change. Robustness analysis of this type has been conducted by, e.g., [11, 35, 43, 44].
Organization of the paper The organization of the paper is the following. In the next section, we introduce basic concepts and notation that will be used in the paper. In Sect. 3, we remind selected valuebased multiple criteria ranking and sorting methods based on partial preference information. In Sect. 4, we review the existing multiple criteria sensitivity analysis methods investigating the impact of varying the performance values on the delivered recommendation. Moreover, we refer to a few management and environmental applications to which our post factum analysis can be applied. Section 5 introduces the framework for post factum analysis. For the purpose of illustration, in Sect. 6, we consider the problem of evaluating environmental impact of main European cities. The last section concludes the paper.
2 Notation

\(A=\{ a_1, a_2, \ldots , a_i, \ldots , a_n \}\)—a finite set of n alternatives;

\(G = \{ g_1, g_2,\ldots , g_j, \ldots , g_m \}\)—a finite set of m performance (evaluation) criteria, \(g_j: A \rightarrow \mathbb {R}\) for all \(j \in J=\{ 1,2, \ldots , m \}\);

\(A^R = \{ a^{*}, b^{*}, \ldots \}\)—a finite set of reference alternatives; in general, \(A^R\) may consist of past decision alternatives or fictitious alternatives, consisting of performances on the criteria which can be easily judged by the DM to express his/her comprehensive judgment; we assume, however, that \(A^R \subseteq A\) is a subset of alternatives relatively wellknown to the DM on which (s)he accepts to express holistic preferences;

\(C_h, \ h=1, \ldots , p\)—predefined preference ordered classes such that \(C_{h+1}\) is preferred to \(C_h\), \(h=1, \ldots , p1\); \(H=\{1, 2, \ldots , p\}\);

\(X_j = \{ x_j \in \mathbb {R}: g_j(a_i)=x_j, a_i \in A \}\)—the set of all different performance values on \(g_j\), \(j \in J\); we assume, without loss of generality, that the greater \(g_j(a_i)\), the better alternative \(a_i\) on criterion \(g_j\), for all \(j \in J\);

\(x_j^1, x_j^2, \ldots , x_j^{n_j(A)}\)—the ordered values of \(X_j\), \(x_j^k < x_j^{k+1}, k=1, \ldots , n_j(A)1\), where \(n_j(A) = X_j\) and \(n_j(A) \le n\); consequently, \(X =\prod _{j=1}^m X_j\) is the performance space;

\(g_{j,*}\) and \(g_{j}^{*}\) are, respectively, the worst and the best possible performances on \(g_j\); \(g_{j,*} = x_j^1\) and \(g_j^* = x_j^{n_j(A)}\) or, if provided, \(g_{j,*}\) and \(g_j^*\) are, respectively, the lower and upper bounds for the performance scale on \(g_j\).
3 Reminder on the necessary, possible, and extreme results
When using partial preference information, there is usually more than a single model compatible with the DM’s judgments. To avoid any arbitrary selection, the prevailing trend in MCDA consists in taking into account the whole set of instances. Examination of its application on the set of alternatives A leads to identifying the necessary and possible consequences confirmed by all or at least one compatible preference model instance, respectively. Additionally, one may also compute the extreme recommendation for each alternative indicating the result observed for it in the most and least advantageous cases. While there are several methods for generating partial preference information, without loss of generality, in this section we will refer only to the basic holistic judgments. These are pairwise comparisons for multiple criteria ranking and assignment examples for sorting problems. In this section, we recall the basic ideas for multiple criteria ranking and sorting with a set of value functions.
3.1 Multiple criteria ranking with a set of value functions

Necessary weak preference relation, \(\succsim ^N\), that holds for a pair of alternatives \((a,b) \in A \times A\), in case \(U(a) \ge U(b)\) for all compatible value functions;

Possible weak preference relation, \(\succsim ^P\), that holds for a pair of alternatives \((a,b) \in A \times A\), in case \(U(a) \ge U(b)\) for at least one compatible value function.
Extreme ranks By considering all complete preorders established by value functions compatible with the preference information, we may also determine the best \(P^{*}(a)\) and the worst \(P_{*}(a)\) ranks attained by each alternative \(a \in A\). Identification of these extreme ranks requires solving some MixedInteger Linear Programming (MILP) problems presented by [21].
3.2 Multiple criteria sorting with a set of value functions
When it comes to methods designed for dealing with multiple criteria sorting problems, let us focus on a thresholdbased sorting procedure, where the limits between consecutive classes \(C_h\), \(h=1,\ldots , p\), are defined by a vector of thresholds \(\mathbf t = \{ t_1, \ldots , t_{p1} \}\) such that \(0 < t_1 < \ldots < t_{p1} < 1 \), and \(t_{h1}\) and \(t_h\) are, respectively, the lower and the upper threshold of class \(C_h, \ h =2, \ldots , p1\) (see, e.g., [45]). These thresholds concern the scores assigned to alternatives by a value function. Note that \(t_1\) is an upper threshold of class \(C_1\) while the lower threshold is 0, and \(t_{p1}\) is a lower threshold of class \(C_p\) while the upper threshold is \(>\)1. Thus, the DM preferences are represented with a pair \((U,\mathbf t )\), where U is an additive value function and \(\mathbf t \) is a vector of thresholds delimiting the classes. Alternative a is assigned to class \(C_h\) \((a \rightarrow C_h)\) iff \(U(a) \in [t_{h1},t_{h}]\).

The possible assignment \(C_P(a) = [L_P(a), R_P(a)]\) is defined as the set of indices of classes \(C_h\) for which there exists at least one compatible pair \((U,\mathbf t )\) assigning a to \(C_h\) (denoted by \(a \rightarrow ^P C_h\));

The necessary assignment \(C_N(a) = [L_N(a), R_N(a)]\) as the set of indices of classes \(C_h\) for which all compatible pairs \((U,\mathbf t )\) assign a to \(C_h\) (denoted by \(a \rightarrow ^N C_h\)).
Necessary and possible assignmentbased preference relations Given a set of compatible preference model instances, the possible assignmentbased preference relation \(a \succsim ^{\rightarrow ,P} b\) holds if a is assigned to a class at least as good as class of b for at least one compatible model, and the necessary assignmentbased preference relation \(a \succsim ^{\rightarrow ,N} b\) is true if a is assigned to a class at least as good as class of b for all compatible models. To verify the truth or falsity of these relations, we need to consider the MILP problems presented by Kadziński and Tervonen [26].
4 Review of multiple criteria sensitivity analysis methods investigating the impact of varying the performance values on the delivered recommendation
The majority of multiple criteria sensitivity analysis methods assess the impact of uncertainty in the preference model parameters (e.g., criteria weights) on the delivered recommendation [7]. In this section, we review few existing approaches which investigate the influence of variability in the performance values on the ranking of alternatives. Then, we refer to their realworld applications.
The early sensitivity analysis methods in this stream are limited to varying the alternative’s performance value on a single criterion. In particular, when using the PROMETHEE method, [41] verified how much a selected value needs to be improved to make the considered alternative ranked first. While considering three different multiple criteria aggregation procedures (the weighted sum model, the weighted product model, and the Analytic Hierarchy Process (AHP)), Triantaphyllou and Sanchez [39] investigated what is the minimum change in the performance value leading to an inversion of the currently observed preference relation for a pair of alternatives. These results were further used to indicate how critical the various performance values (in terms of a single criterion at a time) are in the ranking attained by the alternative.
The more recent methods for sensitivity analysis admit simultaneous variation of several performance values. Hyde and Maier [17, 18] verified what is the minimum modification of the performances to improve score of the alternative so that it becomes weakly preferred to another alternative. Such result is derived from solving an optimization problem which minimizes the distance between the original and optimized performance values. In the proposed setting, one considered both PROMETHEE and the weighted sum method. Moreover, the user was asked to choose between different distance metrics including a Euclidean distance, a Manhattan distance, and a KullbackeLeibler distance. In the same spirit, Beynon and Barton [4] and Beynon and Wells [5] identified the minimum (lean) changes necessary to the criteria values of a considered lower ranked alternative that improve its rank to that of a comparatively higher ranked alternative. Precisely, they considered the problem of minimization of a Euclidean distance in the context of PROMETHEE.
Yet another stream of algorithms admits variation of original performance values and investigates the consequence of thus imposed uncertainty using Monte Carlo simulation (see, e.g., [17, 32]). In this way, one can check the synthetic effect of varying different performance values (within some admissible predefined range or with respect to an assumed distribution) on attaining some target rank or comprehensive score.
 Attaining a particular rank (either the top rank or an arbitrarily selected position):

Wolters and Mareschal [41] considered different designs of a heat exchanger network in terms of modifying the underlying investment costs. The results were used for granting additional funds on a specific alternative.

Beynon and Wells [5] considered a set of motor vehicles in terms of the minimum necessary engineering performance modification in their emission levels. The results were interpreted as the guidelines to be attained by future engineering work in the process of engine calibration and other adjustments.

Beynon and Barton [4] considered a set of individual police forces in the United Kingdom in terms of the minimum changes to sanction detection levels (clear up rates) that need to be attained. The outcomes were useful for strategy planning, offering quantitative evidence of course.

 Inversion of a pairwise preference relation (attaining a rank equivalence):

Hyde and Maier [17] considered alternative water management options in terms of changing their environmental (e.g., efficient water use and reuse), social (e.g., clean industry and employment), and economic (e.g., true or full cost pricing) factors. Their results were helpful in determining the development strategy that offers a more sustainable future for the area.

Ravalico et al. [31] considered different locations of a salt interception scheme (SIS) in the Murray river in terms of changing the parameters related to the travel time and dead storage in the reaches of the river between some locks. This allowed to assess the sensitivity of the preferred location of each SIS and give the DMs a level of confidence in the ranking of these alternatives.

Chen et al. [7] considered a set of management options of diffuse pollution in the Taman river catchment in terms of adjusting the application of fertilisers, stocking density, and land use distribution, restoring wetland area, and increasing the area of nonfarmed land. The outcomes were used to judge the robustness of the original results with respect to variation of performance values.

Rocco and Tarantola [32] considered a set of public investments in refineries, bridges, or petroleum exploration in terms of modifying their financial, economic, and social aspects (e.g., benefitcost ratio, compensation of employees or remunerations, and employment).

5 Post factum analysis
As said before, exploitation of the set of preference model instances compatible with DM’s preferences results in the necessary, possible, and extreme recommendations. Their analysis may stimulate questions concerning the stability of the provided outcomes and conditions under which some parts of the considered recommendation become or remain true. We will call the proposed framework by “post factum analysis”, because it should be employed when some recommendation has been already produced in result of an interaction between the DM (possibly assisted by an analyst) and the method.
 in case of multiple criteria ranking:

the truth of a preference relation for an ordered pair of alternatives: \(a \succsim b\),

the truth of a preference of an alternative with respect to a group of at least two other alternatives: \(a \succsim b, \forall b \in A'\),

attaining a particular rank: \(P^*(a) \le k\) or \(P^*(a) \le k\),

 in case of multiple criteria sorting:

the truth of an assignmentbased preference relation for an ordered pair of alternatives: \(a \succsim ^{\rightarrow } b\),

attaining a particular class assignment: \(L_P(a) \ge k\) or \(R_P(a) \ge k\).

We will analyze conditions that need to be satisfied to achieve or preserve some target from two perspectives. Firstly, we will take into account performances of alternatives. Secondly, we will investigate the change of a comprehensive value (score) of an alternative rather than direct improvement or deterioration of its performances. These complementary perspectives offer a diverse view on the robustness of the delivered recommendation.
In this section, we formalize the framework of post factum analysis by introducing the notions of the possible and necessary improvement, deterioration, missing and surplus values. Let us emphasize that all definitions are conditioned by the use of both a particular preference model and preference information provided by the DM. We will also present the procedures for computing these measures for each out of five accounted targets referring to the truth of a preference relation or assignmentbased preference relation or a group of preference relations as well as to attaining some rank or assignment.

We consider a set of preference model instances compatible with the DM’s indirect preference information for deriving the base recommendation (see Sect. 3) and subsequent formulation of the targets to be attained. On the contrary, the vast majority of existing approaches employ just a single preference model instance with precise parameter values, thus failing to investigate the impact of varying the alternatives’ performances or scores on the robustness of the delivered recommendation.

We formulate a wide spectrum of targets taking into account the specificity of both multiple criteria ranking and sorting problems. We extend the types of targets that can be considered in context of the former, and propose the targets that are of interest in terms of the latter.

Apart from investigating the improvement that needs to be made so that to attain some target (see Sects. 5.2.1, 5.3.1), we enrich the existing methods for sensitivity analysis by considering admissible deterioration by which some already attained target is maintained (see Sects. 5.2.3, 5.3.2).

We investigate two scenarios of attaining/maintaining the targets in terms of the necessary (for all compatible preference model instances) and the possible (for at least one compatible preference model instance). A related setting has been discussed in [31], where the question of changing the rank of an alternative is considered by dividing the space of compatible preference model instances into the three regions where the preference relation is, respectively, unaffected, inverse, and the boundary region separating the previous ones.

We discuss different procedures for changing the alternative’s performance values (see Sect. 5.2.2), primarily focusing on adapting the approach postulated in DEA which consists in multiplying the performances by a common factor (see Sects. 5.2.1, 5.2.3).

Apart from assessing the impact of varying the performance values (see Sect. 5.2), we show how to attain/maintain some target by changing directly the alternative’s comprehensive score (value) (see Sect. 5.3). Such change may be either treated as the result per se or further decomposed into the required/admissible variation of performance values. A related algorithm which admits varying a marginal score on a single criterion only, has been proposed in [7].
5.1 Notation used in post factum analysis

\(W \in \{ \rho , u \}\), with \(\rho \) representing the change of alternative’s performances and u representing the change of alternative’s comprehensive value;

\(X \in \{P, N\}\), with P and N representing, respectively, the possibility or the necessity of achieving/preserving a target;

\(Y \in \{>, <\}\), with \(>\) representing investigation of an improvement that needs to be made in case some target has to be achieved, and \(<\) representing investigation of a deterioration that can be afforded in case some target needs to be preserved;

\(Z \in \{(a \succsim b), (a \succsim b, \forall b \in A'), (P^*(a) \le k), (P_*(a) \le k), (R_P(a) \ge k), (L_P(a) \ge k), (a \succsim ^{\rightarrow } b) \}\) represents the target under consideration.

\(\rho ^P_{>}(P^*(a) \le k)\) is the rankrelated possible improvement, i.e., the improvement \((>)\) of the performances of a \((\rho )\) that is required so that its best rank \((P^*(a))\) is not worse than k for at least one compatible value function (P);

\(u^N_{<}(a \succsim b)\) is the preferencerelated necessary surplus value, i.e., the maximal value (u) that when subtracted \((<)\) from the comprehensive value of a still admits a to be preferred by b \((a \succsim b)\) for all compatible value functions (N).
5.2 Changing performances of an alternative
In this subsection, we will analyze the change of alternative’s performances that needs to be made to achieve or preserve some target. On one hand, when a certain target needs to be achieved, it means that current performances of an alternative on all or some criteria are not sufficiently good and should be improved. Obviously, we are interested in the minimal improvement that guarantees achieving a designated target. On the other hand, if some target has been already acquired, an alternative can possibly afford some deterioration of its performances not exerting an influence on the target maintenance. In this case, the maximal deterioration for which a target is still preserved is of interest to the DM.
Although, in general, one may consider different means for quantifying the improvement or deterioration, we believe that the answer provided to the DM should be as simple as possible. Thus, we focus on considering radial increases and decreases of the performances on different criteria by multiplying them by a common factor, respectively, greater or less than one. This approach has been postulated in DEA where the DMU’s outputs or inputs are multiplied by the common coefficient. Its value indicates whether the unit is efficient, i.e., if it is ranked first for some feasible output and input weights. Nevertheless, we also discuss some other distance metrics that can be employed within the proposed framework.
5.2.1 Possible and necessary improvement
Definition 1

preferencerelated possible (necessary) improvement \(\rho ^P_{>}(a \succsim b)\) \((\rho ^N_{>}(a \succsim b))\) in case \(not(a \succsim ^P~b)\) \((not(a \succsim ^N b))\), i.e., a is not possibly (necessarily) preferred to b, and it needs to improve its performances so that \(a \succsim ^P b\) \((a \succsim ^N b)\);

group preferencerelated possible (necessary) improvement \(\rho ^P_{>}(a \succsim b, \forall b \in A')\) \((\rho ^N_{>}(a \succsim ~b, \forall b \in ~A'))\) in case \(\exists b \in A' \subseteq A,\) \(not(a \succsim ^P b)\) \((not(a \succsim ^N b))\), i.e., a is not possibly (necessarily) preferred to at least one alternative b from a subset of alternatives \(A' \subseteq A\), and it needs to improve its performances so that \(\forall b \in A'\), \(a \succsim ^P b\) \((a \succsim ^N b)\);

rankrelated possible (necessary) improvement \(\rho ^P_{>}(P^*(a) \le k)\) \((\rho ^N_{>}(P_*(a) \le k))\) in case \(P^*(a) >~k\) \((P_*(a) > k)\), i.e., the best (worst) rank of a is worse than k, and a needs to improve its performances so that \(P^*(a) \le k\) \((P_*(a) \le k)\);

assignmentrelated possible (necessary) improvement \(\rho ^P_{>}(R_P(a) \ge k)\) \((\rho ^N_{>}(L_P(a)\ge k))\) in case \(R_P(a) < k\) \((L_P(a) < k)\), i.e., the best (worst) possible class of a is worse than \(C_k\), and a needs to improve its performances so that \(R_P(a) \ge k\) \((L_P(a) \ge k)\),

preferencerelated assignmentbased possible (necessary) improvement \(\rho ^P_{>}(a~\succsim ^{\rightarrow }~b)\) \((\rho ^N_{>}(a \succsim ^{\rightarrow }~b))\) in case \(not(a \succsim ^{P,\rightarrow } b)\) \((not(a \succsim ^{N, \rightarrow } b))\), i.e., a is not possibly (necessarily) assigned to a class at least as good as b, and it needs to improve its performances so that \(a \succsim ^{P, \rightarrow } b\) \((a \succsim ^{N, \rightarrow } b)\).

for preferencerelated possible improvement: \(U(a^{\rho }) \ge U(b)\), which guarantees \(a^{\rho } \succsim ^P b\);

for group preferencerelated possible improvement: for all \(b \in A'\), \(U(a^{\rho }) \ge U(b)\), which guarantees \(a^{\rho } \succsim ^P b\) for all \(b \in A'\);

for rankrelated possible improvement: constraint \(U(a^{\rho }) \ge U(b)\) can be relaxed only for up to \(k1\) alternatives \(b \in A \setminus \{a\}\), which guarantees that at most \(k1\) alternatives are at the same time ranked better than \(a^{\rho }\), i.e., \(a^{\rho }\) attains at least kth rank;

for assignmentrelated possible improvement: \(U(a^{\rho }) \ge b_{k1}\), which guarantees that a is assigned to a class not worse than \(C_k\);

for preferencerelated assignmentbased possible improvement: \(U(a^{\rho }) \ge b_{h1}\) and \(U(b) < b_{h}\) for some \(h \in \{1, \ldots , p\}\), which guarantees \(a^{\rho } \succsim ^{P,\rightarrow } b\).
Although different methods can be used to solve this problem, the binary search has been selected because it is guaranteed to converge, the error of identified solution can be controlled, and the algorithm can be easily explained to nonspecialists in computer science and mathematics. Moreover, its application is supported by the specific characteristics of considered problem. First, if the target under consideration can be achieved, there exists only a single solution. As a result, the binary search is guaranteed to identify it, and there is no need to use methods that detect multiple solutions. Secondly, even though the convergence of binary search is generally slow, in post factum analysis the targets to be analyzed should be indicated by the DM and in most realworld problems a limited number of such targets would be of particular interest to her/him.
Let us provide a few important remarks concerning computation of the possible comprehensive improvement. First of all, the need for identifying the improvement exists iff some target is not attained with the current performance vector. That is why, the lower bound for the search can be set to one, and we are guaranteed indication of \(\rho ^P_{>} > 1\) as the solution. Secondly, when multiplying an evaluation \(g_j(a)\) by a value \(\rho ^P_{>} > 1\), it is possible that \(\rho ^P_{>} \cdot g_j(a) > g_j^*\). We assume, however, that an alternative cannot reach an evaluation greater than \(g_j^*\). Thus, when \(\rho ^P_{>} \cdot g_j(a) > g_j^*\), we assume that \(u_j(\rho ^P_{>} \cdot g_j(a))\) is equal to the maximal marginal value on criterion \(g_j\), \(u_j(g_j^*)\).
Remark 1
For any subsets of criteria \(G_1 \subset G_2 \subseteq G\) allowing achievement of the target, \(\rho ^P_{>, G_1} \ge \rho ^P_{>, G_2}.\)

for preferencerelated necessary improvement, \(U(a^{\rho }) < U(b)\); if this cannot be satisfied, \(a^{\rho } \succsim ^N b\);

for group preferencerelated necessary improvement, \(\exists b \in A', U(a^{\rho }) < U(b)\); if this cannot be satisfied, \(a^{\rho } \succsim ^N b\) for all \(b \in A'\);

for rankrelated necessary improvement, constraint \(U(a^{\rho }) < U(b)\) holds for at least k alternatives \(b \in A \setminus \{a\}\); if this cannot be satisfied, there are at most \(k1\) alternatives \(b \in A \setminus \{a\}\) ranked better than \(a^{\rho }\), i.e., \(a^{\rho }\) attains rank not worse than k for all compatible preference model instances;

for assignmentrelated necessary improvement, \(U(a^{\rho }) < b_{k1}\); if this cannot be possibly satisfied, a is assigned to a class at least as good as \(C_k\) for all compatible preference model instances;

for preferencerelated assignmentbased necessary improvement, \(U(b) \ge b_{h}\) and \(U(a^{\rho }) < b_{h}\) and for some \(h \in \{1, \ldots , p1\}\); if \(U(b) \ge b_h > U(a^{\rho })\) cannot be satisfied for any considered h, then \(a^{\rho } \succsim ^{N, \rightarrow } b\).
Remark 2
For any target under consideration, the necessary comprehensive improvement is not less than the corresponding possible improvement, i.e., \(\rho ^N_{>} \ge \rho ^P_{>}.\)
5.2.2 Remarks on standardization of performance values, distance measures, and performance scales
 the Manhattan distance \(d^{M,\rho ^{abs}}_a\) (used, e.g., in [17]; note that in our case \(\rho ^{abs}_j\) is always greater than zero):$$\begin{aligned} d^{M,\rho ^{abs}}_a = \sum _{j=1}^m \rho ^{abs}_j; \end{aligned}$$(11)
 the Chebyshev distance \(d^{Ch,\rho ^{abs}}_a\):$$\begin{aligned} d^{Ch,\rho ^{abs}}_a = max_{j=1,\ldots ,m} \rho ^{abs}_j; \end{aligned}$$(12)
 the sum of relative improvements \(d^{R,\rho }_a\):$$\begin{aligned} d^{R,\rho }_a = \sum _{j=1}^m \rho _j; \end{aligned}$$(13)
 the KullbackeLeibler distance \(d^{KL,\rho }_a\) (i.e., a relative entropy; used, e.g., in [17]):$$\begin{aligned} d^{KL,\rho }_a = \sum _{j=1}^m g_j(a^{\rho }) \cdot ln(\rho _j). \end{aligned}$$(14)
Note, however, that the optimization problems with nonlinear objective functions (10–14) are even more difficult to solve than these discussed in Sect. 5.2.1. Thus, to differentiate the improvements that need to be made on each criterion, one needs to use more advanced heuristic optimization methods than the binary search. For a discussion on employing the genetic algorithms in this particular context see, e.g., [5, 17].
When ordinal performance scale is employed, neither ratios nor differences between the performances or their codifications can be interpreted. Since in this case the linear interpolation between the characteristic points is not meaningful, these criteria should be modeled with the general marginal value functions with the characteristic points corresponding to all different performances values. Moreover, they need to be excluded from changing the performances by means of multiplication or addition. Instead, one can quantify the modification of performance values only in terms of the required change of performance levels (rank orders by which the performances are sorted). A single “shift” in performance corresponds to a change of alternative’s evaluation to a performance above the original one. In this setting, post factum analysis may indicate, e.g., the need for increasing the criteria values by one performance level (e.g., from “average” to “good”) on \(g_1\) and \(g_2\) for reaching the first rank. This is a rough measure, because the shifts between different performance values (e.g., from “bad” to “medium” and from “medium” to “good” on \(g_1\)) and criteria (e.g., from “bad” to “medium” on \(g_1\) and \(g_2\)) are not comparable. Alternatively, one can focus on computing the missing value. This technique is described in Sect. 5.3.1.
All remarks presented in this subsection can be formulated analogously for the case of deteriorating the performance values so that to maintain the already achieved target.
5.2.3 Possible and necessary deterioration
Definition 2
Assume that some target is attained by an alternative \(a \in A\) in the set of preference model instances compatible with DM’s preference information. A comprehensive possible (necessary) deterioration for a in view of maintaining this target is the minimal real number not greater than one by which the performances of a on all criteria need to be multiplied so that the target is still achieved for at least one (all) compatible preference model instance.
Comprehensive possible and necessary deteriorations in view of some specific targets are defined analogously to the corresponding improvements in Definition 1. For example, preferencerelated possible (necessary) deterioration \(\rho ^P_{<}(a \succsim b)\) \((\rho ^N_{<}(a \succsim b))\) is considered in case \(a \succsim ^P b\) \((a \succsim ^N b)\), i.e., a is possibly (necessarily) preferred to b, and a can afford deteriorating its performances while still \(a \succsim ^P b\) \((a \succsim ^N b)\). The formulation of algorithms for computing the possible and necessary deteriorations are the same as in case of the improvements. Obviously, the upper bound for the search can be set to one (thus, we are guaranteed that \(\rho \le 1\) will be indicated as the solution) and the lower bound can be set to \(\text{ min }_{j \in J} \{g_{j,*}/g_j(a)\}\). We also assume that in case \(\rho \cdot g_j(a) < g_{j,*}\), \(u_j(\rho \cdot g_j(a))\) is equal to the minimal marginal value on criterion \(g_j\) equal to \(u_j(g_{j,*})\). Apart from considering comprehensive deteriorations, we can also refer to the partial necessary and possible deteriorations defined analogously to the partial improvements.
Remark 3
For any subsets of criteria \(G_1 \subset G_2 \subseteq G\) which admit deterioration of the performances not exerting an influence on the target maintenance, \(\rho ^P_{<, G_1} \le \rho ^P_{<, G_2}.\)
Remark 4
For any target under consideration, the necessary comprehensive deterioration is not less than the corresponding possible deterioration, i.e., \(\rho ^N_{<} \ge \rho ^P_{<}.\)
5.3 Changing comprehensive value of an alternative
In this subsection, we investigate the change of a comprehensive value (score) of an alternative rather than direct improvement or deterioration of its performances. In particular, we may analyze what minimal value is missing to attain some target or what maximal value an alternative has in stock when this target is already acquired. Let us emphasize that such investigation is meaningful for the considered additive representation of preference, because the scale of the marginal value functions is a conjoint interval scale. Consequently, the difference between two values has the meaning of preference intensity. Thus, for example, a low missing or surplus value indicates the need for small improvement or a small margin for deterioration, respectively. In any case, such analysis aims at indicating the most (least) favorable compatible value function for an alternative in view of achieving (preserving) a target under consideration with the least improvement (the greatest deterioration) of its comprehensive value.
Note that the possible and necessary missing or surplus values may be treated per se as outcomes of post factum analysis. Nevertheless, the most (least) favorable compatible value functions can be further analyzed to derive the underlying required improvement (allowed deterioration) in the performance values for reaching (maintaining) the target. To decompose the missing or surplus values, one can apply the approaches discussed in Sect. 5.2. Then, the necessary and possible outcomes derived from the analysis of a single compatible value function would be equivalent.
5.3.1 Possible and necessary missing value
Definition 3
Assume that some target is not attained by an alternative \(a \in A\) in the set of preference model instances compatible with DM’s preference information. A possible (necessary) missing value for a in view of achieving this target is the minimal positive value (score) that added to the comprehensive value (score) of a allows achieving the target for at least one (all) compatible preference model instance.
5.3.2 Possible and necessary surplus value
Definition 4
Assume that some target is attained by an alternative \(a \in A\) in the set of preference model instances compatible with DM’s preference information. A possible (necessary) surplus value for a in view of achieving this target is the maximal positive value (score) that subtracted from the comprehensive value (score) of a still allows achieving the target for at least one (all) compatible preference model instance.
6 Illustrative study: application of post factum analysis for assessing the environmental impact of cities

\(\hbox {CO}_2\) \((g_1)\), representing the intensity of CO\(_2\) emissions (the observed performances are between 2.49 and 9.58);

energy \((g_2)\), representing the intensity of energy consumption (the observed performances are between 1.50 and 8.71);

water \((g_3)\), representing water efficiency and treatment policy (the observed performances are between 1.83 and 9.21);

waste and land use \((g_4)\), representing waste reduction and treatment policy (the observed performances are between 1.43 and 8.98).
Cities’ performances
City  \(g_1\)  \(g_2\)  \(g_3\)  \(g_4\) 

Oslo  9.58  8.71  6.85  8.23 
Stockholm  8.99  7.61  7.14  7.99 
Zurich  8.48  6.92  8.88  8.82 
Copenhagen  8.35  8.69  8.88  8.05 
Brussels  8.32  6.19  9.05  7.26 
Paris  7.81  4.66  8.55  6.72 
Rome  7.57  6.40  6.88  5.96 
Vienna  7.53  7.76  9.13  8.60 
Madrid  7.51  5.52  8.59  5.85 
London  7.34  5.64  8.58  7.16 
Helsinki  7.30  4.49  7.92  8.69 
Amsterdam  7.10  7.08  9.21  8.98 
Berlin  6.75  5.48  9.12  8.63 
Ljubljana  6.67  2.23  4.19  5.95 
Riga  5.55  3.53  6.43  5.72 
Istanbul  4.86  5.55  5.59  4.86 
Athens  4.85  4.94  7.26  5.33 
Budapest  4.85  2.43  6.97  6.27 
Dublin  4.77  4.55  7.14  6.38 
Warsaw  4.65  5.29  4.90  5.17 
Bratislava  4.54  4.19  7.65  5.60 
Lisbon  4.05  5.77  5.42  5.34 
Vilnius  3.91  2.39  7.71  7.31 
Bucharest  3.65  3.42  4.07  3.62 
Prague  3.44  3.26  8.39  6.30 
Tallinn  3.40  1.70  7.90  6.15 
Zagreb  3.20  4.34  4.43  4.04 
Belgrade  3.15  4.65  3.90  4.30 
Sofia  2.95  2.16  1.83  3.32 
Kiev  2.49  1.50  5.96  1.43 
Assignment examples
\(C_1\)  Lisbon, Prague, Zagreb 
\(C_2\)  Athens, Budapest, Vilnius 
\(C_3\)  Rome, Helsinki, Berlin 
\(C_4\)  Oslo, Stockholm, Brussels 
The possible and the necessary results
When it comes to the necessary assignmentbased preference relation, cities necessarily assigned to the same class are indifferent. Thus, they are grouped within a single node in Fig. 1 (e.g., Oslo, Stockholm, Zurich, and Brussels are all assigned to \(C_4\) with all compatible preference model instances). Moreover, \(\succsim ^{\rightarrow ,N}\) is transitive, and, thus, e.g., Amsterdam \(\succsim ^{\rightarrow ,N}\) Rome and Rome \(\succsim ^{\rightarrow ,N}\) Dublin implicates Amsterdam \(\succsim ^{\rightarrow ,N}\) Dublin (note that the arcs obtainable by the transitive closure are omitted in the figure). On one hand, for pairs of cities (a, b) connected by an arc in Fig. 1 there is the necessary assignmentbased strict preference relation, i.e., \(a \succ ^{\rightarrow ,N} b\) iff \(a \succsim ^{\rightarrow ,N} b\) and \(not(b \succsim ^{\rightarrow ,N} a)\). Note that it does not exclude that \(b \succsim ^{\rightarrow ,P} a\). On the other hand, pairs of alternatives which are not connected by arcs in the figure are incomparable in terms of \(\succsim ^{\rightarrow ,N}\). This means that with some compatible value functions and class thresholds one of them is assigned to a class strictly better than the other, while with some other compatible preference model instances, the order of classes is inverse. This holds for, e.g., Paris and Vienna or Amsterdam and London.
Within the framework of post factum analysis we offer a set of tools for answering different questions regarding robustness of the provided recommendation. In our view, the targets to be analyzed should be indicated by the DM. This means that we shall rather not analyze all possible targets, but only take into account these preference relations, ranks, or assignments which are of particular interest to the DM. Let us illustrate the type of questions that can be answered within the framework of post factum analysis, starting with the analysis of possible improvements and missing values. When investigating the required/allowed modification of performance values, we focus on radial improvements/deteriorations.
Possible assignmentrelated improvements and missing values
Comprehensive assignmentrelated possible improvements \(\rho ^P_{>}(R_P(a) \ge 4)\) and possible missing values \(u^P_{>}(R_P(a) \ge 4)\) for possible assignment of selected alternatives to class at least \(C_4\)
Ljubljana  Riga  Dublin  Bratislava  Warsaw  Istanbul  

\(\rho ^P_{>}(R_P(a) \ge 4)\)  1.1242  1.2865  1.3347  1.4482  1.4514  1.3955 
\(u^P_{>}(R_P(a) \ge 4)\)  0.1150  0.2286  0.2450  0.2647  0.3082  0.2286 
Bucharest  Tallinn  Belgrade  Sofia  Kiev  

\(\rho ^P_{>}(R_P(a) \ge 4)\)  1.9322  1.8823  1.8413  2.4767  2.6864  
\(u^P_{>}(R_P(a) \ge 4)\)  0.4799  0.3180  0.4566  0.5933  0.5288 

Warsaw which is possibly assigned to \(C_2\) in the best case needs to improve its performances 1.0139 or 1.4514 times to possibly reach, respectively, \(C_3\) or \(C_4\); the respective missing values are 0.0095 and 0.3082;

Bucharest which is possibly and necessarily assigned to \(C_1\) needs to improve its performances 1.1240, 1.3541, or 1.9322 times to possibly reach, respectively, \(C_2\), \(C_3\) or \(C_4\); the respective missing values are 0.0641, 0.1812, and 0.4799.
Partial assignmentrelated possible improvements \(\rho ^P_{>, G_j}(R_P(\hbox {Riga}) \ge 4)\) for possible assignment of Riga to \(C_4\)
\(\{g_1\}\)  \(\{g_2\}\)  \(\{g_3\}\)  \(\{g_4\}\)  

1.3672  –  –  – 
\(\{g_1,g_2\}\)  \(\{g_1,g_3\}\)  \(\{g_1,g_4\}\)  \(\{g_2,g_3\}\)  \(\{g_2,g_4\}\)  \(\{g_3,g_4\}\) 

1.3672  1.2865  1.3343  –  1.8496  – 
\(\{g_1,g_2,g_3\}\)  \(\{g_1,g_2,g_4\}\)  \(\{g_1,g_3,g_4\}\)  \(\{g_2,g_3,g_4\}\)  \(\{g_1,g_2,g_3,g_4\}\)  

1.2865  1.2865  1.2865  1.7483  1.2865 
Comprehensive assignmentrelated necessary improvements \(\rho ^N_{>}(L_P(a) \ge 4)\) and necessary missing values \(u^N_{>}(L_P(a) \ge 4)\) for necessary assignment of selected alternatives to class at least \(C_4\)
Copenhagen  Paris  Vienna  London  Amsterdam  Madrid  

\(\rho ^N_{>}(L_P(a) \ge 4)\)  1.0025  1.1186  1.1035  1.1321  1.1703  1.1686 
\(u^N_{>}(L_P(a) \ge 4)\)  0.0030  0.1040  0.1040  0.1337  0.1337  0.1455 
Possible preferencerelated assignmentbased improvements and missing values
When it comes to the possible assignmentbased preference relation, we investigate the improvement that needs to be made to turn its falsity into the truth. For example, Kiev which is assigned to a class worse than Athens by all compatible preference model instances needs to improve its performances 1.3904 times so that Kiev \(\succsim ^{\rightarrow ,P}\) Athens; the respective missing value is 0.1684. When considering Warsaw and Rome, the corresponding results are 1.0139 (for the possible improvement) and 0.0095 (for the possible missing value).
Necessary assignmentrelated improvements and missing values

Madrid which is possibly assigned to \([C_2, C_4]\) needs to improve its performances 1.0402 or 1.1686 times to be necessarily assigned to class at least \(C_3\) or \(C_4\), respectively; the respective missing values are 0.0346 and 0.1455;

Bratislava which is possibly assigned to \([C_1, C_3]\) needs to improve its performances 1.0287 or 1.5016 times to be necessarily assigned to class at least \(C_2\) or \(C_3\), respectively; the respective missing values are 0.0194 and 0.3337.
Partial assignmentrelated necessary improvements \(\rho ^N_{>, G_j}(L_P(\hbox {Istanbul}) \ge 4)\) for necessary assignment of Istanbul to \(C_2\)
\(\{g_1\}\)  \(\{g_2\}\)  \(\{g_3\}\)  \(\{g_4\}\)  

1.3659  1.5693  1.6475  1.8476 
\(\{g_1,g_2\}\)  \(\{g_1,g_3\}\)  \(\{g_1,g_4\}\)  \(\{g_2,g_3\}\)  \(\{g_2,g_4\}\)  \(\{g_3,g_4\}\) 

1.3659  1.1760  1.2806  1.6475  1.8476  1.2545 
\(\{g_1,g_2,g_3\}\)  \(\{g_1,g_2,g_4\}\)  \(\{g_1,g_3,g_4\}\)  \(\{g_2,g_3,g_4\}\)  \(\{g_1,g_2,g_3,g_4\}\)  

1.1714  1.2806  1.1500  1.2545  1.1500 
Let us also illustrate that achieving a certain target for at least one compatible preference model instance is easier than in the necessary sense. For example, Bratislava which is assigned to \(C_3\) in the best case needs to improve its performances 1.4482 or 1.8304 times to be assigned to class \(C_4\), respectively, for at least one or all compatible preference model instances; the respective missing values are 0.2647 and 0.5124. When considering Istanbul whose best possible class is \(C_2\), it would be possibly or necessarily assigned to \(C_3\) in case its performances are improved, respectively, 1.0062 or 1.4797 times, or its comprehensive value is improved by 0.0042 or 0.3249.
Necessary preferencerelated assignmentbased improvements and missing values
When it comes to the improvement granting the truth of the exemplary necessary assignmentbased preference relation, Madrid which is not assigned to a class at least as good as London with all compatible preference model instances needs to improve its performances 1.3904 times so that Madrid \(\succsim ^{\rightarrow ,N}\) London; the respective missing value is 0.1684. The necessary improvement and missing value needed to instantiate the relation: London \(\succsim ^{\rightarrow ,N}\) Madrid, are equal to 1.0139 and 0.0095, respectively. Further, Warsaw needs to improve its performance 1.3347 times (comprehensive value by 0.2450) to be necessarily assigned to a class at least as good as Athens.
Possible assignmentrelated deteriorations and surplus values
Comprehensive assignmentrelated possible deteriorations \(\rho ^P_{<}(R_P(a) \ge 3)\) and surplus values \(u^P_{<}(R_P(a) \ge 3)\) for possible assignment of selected alternatives to class at least \(C_3\)
Zurich  Copenhagen  Paris  Vienna  London  Amsterdam  

\(\rho ^P_{<}(R_P(a) \ge 3)\)  0.5816  0.5925  0.6345  0.6313  0.6718  0.6416 
\(u^P_{<}(R_P(a) \ge 3)\)  0.4995  0.4785  0.4027  0.4053  0.3413  0.3877 
Madrid  Ljubljana  Riga  Dublin  Bratislava  

\(\rho ^P_{<}(R_P(a) \ge 3)\)  0.6590  0.7468  0.8883  0.9359  0.9895  
\(u^P_{<}(R_P(a) \ge 3)\)  0.3633  0.2354  0.0879  0.0475  0.0086 

Madrid which is possibly assigned to \([C_2, C_4]\) can afford deterioration of its performances by 0.9513, 0.6590, or 0.5454 times to possibly maintain the assignment to, respectively, \(C_4\), \(C_3\), or \(C_2\); the respective surplus values are 0.3633, 0.5454 and 0.6590;

Riga which is possibly assigned to \([C_2, C_3]\) can afford deterioration of its performances by 0.8833 or 0.7374 times to possibly maintain assignment to, respectively, \(C_3\) or \(C_2\); the respective surplus values are 0.0879 and 0.2069.
When it comes to the deterioration for the exemplary possible assignmentbased preference relation, Riga which is assigned to a class at least as good as Dublin for at least one compatible preference model instance may deteriorate its performances by 0.7374 times so that still Riga \(\succsim ^{\rightarrow ,P}\) Dublin; the respective missing value is 0.2069. The possible deterioration and surplus value for the relation: Dublin \(\succsim ^{\rightarrow ,P}\) Riga, are equal to 0.08281 and 0.1246, respectively. Thus, the performances and the comprehensive value of Riga are less sensitive with respect to the truth of the possible assignmentbased preference to Dublin than vice versa.
Necessary assignmentrelated deteriorations and surplus values
Comprehensive assignmentrelated necessary deteriorations \(\rho ^N_{<}(L_P(a) \ge 3)\) and surplus values \(u^N_{<}(L_P(a) \ge 3)\) for necessary assignment of selected alternatives to class at least \(C_3\)
Zurich  Copenhagen  Paris  Vienna  London  Amsterdam  

\(\rho ^N_{<}(L_P(a) \ge 3)\)  0.8607  0.8696  0.9951  0.9278  0.9926  0.9632 
\(u^N_{<}(L_P(a) \ge 3)\)  0.1650  0.1511  0.0044  0.0800  0.0071  0.0391 

Zurich which is necessarily assigned to \(C_4\) can afford deterioration of its performances by 0.9913 or 0.8607 times to maintain assignment to, respectively, \(C_4\) and \(C_3\) with all compatible preference model instances; the respective surplus values are 0.0103 and 0.1650;

Amsterdam which is never assigned to class worse than \(C_3\) can afford deterioration of its performances by 0.9632 or 0.7218 times to maintain assignment to, respectively, \(C_3\) and \(C_2\) with all compatible preference model instances; the respective surplus values are 0.0391 and 0.3149.
Referring to an exemplary necessary assignmentbased preference relation, Madrid can afford deterioration of its performances by 0.7218 times and its comprehensive value by 0.3149 while still being necessarily preferred to Budapest. When comparing Dublin with Vilnius, the corresponding results are 0.9921 (for the necessary deterioration) and 0.0064 (for the necessary surplus value).
Finally, let us emphasize that maintaining the target with at least one compatible preference model instance is much easier than with all compatible preference model instances. Thus, for example, Zurich can deteriorate its performances by 0.9913 or 0.5816 times while still being assigned to \(C_4\), respectively, necessarily or possibly.
In the eAppendix, we analyze the problem of assessing environmental impact of cities in terms of multiple criteria ranking and provide an answer to some representative questions of different type that can be answered within the framework for post factum analysis.
7 Conclusions
In this paper we presented a new approach for sensitivity and robustness analysis of the multiple criteria ranking and sorting recommendations. We have formulated optimization problems for determining the improvement that an alternative needs to make in order to achieve some target result, or the deterioration that it can afford in order to maintain it. We have taken into account five types of targets referring to pairwise preference relations, assignmentbased preference relations, group preference relations, attaining some rank or class assignment. The above targets have been considered in view of their achievement or maintenance for at least one or all preference model instances compatible with Decision Maker’s preference information. We have quantified the required improvement or allowed deterioration in terms of either alternative’s comprehensive values (scores) or its performance vector. For the latter, we referred to changing performances either on all criteria or only on some selected subsets of criteria. Although we elaborated for the valuebased robustness analysis methods, the basic ideas underlying post factum analysis are applicable for a wide spectrum of MCDA approaches.
The usefulness of the results obtained with post factum analysis is twofold. Firstly, they indicate the best strategy for achieving or maintaining the target. As noted by Alexander [1], this kind of recommendation derived from sensitivity analysis can be used in the design, application of allocation formulas, or play a communicative in planning which aims to maximize the efficiency in decision making and minimize resource usage. Secondly, post factum analysis may help in ensuring that the recommended decision is robust. Indeed, by examining the smallest improvement (the greatest deterioration) of the performance values that needs to (can) be made to attain (maintain) the target, the robustness of the recommendation is examined [17]. For example, with small modification required/allowed, the recommended decision can be regarded as being sensitive to the performance values and the DM faces the risk of its change. In case the modification of performances is large, the recommendation can be deemed as robust and the DM can be confident of the validity of current results. From another perspective, the discovered modifications can be used to define a proximity recommendation (ranking or class assignments) that may potentially differ from the original recommendation [41]. This can be achieved, e.g., by studying which alternatives are close to being ranked first or assigned to the best class. Finally, by introducing the concept of improvement reducts, post factum analysis may be used to indicate how critical different performance values are in the ranking or assignment of the alternatives [39]. This can provide direction to the DM for further analysis or stimulate reevaluation of the most critical values more accurately.
We have illustrated the introduced approach using the problem of assessing environmental impact of 30 main European cities. Nevertheless, the scope of decision problems in which answering similar questions may be of interest to the DM is very broad. Indeed, post factum analysis is useful whenever planning, design, or resource allocation are involved in the process. This holds, e.g., in environmental management, manufacturing industry, and fund allocation.

parametric evaluation of research units [24] to indicate the improvements in terms of quality of acquired effects and activities undertook in the evaluation period which are necessary for being assigned to a better class;

environmental management of land zones to provide support in terms of modifying the natural and anthropogenic indicators for acquiring greater resilience against desertification perceived by the experts [38];

information retrieval with respect to changing the relevance and positioning of scientific articles on rankordered lists to improve their perceived physician based ranking [30].
Notes
Acknowledgments
The first and the second authors wish to acknowledge financial support from the Polish National Science Centre (grant SONATA, no. DEC2013/11/D/ST6/03056). The authors thank two anonymous referees for their remarks which helped us to significantly improve the paper.
Supplementary material
References
 1.Alexander, E.: Sensitivity analysis in complex decision models. J. Am. Plan. Assoc. 55(3), 323–333 (1989)CrossRefGoogle Scholar
 2.Ahn, B.S., Park, K.S.: Comparing methods for multiattribute decision making with ordinal weights. Comput. Op. Res. 35(5), 1660–1670 (2008)CrossRefzbMATHGoogle Scholar
 3.Banker, R.D., Charnes, A., Cooper, W.W.: Some models for estimating technical and scale inefficiencies in data envelopment analysis. Manag. Sci. 3, 1078–1092 (1984)CrossRefzbMATHGoogle Scholar
 4.Beynon, M.J., Barton, H.: A PROMETHEE based uncertainty analysis of UK police force performance rank improvement. Int. J. Soc. Syst. Sci. 1(2), 176–193 (2008)CrossRefGoogle Scholar
 5.Beynon, M.J., Wells, P.: The lean improvement of the chemical emissions of motor vehicles based on preference ranking: A PROMETHEE uncertainty analysis. Omega 36(3), 384–394 (2008)CrossRefGoogle Scholar
 6.Charnes, A., Cooper, W.W., Rhodes, E.: Measuring the efficiency of decision making units. Eur. J. Op. Res. 2(6), 429–444 (1978)MathSciNetCrossRefzbMATHGoogle Scholar
 7.Chen, H., Wood, M.D., Linstead, C., Maltby, E.: Uncertainty analysis in a GISbased multicriteria analysis tool for river catchment management. Environ. Model. Softw. 26, 395–405 (2011)CrossRefGoogle Scholar
 8.Corrente, S., Greco, S., Kadziński, M., Słowiński, R.: Robust ordinal regression in preference learning and ranking. Mach. Learn. 93(2–3), 381–422 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
 9.Doumpos, M., Zopounidis, C., Galariotis, E.: Inferring robust decision models in multicriteria classification problems: an experimental analysis. Eur. J. Op. Res. 236(2), 601–611 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
 10.EIU. European Green City Index. Assessing the environmental impact of Europe’s major cities. Economist Intelligence Unit, London (2009)Google Scholar
 11.Gouveia, M., Dias, L.C., Antunes, C.H.: Superefficiency and stability intervals in additive DEA. J. Op. Res. Soc. 64, 86–96 (2013)CrossRefGoogle Scholar
 12.Greco, S., Mousseau, V., Słowiński, R.: Ordinal regression revisited: multiple criteria ranking using a set of additive value functions. Eur. J. Op. Res. 191(2), 415–435 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
 13.Greco, S., Mousseau, V., Słowiński, R.: Multiple criteria sorting with a set of additive value functions. Eur. J. Op. Res. 207(4), 1455–1470 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
 14.Guan, X., Pardalos, P.M., Zuo, X.: Inverse Max \(+\) Sum spanning tree problem by modifying the sumcost vector under weighted \(l_\infty \) Norm. J. Glob. Optim. 61(1), 165–182 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
 15.Hasuike, T.: Riskcontrol approach for bottleneck transportation problem with randomness and fuzziness. J. Glob. Optim. 60(4), 663–678 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
 16.Hazen, G.B.: Partial information, dominance, and potential optimality in multiattribute utility theory. Op. Res. 34(2), 296–310 (1986)MathSciNetCrossRefzbMATHGoogle Scholar
 17.Hyde, K.M., Maier, H.R.: Distancebased and stochastic uncertainty analysis for multicriteria decision analysis in Excel using Visual Basic for Applications. Environ. Model. Softw. 21(12), 1695–1710 (2006)CrossRefGoogle Scholar
 18.Hyde, K.M., Maier, H.R., Colby, C.B.: A distancebased uncertainty analysis approach to multicriteria decision analysis for water resource decision making. J. Environ. Manag. 77(4), 278–290 (2005)CrossRefGoogle Scholar
 19.Kadziński, M., Ciomek, K., Słowiński, R.: Modeling assignmentbased pairwise comparisons within integrated framework for valuedriven multiple criteria sorting. Eur. J. Op. Res. 241(3), 830–841 (2015)MathSciNetCrossRefGoogle Scholar
 20.Kadziński, M., Corrente, S., Greco, S., Słowiński, R.: Preferential reducts and constructs in robust multiple criteria ranking and sorting. OR Spectr. 36(4), 1021–1053 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
 21.Kadziński, M., Greco, S., Słowiński, R.: Extreme ranking analysis in robust ordinal regression. Omega 40(4), 488–501 (2012)CrossRefzbMATHGoogle Scholar
 22.Kadziński, M., Greco, S., Słowiński, R.: RUTA: a framework for assessing and selecting additive value functions on the basis of rank related requirements. Omega 41(4), 735–751 (2013)CrossRefGoogle Scholar
 23.Kadziński, M., Słowiński, R.: DISCARD: a new method of multiple criteria sorting to classes with desired cardinality. J. Glob. Optim. 56(3), 1143–1166 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
 24.Kadziński, M., Słowiński, R.: Parametric evaluation of research units with respect to reference profiles. Decis. Support Syst. 72, 33–43 (2015)CrossRefGoogle Scholar
 25.Kadziński, M., Tervonen, T.: Robust multicriteria ranking with additive value models and holistic pairwise preference statements. Eur. J. Op. Res. 228(1), 169–180 (2013)CrossRefGoogle Scholar
 26.Kadziński, M., Tervonen, T.: Stochastic ordinal regression for multiple criteria sorting problems. Decis. Support Syst. 55(1), 55–66 (2013)CrossRefGoogle Scholar
 27.Kadziński, M., Tervonen, T., Figueira, J.R.: Robust multicriteria sorting with the outranking preference model and characteristic profiles. Omega 55, 124–140 (2015)Google Scholar
 28.Malakooti, B.: Ranking and screening multiple criteria alternatives with partial information and use of ordinal and cardinal strength of preferences. IEEE Trans. Syst. Man Cybern. A Syst. Hum. 30, 355–368 (2000)CrossRefGoogle Scholar
 29.Matsatsinis, N.F., Grigoroudis, E., Samaras, A.P.: Aggregation and disaggregation of preferences for collective decisionmaking. Group Decis. Negot. 14(3), 217–232 (2005)CrossRefGoogle Scholar
 30.O’Sullivan, D., Wilk, Sz, Michalowski, W., Słowiński, R., Thomas, R., Kadziński, M., Farion, K.: Learning the preferences of physicians for the organization of result lists of medical evidence articles. Methods. Inf. Med. 53(5), 344–356 (2014)Google Scholar
 31.Ravalico, J.K., Dandy, G.C., Maier, H.R.: Management option rank equivalence (MORE)—a new method of sensitivity analysis for decisionmaking. Environ. Model. Softw. 25, 171–181 (2010)CrossRefGoogle Scholar
 32.Rocco, C.M., Tarantola, S.: Evaluating ranking robustness in multiindicator uncertain matrices: an application based on simulation and global sensitivity analysis. In: Bruggemann, R., Carlsen, L., Jochen, W. (eds.) Multiindicator Systems and Modelling in Partial Order, pp. 275–292. Springer, Berlin (2014)CrossRefGoogle Scholar
 33.Rocha, C., Dias, L.C.: An algorithm for ordinal sorting based on ELECTRE with categories defined by examples. J. Glob. Optim. 42, 255–277 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
 34.Salo, A., Punkka, A.: Ranking intervals and dominance relations for ratiobased efficiency analysis. Manag. Sci. 57(1), 200–214 (2011)CrossRefzbMATHGoogle Scholar
 35.Seiford, L.M., Zhu, J.: Stability regions for maintaining efficiency in data envelopment analysis. Eur. J. Op. Res. 108, 127–139 (1998)CrossRefzbMATHGoogle Scholar
 36.Spliet, R., Tervonen, T.: Preference inference with general additive value models and holistic pairwise statements. Eur. J. Op. Res. 232(3), 607–612 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
 37.Tervonen, T., Figueira, J.R., Lahdelma, R., Almeida Dias, J., Salminen, P.: A stochastic method for robustness analysis in sorting problems. Eur. J. Op. Res. 192(1), 236–242 (2009)CrossRefzbMATHGoogle Scholar
 38.Tervonen, T., Sepehr, A., Kadziński, M.: A multicriteria inference approach for antidesertification management. J. Environ. Manag. 162, 9–19 (2015)CrossRefGoogle Scholar
 39.Triantaphyllou, E., Sanchez, A.: A sensitivity analysis approach for some deterministic multicriteria decisionmaking methods. Decis. Sci. 28(1), 151–194 (1997)CrossRefGoogle Scholar
 40.White, C.C., Sage, A.P., Dozono, S.: A model of multiattribute decision making and tradeoff weight determination under uncertainty. IEEE Trans. Syst. Man Cybern. 14(2), 223–229 (1984)MathSciNetCrossRefGoogle Scholar
 41.Wolters, W.T.M., Mareschal, B.: Novel types of sensitivity analysis for additive MCDM methods. Eur. J. Op. Res. 81, 281–290 (1995)CrossRefzbMATHGoogle Scholar
 42.Zheng, J., Cailloux, O., Mousseau, V.: Constrained multicriteria sorting method applied to portfolio selection. In: Proceedings of the Second International Conference on Algorithmic Decision Theory, ADT’11, pp 331–343, Berlin, Heidelberg, Springer (2011)Google Scholar
 43.Zhu, J.: Robustness of the efficient DMUs in data envelopment analysis. Eur. J. Op. Res. 90, 451–460 (1996)CrossRefzbMATHGoogle Scholar
 44.Zhu, J.: Superefficiency and DEA sensitivity analysis. Eur. J. Op. Res. 129, 443–455 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
 45.Zopounidis, C., Doumpos, M.: PREFDIS: a multicriteria decision support system for sorting decision problems. Comput. Op. Res. 27(7–8), 779–797 (2000)CrossRefzbMATHGoogle Scholar
Copyright information
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.