Questions guiding the choice of a multicriteria decision aiding method
Abstract
We formulate some questions that may help an analyst to choose a multicriteria decision aiding method well adapted to the decision context. These questions take into account several aspects of the decision process and of the cooperation between the analyst and the decision maker. We present these questions in a hierarchical order, from the most general and crucial, through other pertinent questions concerning the multicriteria aggregation, to the secondary ones. The initial question is what type of results the method is expected to bring. The next questions concern requirements on preference scales, acquisition of preference information, handling of imperfect knowledge, acceptance of compensation among criteria, and existence of interaction among criteria. The last questions are about intelligibility, axiomatic characterization, and weaknesses of the considered methods. To illustrate these questions, we introduce twelve representative and realistic decision contexts.
Keywords
Decision process Multicriteria decision aiding Analyst Decision makerMathematics Subject Classification
90B50 91B02 91B06Introduction

The goal of the decision process, as well as possible ways of achieving it, has been discussed sufficiently well to define a set of potential actions (alternatives).

The need to account for multiple and somehow conflicting points of view has been recognized and, based on this, a family of criteria or attributes has been outlined.

The analyst has acquired a good knowledge of the decision context, and of the possibilities of interactions she may have with the decision maker (or his representative), and other stakeholders, in order to get all necessary information.
Consequently, we suppose that the analyst has arrived at the stage of reflection where she is about to choose the most suitable multicriteria method to be used within the decision process. In our opinion, this method should be seen as a tool for going deeper into the decision problem, for exploring various possibilities, interpreting them, debating and arguing, rather than a tool able to make the decision. We suppose further that the model of preferences used by the method is, at least partially, coconstructed through interaction between the analyst and the decision maker (or his representative). This coconstruction should account for the consequences on which the actions will be judged and for value systems related to the decision context. It should also help the analyst to formulate and express the working hypotheses on which her advice will be based. Moreover, once the method has been chosen, the analyst will collaborate with the decision maker (or his representative) to specify certain characteristics (notably the values attributed to the different parameters) of the preference model that the method requires.
It follows from the above that, in our understanding, the method to be chosen is not expected to discover any good approximation of an objectively best decision, taking into account a preexisting preference system of the decision maker, but rather more modestly, it is expected to provide the decision maker with results that follow from an adopted way of reasoning, consistent with the working hypotheses (Roy 1999, 2010).
This paper presents questions to guide in the choice of a multicriteria decision aiding method. These questions are presented in a hierarchical order. In our opinion, answering these questions in this order may help the analyst choose the most appropriate method for the decision context. In “A number of actual decision contexts”, we present a number of such contexts to illustrate the diversity of decision situations and to demonstrate further how they are conditioning the analyst’s answers to subsequent questions. The first of these questions—which seems to us to be crucial—is analyzed in “A crucial question conditioning the choice of the method by the analyst”. “Five other key questions to choose the right method” presents the first series of key questions which has to be tackled in relation to the first one. In “Secondary questions”, we present another series of questions that may be addressed to the analyst, which, although realistic, seems to be less general than the previous ones. In “Conclusions”, we highlight two difficulties that the analyst can face in her choice of a multicriteria decision aiding method.
It should be emphasized that we do not aim at providing the reader with strict conditions under which such or such method is the most appropriate, but rather at providing guidelines facilitating the choice of a multicriteria decision aiding method well adapted to the specific context of a case study.
A number of actual decision contexts
Here, we present briefly a number of actual decision contexts to which we will refer later on. Most of these concern a real case that one of us has studied. Note that the list of contexts presented below does not pretend to be representative of the use of MCDA methodology in real world decision problems: it only aims at helping the reader to better understand the arguments that are put forward in the second part of the paper.
At the first reading, this section can be skipped. The reader will be sent back to each of these contexts later, while going along the following sections.
Context no. 1: Commuter rail line. In a large urbanized area, the policy makers have decided to improve the public transportation system. To increase access to different zones, especially for residents of a growing suburb, while using the existing infrastructure as much as possible, the decision to build a rail link (tramway) between the suburb and the employment zone has been taken. Providing such a service does not present any major problem in terms of the line’s layout. The difficulties arise, rather, from design issues related to the number and location of stations as well as capacityrelated features. A dozen of variants have already been considered, and all the interested parties have already agreed on five general objectives: (1) minimize investment and operating costs, (2) minimize access time to the stations and line haul times along the rail line, (3) improve the compatibility among urban development, employment, and the transportation system, (4) maximize the wellbeing of the transport users (increase comfort, safety, etc.), (5) avoid environmental disruption as much as possible. More details about this problem context can be found in example 2 of Roy (1996) and in Labbouz et al. (2008).
Context no. 2: Siting of a nuclear powerplant. The public power supply authority of a country planned to build a nuclear powerplant on a seaside. At the time the consultants’ bureau was called to work on this problem, nine potential sites for a technically feasible placement of the powerplant had already been identified. In order to evaluate and compare these sites, six points of view were chosen: (1) the health and security of the population in the surrounding region, (2) the loss of salmonids in streams absorbing the heat from the powerplant, (3) the biological effects on the surrounding region, (4) the socioeconomic impact of the installation, (4) the esthetic impact of the power lines, (5) the investment cost and (6) the operating cost of the powerplant. Further details may be found in Keeney and Nair (1976), Keeney and Robillard (1977), and Roy and Bouyssou (1986).

no new MWIP is build, and the waste of Vaud and Fribourg are distributed over existing plants,

two MWIPs are built, one in Vaud and another in Fribourg,

one MWIP is built, in Vaud only,

one MWIP is built, in Fribourg only,

one MWIP is built, at the frontier of Vaud and Fribourg.
Considering different ways of distributing the waste, and a possible extension of the existing MWIP in Vaud (TRIDEL in Lausanne), the ‘task force’ team constructed 17 potential actions. The construction of these potential actions was followed by construction of a consistent family of criteria. This family included 20 criteria, covering diversified aspects such as: ecology (3), economy (5), organization (4), law (3), and psychology (5). One can notice a strongly noncompensatory character of such heterogeneous criteria. Moreover, the four stakeholders had different views on the relative importance of each criterion. The construction of actions and criteria required a substantial amount of work by the body of the four stakeholders helped by the facilitators and by an engineers’ office employed for this purpose. More details can be found in Bollinger et al. (1997) and Maystre and Bollinger (1999).
Context no. 4: Line extension of the Paris metro. Due to important development of the metropolitan Paris, the metro lines set up in the past needed to be extended. At the end of 1980s, the Régie Autonome des Transports Parisiens (RATP) elaborated 12 new metro extension projects into the suburbs, which added up to 42.6 km, and consisted of the construction of 38 new stations, doubling the number of stations in the suburbs. The decision about a priority ranking of these projects was not within the purview of the RATP: it was in charge of national and regional agencies responsible for planning and programming the infrastructure. RATP was chartered, however, to elaborate all the technical, financial, economic and social considerations, necessary to pointing up all the priorities which may validly be set up, independently of any value system pertaining to RATP. To clarify the decisions concerning the timedependent construction of the various segments of such line extensions, the department of Operational Research of the RATP was in charge of their evaluation and, to a possible extent, of their classification, taking into account the following six criteria of which each reflects a specific point of view of one of the stakeholders: (1) the number of residents and jobs served by the project, (2) the number of daily passengers entering the stations on the line concerned, (3) the capital cost of the project per 1 km of line, (4) the internal rate of return, (5) the advisability of the extension with respect to the general organization of the transit systems in the considered sector, and (6) the structuring effects on urban development. In order to fix a performance on a particular criterion for each extension, it was necessary to rely on many estimations of the population affected, cost elements, value of time and discount rate. One had also to adopt prospective views on such issues as urban development, and the behavior of public transport users. It is thus evident that the validity of any such assessment was subject to a margin of error. A report of this study can be found in Roy and Hugonnard (1982).
 1.
The evaluation of responses on some criteria was partially uncertain or even arbitrary.
 2.
The four directors had different views on the relative importance of each criterion.

no restriction concerning acceptance,

some hesitation concerning acceptance (a doubtful “yes”),

hesitation concerning refusal (a doubtful “no”),

unhesitating refusal.
To foster a debate within the committee, each of its members gets information about the nine evaluations of each file, and about the category to which each of them was assigned. Any member of the committee may query any item of the information leading to the assignment (assessment of certain risks, allocation of resources other than those requested, etc.). When some items of information are thus modified, new evaluations are immediately recalculated and the resulting reassignment is made known to committee members.
Context no. 7: Management of highway assets. A road network is managed by a central agency that needs to coordinate and control the activities of many local districts spread over a wide geographic area. Considering that the funds required to satisfy the needs of maintenance usually exceed the available resources, the decision about annual budget allocation for routine maintenance is crucial to achieve the best possible efficiency of the road network. In this decision context, there are many decision levels and a hierarchy of stakeholders with different perspectives and specific objectives. The large amount of data makes the decision process very complex. Thus, highway agencies need tools for the coordinated management of their assets that allow interactions between the stakeholders and the analyst in course of the allocation of available funds according to their preferences.

type of distress recorded during the periodical survey,

geometric characteristics of the road section,

road functional class,

intensity of traffic,

accident rate.
The road sections were sorted by road experts into four categories of urgency. The task was to develop an intelligible decision model that would reproduce the expert decisions on the road sections and would facilitate prospective decisions about the degree of urgency of the pavement on new road sections.
 (a)
setting up a priority order in which water users should be connected to a new WSS, taking into account economic, agricultural, and sociological consequences of the investment,
 (b)
choosing the best variant of technical construction of the regional WSS evaluated from technical and economic viewpoints and from the viewpoint of relationship with the priority order of users coming from problem (a).
The study took into account both purely technical and socioeconomic aspects of WSS programming in the form of distinct criteria. A single WSS typically concerns a set of 20–40 users. In problem (a), they were evaluated with respect to the following criteria: water deficiency, farm production potential, function and activity of the user, structure of settlements, water demand, share of water supply installations in all investments concerning the user, possibility of connecting the user to another existing WSS. The data used to calculate user performances on these heterogeneous criteria were affected by some imprecision and indetermination. As to problem (b), there can be hundreds of technical variants, because it involves different types and locations of water sources, types and capacities of system components, and feasible structures of the distribution network. Each feasible variant was characterized by four criteria: (1) investment and (2) operating costs, (3) reliability and (4) a distance between the socioeconomic priority order of users, and the precedence order of users connected to the WSS constructed according to a given variant. The last criterion played a coordinating role between both problems of the programming task. The recommended variant was the one which ensured the best compromise between the four criteria.
A similar problem has been considered by Słowiński (1986), however, with respect to development planning of a jointly operated urban water supply and wastewater treatment system in a 20year planning horizon. This problem has been formulated as a multiobjective LP problem, where the variables were daily water flows on main pipeline connections between sources and users, and daily inflows of wastewater to discharging treatment plants, both of them in consecutive time periods. The objectives were: expansion and operating cost of (separately) water intakes, recycling treatment plants and discharging treatment plants, reliability of water supply, and environmental quality. Because there was no precise data about cost and reliability coefficients, water pollution indices, discount factors and the user’s demands, the experts accepted to specify for each of them an interval of the most possible values that were included in an interval of least possible although realistic values. This corresponded to the definition of trapezoidal fuzzy numbers and lead to a fuzzy multiobjective LP formulation.
Context no. 9: Clinical decision support in emergency room. Abdominal pain in childhood is a highly prevalent symptom caused by organic diseases, psychosocial disturbances and emotional disorders. In many cases, the exact cause is never known. Medical staff must focus on identifying a minority of cases in need of urgent treatment. The child who complains about abdominal pain is initially examined in the emergency room by a medical intern. The possible outcomes of this evaluation are: ‘discharge’, ‘surgical consult’, and ‘inhospital observation’. A limited number of clinical signs, symptoms and tests, available at the early stage, make such a triage very difficult. To increase the accuracy of the triage, the Children’s Hospital of Eastern Ontario in Ottawa (CHEO) called for a study aiming at developing a clinical decision support system that would assist medical interns in the emergency room. In order to learn diagnostic patterns from past diagnoses made by surgeons, CHEO provided a data set including records of 647 patients with abdominal pain seen during a 3year period in the emergency room of CHEO. The patients were described by 12 early stage symptoms, called attributes. The data was collected as part of a retrospective chart study, and thus were not complete (10–20 % of missing values). Most of the attributes were nominal (gender, type of pain), some were numerical (age, duration of pain, temperature, white blood cell count), some were binary (vomiting, recent visit in emergency room, muscle guarding, rebound tenderness), and some indicated a location of a condition on patient’s abdomen (location of pain, location of tenderness). According to medical practice, patient records stored in the data set were sorted into three categories: ‘discharge’, ‘surgical consult’, and ‘inhospital observation’. The focus of the study—which is reported in Michalowski et al. (2003) and Wilk et al. (2005)—was on inducing from the data set an intelligible decision model consistent with the past decisions of specialists, involving the most relevant attributes from among the 12 available attributes. This decision model has been designed to support emergency room staff and has been embedded into a decision support system on a mobile platform, called MET (Mobile Emergency Triage).
Context no. 10: Credit granting. Every day, bank B receives several credit applications from various firms wishing, for example, to put a hotel or a clinic, or to buy machines for some public works. These files are submitted to a credit evaluator whose mission is to consider all of them and decide about the sort of each particular application. This credit evaluator would like to get a help from operational research department of B, by asking it to develop a software tool that would make a preliminary sorting of the incoming files. He wants that the sorting is based on performances of each file on a family of criteria that takes into account a list of viewpoints he provides. These viewpoints cover three main concerns: profitability, the risk of mortgage nonpayment, and commercial impact. The family of criteria has to take into account how the application looks like (in particular, what is motivation, requested amount and duration), various characteristics of personality and situation of the applicant (in particular, legal status of the applicant’s enterprise, age of managers, balance sheets from recent years), as well as the history of possible past applications and commercial relations of bank B with the applicant. As soon as a new application for credit arrives, an assistant of the credit evaluator analyzes it in view of defining its performances on the family of criteria. Taking into account an imprecision, or even a lack of some data, which makes the definition of performances highly subjective, the assistant of the credit evaluator can be led to fix not one but two performances per criterion, which correspond to optimistic and pessimistic value, respectively. The credit evaluator requires that on this base each file is assigned to one of the following categories:
Category C_{1}: Files apparently bad that should be rejected after a quick verification.
Category C_{2}: Files rather bad that, for some commercial reasons, could be nevertheless accepted and that have to be transferred to another department.
Category C_{3}: Files rather good that nevertheless need to be completed with some additional information before being carefully examined.
Category C_{4}: Files apparently good that should be accepted after a quick verification.
If necessary, the credit evaluator would like a fifth category to be appended:
Category C_{5}: Atypical files that could hardly be assigned to one of the above four categories.
The multicriteria decision aiding method used to assist the credit evaluator in the above context has been implemented as a computer program which served many years and remained confidential. Even if it was more complex, this method inspired the ELECTRE TRI method (Yu 1992; Roy and Bouyssou 1993).
Context no. 11: Monitoring of risk zones. One is considering here a territory T where some old iron mines exploited using the ‘room and pillar’ caving method present a risk of collapse or ground subsidence causing major damages to surface buildings and infrastructures. T has been partitioned into zones which exhibit homogeneous characteristics of the underground and of the surface infrastructure (segments of highways and other roads, schools, commercial centers, public buildings, apartment buildings, entertainment parks, etc.). For more details, see Merad et al. (2004). A family of criteria has been defined to assess the gravity of risk that a zone may present based on its characteristics. The risk prevention authority responsible for the monitoring of territory T is wishing that each zone is assigned to one of the following categories:
Category C_{1}: Very low risk zones requiring reference leveling (topographic surveys) only.
Category C_{2}: Low risk zones requiring reference and annual leveling.
Category C_{3}: Zones of risk sufficiently high for an in depth investigation including geological boring if necessary.
Category C_{4}: High risk zones requiring longterm continuous monitoring based on recording of underground microseismic activity.
Criteria used for evaluation of the homogeneous zones are grouped under two headings: ‘susceptibility of the mine to collapse’ and ‘surface sensitivity’. The first refers to the ‘probability of rupture’ and the second to both the ‘intensity of the rupture’ and the ‘value and vulnerability of assets’. In the first group, there are criteria such as: mean stress applied on pillars, existence of fault, superimposition of pillars, size and regularity of pillars, and sensitivity of rock to flooding. The second group has criteria such as: depth of the top mined layer, maximum expected subsistence, expected surface deformation, zone extent, and vulnerability of buildings, roads, railways, bridges and various networks (electricity, water, gas). For some criteria the risk grows with the performance, and for the others, it decreases.
This is a complex decisionmaking problem in which the available information is uncertain (missing information, such as geological data) and imprecise (mining works maps), and in which knowledge is incomplete (e.g., soilstructure interaction). The risk prevention authority would like to be supported by a method permitting identification and sorting of homogeneous zones into the four predefined risk categories.
Context no. 12: Engineering design of a chemical reactor. In the engineering design of a chemical reactor for the production of pxylene, it is necessary to determine the most appropriate parameters for the efficient production of this compound. It is used as an intermediate in production of various plastics such as: polyesters and polyamides. pXylene is to be produced by isomerization of oxylene over Hmodernite catalyst in a flow reactor. A kinetic model of this reaction, defined by a set of differential equations, is a basis for formulation of a multiobjective nonlinear programming problem. The decision (design) variables are: temperature and pressure of the process, reactor volume, feed flow rate and catalyst weight. The feasible values of the decision variables are constrained by technological requirements and by equations following from the kinetic model. The values of the decision variables should give the best compromise between four conflicting objectives. The first objective—reactor volume, to be minimized—represents the designer’s attitude to reduce the size of the chemical installation. It influences both investment and operating costs of the reactor. The second objective—catalyst weight, to be minimized—expresses the designer’s aspiration to reduce the weight of the Hmodernite used in the reactor. The third objective—mass production of pxylene—is to be maximized. The last objective—concentration ratio between pxylene and oxylene, to be maximized—is correlated with the quality of the final product and with the level of transformation of oxylene, i.e. with the efficiency of the production process (Jaszkiewicz et al. 1995). One can notice that the objectives involve heterogeneous scales. Moreover, as the kinetic model of the reaction is an approximation of the real process, the performances of feasible solutions on particular objectives should also be considered as approximate during the search of the best compromise solution.
A crucial question conditioning the choice of the method by the analyst
Taking into account the context of the decision process, what type(s) of results the method is expected to bring, so as to allow elaboration of relevant answers to questions asked by the decision maker?
The type of results produced is a feature which distinguishes various methods of multicriteria decision aiding. Depending on the decision context, this is not the same type of results that may bring useful information able to guide the decision aiding process in the right way, and to work out some conclusions, or even a recommendation. Moreover, the type of results is conditioning the way in which the analysis is inserted in the decision process. The analyst should keep all this in mind when answering the above question.
A review of multicriteria decision aiding methods (well defined and sufficiently operational to be taken into account by the analyst) led us to distinguish five main types of results that the analyst may want to consider in relation to methods clearly associated with them. Other types of results corresponding to different decision problems could also be considered (Bana e Costa 1996; Tsoukiàs 2007) but, to our knowledge, there is no wellestablished method that could produce them.
 (a)
Type 1: A numerical value (utility, score) is assigned to each potential action.
It is possible that this type of result is imposed by the decision maker. This could be the case of contexts: Commuter rail line (1), Siting of a nuclear powerplant (2), and possibly Responses to tenders (6). Many methods can produce this type of results: MAVT (Keeney and Raiffa 1976), MAUT (Dyer 2005), UTA (JacquetLagrèze and Siskos 1982), MACBETH (Bana e Costa et al. 2005), AHP, (Saaty 2005), SMART (Winterfeld and Edwards 1986), TOPSIS (Hwang and Yoon 1981), Choquet Integral (Grabisch and Labreuche 2005), representative value function of UTA^{GMS} (Kadziński et al. 2012), and others.
The application of these methods requires (either in course of applying the method, or even before) that the scale of each evaluation criterion is an interval scale. Sometimes, these scales have to be identical. Constructing such scales calls for interacting with the decision maker (or his representative) in a way which is specific for each of these methods. The analyst should keep this requirement in mind when drawing up a short list of methods.
 (b)
Type 2: The set of actions is ranked (without associating a numerical value to each of them) as a complete or partial weak order.
This type of result can be considered only if the set A of potential actions is known a priori. It is not convenient when the potential actions are examined as they arrive (week after week, month after month, etc.); this is the case of such contexts as Clinical decision support in emergency room (9), or Credit granting (10). It seems well adapted to such contexts as Line extension of the Paris metro (4), or Programming of water supply systems for rural areas (8).
The methods relevant here are: ELECTRE III, IV (Figueira et al. 2005, 2013), PROMETHEE I and II (Brans and Mareschal 2005), all Robust Ordinal Regression methods (Greco et al. 2010b) producing necessary and possible rankings, like UTA^{GMS} (Greco et al. 2008c), GRIP (Figueira et al. 2009b), Extreme Ranking Analysis (Greco et al. 2012), RUTA (Kadziński et al. 2013), ELECTRE^{GKMS} and PROMETHEE^{GKMS} (Greco et al. 2011), and moreover, the Dominancebased Rough Set Approach to ranking (Greco et al. 2001; Słowiński et al. 2009; Szeląg et al. 2013), and Machine Learning approach (Dembczyński et al. 2010).
 (c)
Type 3: A subset of actions, as small as possible, is selected in view of a final choice of one or, at first, few actions.
As in the case of type 2, this type of result is not convenient when the set A of potential actions is not known a priori. It is also not convenient in the contexts like Line extension of the Paris metro (4), Management of highway assets (7), Programming of water supply systems for rural areas (8), Clinical decision support in emergency room (9), or Monitoring of risk zones (11). In these contexts, many potential actions, and not only one, are intended for joint execution. The type of result considered here is convenient when the potential actions are modeled as alternatives, i.e. such that a joint execution of any two of them is excluded. This is the case of such contexts as Commuter rail line (1), Siting of a nuclear powerplant (2), Location of a municipal waste incineration plant (3), Supplier selection (5), or Engineering design of a chemical reactor (12).
This type of results is produced directly by such methods as: ELECTRE I and IS (Figueira et al. 2005, 2013), PROMETHEE V (Brans and Mareschal 2005), Rubis (Bisdorff et al. 2008). The methods of multiobjective optimization (Branke et al. 2008) also lead to this type of results, however, they are applied when actions of the set A are vectors of variables subject to some mathematical programming constraints. These methods will also be considered later for type 5 of results.
Note that this type of result is also relevant to the case of a multicriteria choice of the best portfolio of objects with cardinality and cost constraints. In this case, the set A is composed of alternatives that represent those combinations of objects which respect cardinality and cost constraints. The methods designed for this case are discussed, e.g., by Liesiö et al. (2008), Metaxiotis and Liagkouras (2012) and Greco et al. (2013a).
 (d)
Type 4: Each action is assigned to one or several categories, given that the set of categories has been defined a priori.
It is possible that this type of results is imposed by the decision maker. This could be the case of such contexts as: Responses to tenders (6), Management of highway assets (7), Clinical decision support in emergency room (9), Credit granting (10), or Monitoring of risk zones (11). This type of results is particularly well adapted to the contexts where the set A is not defined a priori, like Clinical decision support in emergency room (9) and Credit granting (10). It can also be convenient to presort when a large number of potential actions have been listed at the starting point of the decision process (Jaszkiewicz and Ferhat 1999). This could happen in such contexts as: Commuter rail line (1), or Siting of a nuclear powerplant (2). Such a presorting is also used in some interactive multiobjective optimization procedures, where it concerns a number of nondominated solutions proposed for evaluation to the decision maker in each dialogue phase (Greco et al. 2008a).
 (e)
Type 5: A subset of potential actions enjoying some remarkable properties is provided to serve as a base in the following stage of the decision aiding process.
This type of result may be required when the set of alternatives A contains a very large number of actions (more than one hundred). This is, for example, the case when the actions are defined by vectors of variables subject to some mathematical programming constraints. Then, it may be interesting to get a restricted subset of A, called A′, composed of actions enjoying some remarkable properties, and then replace A by A′ at later stages of the decision aiding process. In multiobjective optimization, the set A′ is a set of nondominated actions (also called efficient or Paretooptimal solutions) or an approximation of this set. Mathematical foundations of the completeness and constructiveness of parametric characterization of the set of nondominated actions has been given by Wierzbicki (1986). From a practical point of view, evolutionary algorithms appeared to be particularly effective in finding a good approximation of the set of nondominated actions in multiobjective optimization (Deb 2008).
 (1)
procedures based on exploration of A′,
 (2)
procedures based on progressive contraction of A′.
Typical examples of category (1) are procedures using reference points defined by aspiration levels in the criteria space. The reference points are projected onto the set A′ in order to find the ‘closest’ nondominated actions to be proposed to the decision maker. The projection is done using Chebyshevlike achievement (scalarizing) functions. Changing the reference point, one can browse the whole set A′. The reference point approaches have been described by Wierzbicki (1999). Some implementations of the projection principle give to the user an impression of driving a vehicle over the nondominated set—this is, for example, the case of Pareto race (Korhonen and Wallenius 1988) or NIMBUS method (Miettinen 1999).
Interactive procedures of category (2) present to the decision maker in each dialogue phase a sample of nondominated actions picked from a gradually reduced region of the set A′. A typical example is the method of Choo and Atkins (1980). The interesting subregion of the set A′ is often delimited by a polyhedral cone with the origin in a reference point, oriented toward the nondominated set. The spread of this cone is controlled by interval values of weights assigned to criteria in the weighted Chebyshev achievement function, like in the cone contraction method (Steuer 1986). In the ‘Light Beam Search’ method (Jaszkiewicz and Słowiński 1999), the spread of the cone is defined by an outranking relation between a nondominated action, called middle point, resulting from projection of a reference point onto the set A′, and its neighborhood actions, such that the subregion of A′ is composed of nondominated actions which are not worse than the middle point, i.e. outrank the middle point. In the robust cone contraction method (Kadziński and Słowiński 2012), the spread of the cone with the origin in a reference point is defined by directions of the isoquants of all weighted Chebyshev achievement functions compatible with pairwise comparisons of some nondominated actions from the current set A′, specified by the decision maker. In successive iterations, each new pairwise comparison contracts the cone which is zooming on a subset of nondominated actions of greatest interest for the decision maker. It is also worth mentioning the NEMO method which combines an evolutionary multiobjective optimization with robust ordinal regression in an interactive procedure (Branke et al. 2010); this combination allows speeding up convergence to the most preferred subregion of the set A′.
Note that, independently of category (1) or (2), the final result of interactive multiobjective optimization methods is of type 3.
The context fully adequate to type 5 of results is Engineering design of a chemical reactor (12). It may also be useful in Programming of water supply systems for rural areas (8), in part (b) concerning the choice of the best variant of technical construction of the regional WSS, and, particularly, in development planning of a jointly operated urban water supply and wastewater treatment system, formulated as a multiobjective fuzzy LP problem.
Five other key questions to choose the right method
 Question 1a

Do the original performance scales have all required properties for a rightful application of the considered method?
Some methods (in particular those mentioned in “A crucial question conditioning the choice of the method by the analyst” in point (a)) cannot handle directly the evaluation criteria whose performances are located on verbal scales, or even numerical but purely ordinal scales. When the answer to question 1a is “no”, the analyst has to check if it is possible to code or transform in a meaningful way the original scales, such that the properties of scales required by the considered method are satisfied. The analyst can carry out this check by looking for an additional information, especially in the course of interaction with the decision maker (or his representative). In order to decide about keeping or rejecting the considered method, the analyst will have to examine to what extent the transformations are arbitrary, and how much the numerical coding of performances masks their meaning. If among the shortlisted methods there are some methods that do not require this type of transformation, she will have to assess the tradeoff between the advantage of keeping the original scales (for interpretation of results and organization of a debate about them) and the inconvenience that may have these alternative methods.
 Question 1b

Is it simple or hard (even impossible) to get preference information that the method requires?
In order to make the results provided by a shortlisted method pertinent for decision aiding, it is necessary to adjust some of its characteristics so as to take into account in the best possible way the preferences of the decision maker; these are usually some wishes underlying the value system of the decision maker. The analyst has thus to acquire what is called preference information. This information takes different forms for various methods: ordering of criteria, ordering of some actions, acceptable tradeoffs, pairwise comparisons of some actions, assignment of some actions to categories, comparisons of some actions with respect to intensity of preference, assessment of lotteries, specification of the relative importance of criteria, presence of veto, etc. To acquire this information, the analyst has to interact with the decision maker (or with his representative), in view of coconstructing the model of preferences that the considered method exploits to work out expected results. A key issue is to organize this interaction such that the analyst is able to elaborate meaningful results. This implies that the interaction protocol or the software tool involved should be compatible with the way in which the analyst has been inserted in the decision process, with the way of reasoning of the inquired people, and with their meaning of useful results. This protocol or software tool has to ensure, moreover, an intelligibility and a traceability of the impact of the preference information on the results. If such an interaction appears to be impossible, the considered method has to be rejected. In all cases the analyst has to assess the part of arbitrariness that the acquired information may contain. She has to remember that it will be necessary to analyze its impact on the results provided by the method (sensitivity analysis, robustness concern).
In the case study Line extension of the Paris metro (4), the impossibility of acquiring preference information about relative importance of criteria led to elaboration of a new method: ELECTRE IV (Figueira et al. 2005). In cases considered within the contexts of Management of highway assets (7), and Clinical decision support in emergency room (9), direct elicitation of preference information about values of parameters of a preference model (in particular, the relative importance weights of criteria) also appeared impossible, which put forward the DRSA method based on rough sets. DRSA accepts sorting examples as input preference information (Greco et al. 2005). Remark that sorting examples, i.e. assignments of some wellknown actions to decision categories, does not need as much cognitive effort from the decision maker as direct specification of preference model parameters required by many other methods.
 Question 1c

Should the part of imprecision, uncertainty or indetermination in the definition of performances be taken into account, and if so, in what way?
It is rare that the performances of the considered actions can be evaluated on each criterion without any ambiguity. The way in which a criterion is modeling preferences related to a specific point of view, or the role that an attribute is supposed to play, can be illdetermined or contain a part of arbitrariness. Some data used to construct criteria can be imprecise or defined in an ambiguous way. When it is important to take into account such an imperfect knowledge, the analyst should examine carefully the possibility of handling this knowledge by the shortlisted methods.
We note that, when performance data are ambiguous, this can cause inconsistency in the indirect preference information given by the decision maker in the form of decision examples (sorting examples or pairwise comparisons). In case of inconsistency, the rough set concept is useful for discerning certain from possible knowledge in reasoning about ordinal data (Słowiński et al. 2012).

Either by sensitivity or robustness analysis of results provided by the method. This approach becomes quickly too onerous when the sources of imperfect knowledge affect more than one or two criteria, because this sensitivity or robustness analysis has often to be combined with a similar analysis taking into account imperfect preference information (partial, inconsistent, vague, etc.; see question 1b).

Or, in an indirect way, by modeling the imperfect knowledge using probability or possibility distribution of actions’ performances on considered criteria. In probabilistic case, the action’s performance on a criterion is set to an expected value of probabilistic distribution. In possibilistic case, it is set to a “mean value” of fuzzy number representing possibilistic distribution (Dubois and Prade 1987). This approach admits, however, that it is possible to model the indetermination in probabilistic or possibilistic way on the base of a relatively objective assessment. A frequent use of the Gaussian distribution in the probabilistic approach, and of the triangular fuzzy numbers in the possibilistic approach, is often missing a sound foundation, given that the performances can vary in a rather narrow interval and that extending it for very low probabilities or possibilities is rather arbitrary. To validate a probabilistic approach, the analyst can sometimes be tempted to ask the decision maker (or his representative) to compare some lotteries in view of revealing his perception of preference or indifference. Proceeding in this way, the analyst is led to translate the indetermination of performances into the terms of preference information (see point 1b), which involves an attitude toward risk. These questions can, however, confuse the decision maker.
The methods that take into account pseudocriteria are able to deal with imperfect knowledge about performances through the use of discrimination thresholds (Roy and Vincke 1987): indifference and preference thresholds. In order to assign values to these thresholds, the analyst has to investigate what sources of imprecision, uncertainty and indetermination affect the performance of an action on the considered criterion. These sources can originate from imperfect representation of the specific viewpoint by this criterion as well as from imperfect knowledge of data to be used for definition of the performance. Being conscious of these sources, the analyst should be able to assess the smallest difference between two performances which, when growing, becomes significant for preference of the action with the better evaluation over the action with the worse evaluation (this is the preference threshold), as well as the greatest difference between two performances which, when decreasing, becomes insignificant and leads to indifference of the two actions (this is the indifference threshold). These two values are not necessarily equal. The analyst must be convinced that the values assigned to these thresholds are appropriate for a right handling of the imperfect knowledge, or using other words, for giving to the comparison of two performances the meaning it deserves (for more details, see Roy and Figueira 2013).
Among the methods shortlisted by the analyst, there can be methods which have been developed for the handling of a particular form of imperfect knowledge. Taking into account that reasoning based on imperfect knowledge can lead to uncertain conclusions, three kinds of uncertainty can be distinguished (Zadeh 1999): (1) uncertainty following from a random change of some variables, called veristic, which can be modeled by probability, (2) uncertainty following from subjective judgments, called possibilistic, which can be modeled by fuzzy sets, and (3) uncertainty caused by granularity of information, called inconsistency, which can be modeled by rough sets. The theories standing behind these concepts of uncertainty are: (1) probability theory, (2) possibility theory (Dubois and Prade 1988) and fuzzy set theory (Słowiński 1998), and (3) rough set theory (Pawlak 1991; Greco et al. 2001). Sometimes the uncertainty is more complex and needs hybridization of the above theories, e.g., probabilisticfuzzy (van den Berg et al. 2004), fuzzyrough (Greco et al. 2008b), or probabilisticrough (Kotłowski et al. 2008). Some comparative studies have been carried out between stochastic and fuzzy approaches on the ground of multiobjective optimization (Słowiński and Teghem 1990).
If some methods accepting pseudocriteria have been shortlisted together with some methods specialized in handling a particular type of imperfect knowledge, the analyst must check which one of them will fit the context under study.
 Question 1d

Is the compensation of bad performances on some criteria by good ones on other criteria acceptable?
In the context of multicriteria decision aiding methods, compensation means that the method offers possibilities of the following type. Let a be an action strictly preferred to another action b, both having the same performances on all but one criterion i on which b is significantly worse. One says that the method offers a possibility of compensation if improving one or more performances of b on other criteria than i it is possible to define an action c indifferent to a. These improvements compensate the bad performance of b compared to a on the ith criterion. Such possibilities of compensation are offered in many ways by additive methods. In a lexicographic method, they exist only if the ith criterion is not the most important. In ELECTRE type methods, they exist under extremely restrictive conditions.

in case of sorting:
“if g _{ i1}(a) ≥ r _{ i1} and g _{ i2}(a) ≥ r _{ i2} and… g _{ ik }(a) ≥ r _{ ik }, then a is assigned to category t or better”,
“if g _{ j1}(a) ≤ r _{ j1} and g _{ j2}(a) ≤ r _{ j2} and… g _{ jh }(a) ≤ r _{ jh }, then a is assigned to category t or worse”,
where r _{ i1}, r _{ i2},…, r _{ ik } and r _{ j1}, r _{ j2},…, r _{ jh } are some threshold performances on criteria i1,…, ik, j1,…, jh ∈ I, found during induction of rules from preference information given in the form of sorting examples and structured using the rough set concept,

in case of choice and ranking:
“if g _{ i1}(a) ≥ r _{ i1} and g _{ i1}(b) ≤ s _{ i2} and… g _{ ik }(a) ≥ r _{ ik } and g _{ ik }(b) ≤ s _{ ik }, then a is outranking b”,
“if g _{ j1}(a) ≤ r _{ j1} and g _{ j1}(b) ≥ s _{ j2} and… g _{ jh }(a) ≤ r _{ jh } and g _{ jh }(b) ≥ s _{ jh }, then a is not outranking b”,
where r _{ i1}, r _{ i2}, …,r _{ ik }, r _{ j1}, r _{ j2},…, r _{ jh } and s _{ i1}, s _{ i2}, …, s _{ ik }, s _{ j1}, s _{ j2},…, s _{ jh } are some threshold performances on criteria i1,…, ik, j1, …, jh ∈ I, found during induction of rules from preference information given in the form of pairwise comparisons of some actions and structured using the rough set concept.
We note that each rule is a scenario of a causal relationship between performances on a subset of criteria and a comprehensive judgment. The rules are noncompensatory aggregators that do not convert ordinal scales of criteria into a richer (interval or ratio) scale.
There are also other methods using noncompensatory aggregation, such as the lexicographic method and the method using the Sugeno integral. A comparison of these methods at an axiomatic level has been done by Greco et al. (2003, 2004), and by Słowiński et al. (2002).
 Question 1e

Is it necessary to take into account some forms of interaction among criteria?

a special form of MAUT, involving either a decomposition and graphical representation of additive value functions under ceteris paribus assumption, like GAInetworks (Gonzales and Perny 2005) and UCPnetworks (Boutilier et al. 2001), or on one hand, a multilinear value function (Keeney and Raiffa 1976), and on the other hand, Choquet or Sugeno integral (Grabisch and Labreuche 2005),

the robust ordinal regression methods with an additive utility function augmented by components accounting for positive and negative synergy of pairs of criteria (Greco et al. 2013b),

ELECTRE method designed to handle interactions (Figueira et al. 2009a), and

the methods using a set of “if…, then…” decision rules as a preference model, since this model based on a very simple syntax of rules is able to handle the most complex interactions (Greco et al. 2002b, 2004; Słowiński et al. 2002).
None of the cases considered in “A number of actual decision contexts” required a priori recognition of some form of interaction among criteria. However, in the case studies referring to contexts: Management of highway assets (7) and Clinical decision support in emergency room (9), the rough set approach has been used, which is able to handle interactions through decision rules, if such interactions would appear in the preference information.
Secondary questions
 Question 2a

Is the method able to satisfy properly the needs of comprehension from the part of stakeholders involved in the decision process?
The needs of comprehension can come not only from the decision maker (or his representative) but also from other stakeholders. If they are not properly satisfied, they can compromise a good insertion of the analyst in the decision process. Thus, she must try to assess up to what level of detail she should explain the way of functioning of the tool, which is the method. Depending on the level at which the requests are placed, she must also assess if those making the requests are ready to devote enough time for listening to the explanation. In many cases, what matters is the possibility of explaining the link that exists between information and data provided from one side, and the results obtained by the method from the other side, without entering into details of the method. As an example, let us mention the ‘even swaps’ method which typically provides nontechnical explanation of this link for an additive preference model (Hammond et al. 1998).
 Question 2b

Is an axiomatic characterization of the method available, and if so, is it acceptable in the considered decision context?
 1.
Suppose first that the analyst accepts the hypothesis that the decision maker has in his mind a relatively welldefined system of preferences, and that the analyst wants to know if every axiom characterizing the shortlisted method is satisfied by this preexisting system of preferences. A classic way to answer this question is to involve the decision maker (in practice, the person being inquired is rarely the decision maker; usually, this is somebody who represents the decision maker—sometimes a member of the ‘task force’ team) in a series of choice situations, and to ask him for precise answers. The decision maker is not necessarily familiar with these situations. He may find them artificial and thus unrealistic. In consequence, the answers given by the decision maker should be interpreted cautiously. Nevertheless, if in a given situation an answer violated clearly the considered axiom, the analyst would be allowed to conclude that the process which determines the decision maker’s preferences with respect to this choice situation violates the axiom in question. If, however, such a violation would not happen for any presented choice situation, the analyst could not be certain if in other choice situations the axiom would not be violated. The analyst should also take into account the fact that the process which determines the answer of the decision maker in choice situation n can be influenced by the way in which he was led to think and answer in n − 1 previous situations. Interrogation protocols which do not escape the trap of this influence obscure the conditions in which the analyst can check if a set of axioms conforms with a system of preferences that preexists in the decision maker’s mind (Roy 2010).
 2.
Let us suppose now that the analyst is not interested in knowing if the decision maker has in his mind a system of preferences concordant with the considered set of axioms, but, more modestly, if he accepts each axiom as a working hypothesis. Such a question can be posed directly to the decision maker only if the axioms are sufficiently simple and well understandable by him. For any other axiom, the analyst will have to present as realistic situations as possible and check the decision maker’s reaction to these situations. They should be conceived in a way permitting to show the decision maker that accepting the axiom implies one type of answers, and rejecting it implies another type. Also here, the way of conceiving the situations and the underlying questions are not neutral. In consequence, the responses being obtained have to be seen as constructs rather than a reflection of an objective reality. It follows that the analyst is influencing more or less consciously the acceptance or rejection of an axiom as a working hypothesis.
 3.
Accepting one by one the axioms from a certain set does not ensure that the set would be accepted in total by the decision maker. Taking into account all axioms of a certain set jointly may show some emerging phenomena that the decision maker cannot see when analyzing sequentially these axioms. It would be thus wrong to suggest that accepting each axiom from a set would imply the validity of all results following from the totality of the set of axioms. The wrong conclusion follows from the confusion between two levels of logics: the one of parts, and the one of the whole. This probably explains why the answers obtained by Maurice Allais to the choice situations presented to the adherents of the utility theory (and, in particular, to many of its founders) appeared to be discordant with the results following from the set of axioms on which this theory is based (Allais 1953; Allais and Hagen 1979).
The above considerations should not lead to believe that the axiomatic work is not useful. On the contrary, it can provide a better understanding and insight to the analyst on how the method actually behaves and how it compares to other methods at the axiomatic level. Such an analysis permitted, for example, to compare the capacity of preference representation of a general utility function and three of its special cases: associative operator, Sugeno integral and ordered weighted maximum on one hand, and a set of rough set decision rules on the other hand (Greco et al. 2004). The formal proof that the decisionrule aggregation (preference) model is the most general among the known aggregation functions is a useful conclusion of this study.
 Question 2c

Can the weak points of the method affect the final choice?
All methods of decision aiding have some weak points. The analyst should know them if she is familiar with the method. She has to examine the possible impact of these weak points in the considered decision context.
An example of the weak point of many methods that involve pairwise comparisons of actions to build a preference model is known under the name of “rank reversal”. This weakness can compromise a method that one would like to use in a decision context where the set of actions may be modified incrementally; this may be the case of ranking an unstable set of actions. Another weak point could be the calculation time if it would be incompatible with the conditions of using the method in the considered decision context; for example, too long calculations between dialogue phases in a multiobjective optimization method.
The weak points of some popular methods have been clearly shown and discussed in various publications, e.g.: for ELECTRE methods in Figueira et al. (2013), for AHP in Bana e Costa and Vansnick (2008) and in Bouyssou et al. (2000, point 6.3.2), for TOPSIS in Martel and Roy (2006), for MAUT in McCord and de Neufville (1983), for methods based on Choquet integral in Roy (2009) and in Greco et al. (2013b), and for methods based on Sugeno integral in Słowiński et al. (2002).
In all the considered decision contexts, the week points of the selected methods have been examined to feel reassured about the final choice.
Conclusions
In this paper, we have reviewed the questions which, in our view, should be answered by an analyst before choosing the method to be used in a decision context. We have proposed a hierarchy of questions which, although quite general, may appear incomplete or inadequate in some specific cases.
 (a)
Her answers to the questions lead her to select a method which she does not control well enough or for which there is no software implementation available. If the analyst does not have enough time to learn the method, or if she is not able to make or order its software implementation, then she will be obliged to disregard this method and reconsider her answers to these questions.
 (b)
Taking into account her answers to the questions, there is no appropriate method. If the analyst can afford (competence, time and financing), she can try to design and implement an appropriate method. Otherwise, she must revisit her answers to these questions, and accept some less satisfactory answers with the aim of permitting to find a method.
The content of the questions, and the diversity of answers that can be given with respect to the decision context, lead us to the conclusion that it is not possible to conceive a family of criteria which would permit a multicriteria formulation of the problem of choosing a multicriteria decision aiding method. The few attempts known from the literature do not seem to be a success (Ozernoy 1988, 1992; Guitouni and Martel 1998). The literature confirms that we are not alone to claim that the choice of an appropriate method is one of the most difficult problems to which the analyst is confronted in multicriteria decision aiding (Belton and Stewart 2002).
The fact that in real world contexts the answers to the nine questions formulated above do not boil down to terms “yes” or “no”, as well as the fact that the shortlisted methods are usually more or less adequate to the considered context, explain why so many methods have been developed, and why it is difficult to compare one method to another in an insightful way. Unfortunately, many researchers are tempted to compare different multicriteria decision aiding methods by basing their conclusions mainly on comparison of results obtained by these methods. The arguments enunciated in this article highlight the fact that such a comparison is illfounded.
Footnotes
 1.
To avoid the complex expression ‘he/she’ when talking about the roles of the analyst and the decision maker, we are using the pronoun ‘he’ for the decision maker, and ‘she’ for the analyst.
Notes
Acknowledgments
The second author wishes to acknowledge financial support from the Polish National Science Centre, Grant No. N N519 441939.
References
 Allais M (1953) Le comportement de l’homme rationnel devant le risquecritique des postulats et axiomes de l’école américaine. Econometrica 21:503–546CrossRefGoogle Scholar
 Allais M, Hagen O (eds) (1979) Expected utility hypotheses and the Allais Paradox. Theory and Decision Library, Reidel, DordrechtGoogle Scholar
 AlmeidaDias J, Figueira JR, Roy B (2010) ELECTRE TRIC: a multiple criteria sorting method based on characteristic reference actions. Eur J Oper Res 204:565–580CrossRefGoogle Scholar
 AlmeidaDias J, Figueira JR, Roy B (2012) A multiple criteria sorting method where each category is characterized by several reference actions: the ELECTRE TRINC method. Eur J Oper Res 217:567–579CrossRefGoogle Scholar
 Augeri MG, Colombrita R, Greco S, Lo Certo A, Matarazzo B, Słowiński R (2011) Dominancebased rough set approach to budget allocation in highway maintenance activities. J Infrastruct Syst 17:75–85CrossRefGoogle Scholar
 Bana e Costa CA (1996) Les problématiques de l’aide à la décision : vers l’enrichissement de la trilogie choixtrirangement. RAIRO/Recherche Opérationnelle 30:191–216Google Scholar
 Bana e Costa CA, Vansnick JC (2008) A critical analysis of the eigenvalue method used to derive priorities in AHP. Eur J Oper Res 187:1422–1428CrossRefGoogle Scholar
 Bana e Costa CA, De Corte JM, Vansnick JC (2005) On the mathematical foundations of MACBETH. In: Figueira JR, Greco S, Ehrgott M (eds) Multiple criteria decision analysis: state of the art surveys, Chap 10. Springer, New York, pp 409–442Google Scholar
 Belacel N (2000) Multicriteria assignment method PROAFTN: methodology and medical applications. Eur J Oper Res 125:175–183CrossRefGoogle Scholar
 Belton V, Stewart TJ (2002) Multiple criteria decision analysis: an integrated approach. Kluwer, BostonCrossRefGoogle Scholar
 Bisdorff R, Meyer P, Roubens M (2008) Rubis: a bipolarvalued outranking method for the best choice decision problem. 4OR 6:143–165CrossRefGoogle Scholar
 Błaszczyński J, Greco S, Słowiński R (2007) Multicriteria classification: a new scheme for application of dominancebased decision rules. Eur J Oper Res 181:1030–1044CrossRefGoogle Scholar
 Bollinger D, Maystre LY, Słowiński R (1997) Un cas concret de la contribution de l’aide multicritère à la décision à la coordination intercantonale pour l’incinération des déchets urbains. In: Proceedings of the 45th meeting of the European working group on multicriteria aid for Decisions, Čelákovice, Czech Republic, pp 5–70Google Scholar
 Boutilier C, Bacchus F, Brafman RI (2001) UCPnetworks: a directed graphical representation of conditional utilities. In: Proceedings of the 17th annual conference on uncertainty in artificial intelligence, pp 56–64Google Scholar
 Bouyssou D, Marchant T, Pirlot M, Perny P, Tsoukiàs A, Vincke Ph (2000) Evaluation and decision models. A critical perspective. Kluwer, BostonGoogle Scholar
 Branke J, Deb K, Miettinen K, Słowiński R (eds) (2008) Multiobjective optimization: interactive and evolutionary approaches. Stateoftheart survey series, LNCS, vol 5252. Springer, BerlinGoogle Scholar
 Branke J, Greco S, Słowiński R, Zielniewicz P (2010) Interactive evolutionary multiobjective optimization driven by robust ordinal regression. Bull Pol Acad Sci Techn Sci 58:347–358Google Scholar
 Brans JP, Mareschal B (2005) PROMETHEE methods. In: Figueira JR, Greco S, Ehrgott M (eds) Multiple criteria decision analysis: state of the art surveys, Chap 5. Springer, New York, pp 163–195Google Scholar
 Choo EU, Atkins DR (1980) An interactive algorithm for multicriteria programming. Comput Oper Res 7:81–87CrossRefGoogle Scholar
 Deb K (2008) Introduction to evolutionary multiobjective optimization. In: Branke J, Deb K, Miettinen K, Słowiński R (eds) Multiobjective optimization: interactive and evolutionary approaches, Chap 3. Stateoftheart survey series, LNCS, vol 5252. Springer, Berlin, pp 59–96Google Scholar
 Dembczyński K, Greco S, Słowiński R (2009) Rough set approach to multiple criteria classification with imprecise evaluations and assignments. Eur J Oper Res 198:626–636CrossRefGoogle Scholar
 Dembczyński K, Kotłowski W, Słowiński R, Szeląg M (2010) Learning of rule ensembles for multiple attribute ranking problems. In: Fürnkranz J, Hüllermeier E (eds) Preference learning. Springer, Berlin, pp 217–247CrossRefGoogle Scholar
 Devaud JM, Groussaud G, JacquetLagrèze E (1980) UTADIS: Une méthode de construction de fonctions d’utilité additives rendant compte de jugements globaux. In: Proceedings of the European working group on MCDA. Bochum, GermanyGoogle Scholar
 Doumpos M, Zopounidis C (2004) A multicriteria classification approach based on pairwise comparisons. Eur J Oper Res 158:378–389CrossRefGoogle Scholar
 Dubois D, Prade H (1987) The mean value of a fuzzy number. Fuzzy Sets Syst 24:279–300CrossRefGoogle Scholar
 Dubois D, Prade H (1988) Possibility theory. Plenum Press, New YorkCrossRefGoogle Scholar
 Dyer J (2005) MAUTmultiattribute utility theory. In: Figueira JR, Greco S, Ehrgott M (eds) Multiple criteria decision analysis: state of the art surveys, Chapt 7. Springer, New York, pp 265–292Google Scholar
 Fernandez ER, Navarro J (2011) A new approach to multicriteria sorting based on fuzzy outranking relations: the THESEUS method. Eur J Oper Res 213:405–413CrossRefGoogle Scholar
 Figueira JR, Mousseau V, Roy B (2005) ELECTRE methods. In: Figueira JR, Greco S, Ehrgott M (eds) Multiple criteria decision analysis: state of the art surveys, Chap 4. Springer, New York, pp 133–162Google Scholar
 Figueira JR, Greco S, Roy B (2009a) ELECTRE methods with interaction between criteria: an extension of the concordance index. Eur J Oper Res 199:496–505CrossRefGoogle Scholar
 Figueira JR, Greco S, Słowiński R (2009b) Building a set of additive value functions representing a reference preorder and intensities of preference: GRIP method. Eur J Oper Res 195:460–486CrossRefGoogle Scholar
 Figueira JR, Greco S, Roy B, Słowiński R (2013) An overview of ELECTRE methods and their recent extensions. J MultiCriteria Decis Anal 20:61–85Google Scholar
 Gonzales C, Perny P (2005) GAI networks for decision making under certainty. In: Brafman R, Junker U (eds) Proceedings of the 19th international joint conference on artificial intelligence—workshop on advances in preference handling, pp 100–105Google Scholar
 Grabisch M, Labreuche C (2005) Fuzzy measures and integrals in MCDA. In: Figueira JR, Greco S, Ehrgott M (eds) Multiple criteria decision analysis: state of the art surveys, Chap 14. Springer, New York, pp 563–608Google Scholar
 Greco S, Matarazzo B, Słowiński R (2001) Rough sets theory for multicriteria decision analysis. Eur J Oper Res 129:1–47CrossRefGoogle Scholar
 Greco S, Matarazzo B, Słowiński R (2002a) Rough sets methodology for sorting problems in presence of multiple attributes and criteria. Eur J Oper Res 138:247–259CrossRefGoogle Scholar
 Greco S, Matarazzo B, Słowiński R (2002b) Preference representation by means of conjoint measurement and decision rule model. In: Bouyssou D, JacquetLagrèze E, Perny P, Słowiński R, Vanderpooten D, Vincke Ph (eds) Aiding decisions with multiple criteriaessays in honor of Bernard Roy. Kluwer, Boston, pp 263–313CrossRefGoogle Scholar
 Greco S, Matarazzo B, Słowiński R (2003) The axiomatic basis of multicriteria noncompensatory preferences. In: Proceedings of the 4th international workshop on preferences and decisions. Trento, pp 81–86Google Scholar
 Greco S, Matarazzo B, Słowiński R (2004) Axiomatic characterization of a general utility function and its particular cases in terms of conjoint measurement and roughset decision rules. Eur J Oper Res 158:271–292CrossRefGoogle Scholar
 Greco S, Matarazzo B, Słowiński R (2005) Decision rule approach. In: Figueira JR, Greco S, Ehrgott M (eds) Multiple criteria decision analysis: state of the art surveys, Chap 13. Springer, New York, pp 507–562Google Scholar
 Greco S, Matarazzo B, Słowiński R (2008a) Dominancebased rough set approach to interactive multiobjective optimization. In: Branke J, Deb K, Miettinen K, Słowiński R (eds) Multiobjective optimization: interactive and evolutionary approaches, Chap 5. Stateoftheart surveys, LNCS, vol 5252. Springer, Berlin, pp 121–155Google Scholar
 Greco S, Matarazzo B, Słowiński R (2008b) Fuzzy set extensions of the dominancebased rough set approach. In: Bustince H, Herrera F, Montero J (eds) Fuzzy sets and their extensions: representation, aggregation and model, Chap III. 3. Studies in fuzziness and soft computing, vol 220. Springer, Berlin, pp 239–261CrossRefGoogle Scholar
 Greco S, Mousseau V, Słowiński R (2008c) Ordinal regression revisited: multiple criteria ranking using a set of additive value functions. Eur J Oper Res 191:416–436CrossRefGoogle Scholar
 Greco S, Mousseau V, Słowiński R (2010a) Multiple criteria sorting with a set of additive value functions. Eur J Oper Res 207:1455–1470CrossRefGoogle Scholar
 Greco S, Słowiński R, Figueira J, Mousseau V (2010b) Robust ordinal regression. In: Ehrgott M, Figueira J, Greco S (eds) Trends in multiple criteria decision analysis, Chap 9. Springer, New York, pp 241–283CrossRefGoogle Scholar
 Greco S, Kadziński M, Mousseau V, Słowiński R (2011) ELECTRE^{GKMS}: robust ordinal regression for outranking methods. Eur J Oper Res 214:118–135CrossRefGoogle Scholar
 Greco S, Kadziński M, Słowiński R (2012) Extreme ranking analysis in robust ordinal regression. OMEGA 40:488–501CrossRefGoogle Scholar
 Greco S, Matarazzo B, Słowiński R (2013a) Beyond Markowitz with multiple criteria decision aiding. J Bus Econ 83:29–60Google Scholar
 Greco S, Mousseau V, Słowiński R (2013b) UTA^{GMS}INT: robust ordinal regression of value functions handling interacting criteria. Eur J Oper Res (to appear)Google Scholar
 Guitouni A, Martel JM (1998) Tentative guidelines to help choosing an appropriate MCDA method. Eur J Oper Res 109:501–521CrossRefGoogle Scholar
 Hammond JS, Keeney RL, Raiffa H (1998) Even swaps: a rational method for making tradeoffs. Harv Bus Rev 76:137–149Google Scholar
 Hwang CL, Yoon KS (1981) Multiple attribute decision making: methods and applications. Springer, BerlinCrossRefGoogle Scholar
 JacquetLagrèze E, Siskos Y (1982) Assessing a set of additive utility functions for multicriteria decision making: the UTA method. Eur J Oper Res 10:151–164CrossRefGoogle Scholar
 Jaszkiewicz A, Ferhat AB (1999) Solving multiple criteria choice problems by interactive trichotomy segmentation. Eur J Oper Res 113:271–280CrossRefGoogle Scholar
 Jaszkiewicz A, Słowiński R (1999) The ‘Light Beam Search’ approach—an overview of methodology and applications. Eur J Oper Res 113:300–314CrossRefGoogle Scholar
 Jaszkiewicz A, Słowiński R, Gawałek A (1995) The ‘Light Beam Search’ over a nondominated set and its application in Polish chemical industry. Cent Eur J Oper Res Econ 3:157–169Google Scholar
 Kadziński M, Słowiński R (2012) Interactive robust cone contraction method for multiple objective optimization problems. Int J Inf Technol Decis Mak 11:327–357CrossRefGoogle Scholar
 Kadziński M, Greco S, Słowiński R (2012) Selection of a representative value function in robust multiple criteria ranking and choice. Eur J Oper Res 217:541–553CrossRefGoogle Scholar
 Kadziński M, Greco S, Słowiński R (2013) RUTA: a framework for assessing and selecting additive value functions on the basis of rank related requirements. OMEGA 41:735–751CrossRefGoogle Scholar
 Keeney RL, Nair K (1976) Evaluating potential nuclear power plant sites in the Pacific Northwest using decision analysis, IIASA Professional Paper no. 761; also in: Bell DE, Keeney RL, Raiffa H (eds) (1977), Conflicting Objectives in Decisions, Chapter 14, Wiley, and in: Keeney RL (1980), Siting Energy Facilities, Chapter 3, Academic Press, New YorkGoogle Scholar
 Keeney RL, Raiffa H (1976) Decisions with multiple objectives: preferences and value tradeoffs. Wiley, New YorkGoogle Scholar
 Keeney RL, Robillard GA (1977) Assessing and evaluating environmental impacts at proposed nuclear power plant sites. J Environ Econ Manag 4:153–166CrossRefGoogle Scholar
 Korhonen P, Wallenius J (1988) Pareto race. Nav Res Log 35:615–623CrossRefGoogle Scholar
 Kotłowski W, Dembczyński K, Greco S, Słowiński R (2008) Stochastic dominancebased rough set model for ordinal classification. Inf Sci 178:4019–4037CrossRefGoogle Scholar
 Labbouz S, Roy B, Diab Y, Christen M (2008) Implementing a public transport line. Multicriteria decisionmaking method that facilitate concertation. Oper Res Int J 8:5–31CrossRefGoogle Scholar
 Léger J, Martel JM (2002) A multicriteria assignment procedure for a nominal sorting problematic. Eur J Oper Res 138:349–364CrossRefGoogle Scholar
 Liesiö J, Mild P, Salo A (2008) Robust portfolio modeling with incomplete cost information and project interdependencies. Eur J Oper Res 190:679–695CrossRefGoogle Scholar
 Martel JM, Roy B (2006) Analyse de la signifiance de diverses procédures d’agrégation multicritère. INFOR 44:191–214Google Scholar
 Maystre LY, Bollinger D (1999) Aide à la négotiation multicritère: pratique et conseils, Chapitre 4, Niveau de centralisation de la gestion des déchets en Suisse Occidentale. Presses polytechniques et universitaires romandes, LausanneGoogle Scholar
 McCord M, de Neufville R (1983) Fundamental deficiency of expected utility decision analysis. In: French S, Hartley R, Thomas LC, White DJ (eds) Multiobjective decision making. Academic Press, New York, pp 279–305Google Scholar
 Merad M, Verdel T, Kouniali S, Roy B (2004) Use of multicriteriaaids for risk zoning and management of a large area subjected to mining induced hazards. Tunn Undergr Space Technol 19:125–138CrossRefGoogle Scholar
 Metaxiotis K, Liagkouras K (2012) Multiobjective evolutionary algorithms for portfolio management: a comprehensive literature review. Expert Syst Appl 39:11685–11698CrossRefGoogle Scholar
 Michalowski W, Rubin S, Słowiński R, Wilk Sz (2003) Mobile clinical support system for pediatric emergencies. Decis Support Syst 36:161–176CrossRefGoogle Scholar
 Miettinen K (1999) Nonlinear multiobjective optimization. Kluwer, BostonGoogle Scholar
 Mousseau V, Dias LC, Figueira J (2003) On the notion of category size in multiple criteria sorting models. Cahier du LAMSADE 205, Université ParisDauphine, ParisGoogle Scholar
 Ozernoy VM (1988) Some issues in designing an expert system for multiple criteria decision making. Acta Psychol 68:237–253CrossRefGoogle Scholar
 Ozernoy VM (1992) Choosing the ‘best’ multiple criteria decisionmaking method. INFOR 30:159–171Google Scholar
 Pawlak Z (1991) Rough sets: theoretical aspects of reasoning about data. Kluwer, DordrechtGoogle Scholar
 Perny P (1998) Multicriteria filtering methods based on concordance/nondiscordance principles. Ann Oper Res 80:137–167CrossRefGoogle Scholar
 Roy B (1996) Multicriteria methodology for decision aiding. Kluwer, DordrechtCrossRefGoogle Scholar
 Roy B (1999) Decisionaiding today: what should we expect? In: Gal T, Stewart TJ, Hanne T (eds) Multicriteria decision makingadvances in MCDM models, algorithms, theory and applications, Chap 1. Kluwer, Dordrecht, pp 11–135Google Scholar
 Roy B (2009) A propos de la signification des dépendances entre critères: quelle place et quells modes de prise en compte pour l’aide à la décision? RAIRO Recherche Opérationnelle 43:255–275CrossRefGoogle Scholar
 Roy B (2010) Two conceptions of decision aiding. Int J Multicrit Dec Mak 1:74–79CrossRefGoogle Scholar
 Roy B, Bouyssou D (1986) Comparison of two decisionaid models applied o a nuclear power plant siting example. Eur J Oper Res 25:200–215CrossRefGoogle Scholar
 Roy B, Bouyssou D (1993) Aide Multicritère à la Décision: Méthodes et Cas. Economica, ParisGoogle Scholar
 Roy B, Figueira J (2013) Discriminating thresholds as a tool to cope with imperfect knowledge in multicriteria decision aiding: theoretical results and practical issues. Cahier du LAMSADE no. 328, Université Paris DauphineGoogle Scholar
 Roy B, Hugonnard JC (1982) Ranking of suburban line extension project on the Paris metro system by a multicriteria method. Transp Res 16A:301–312Google Scholar
 Roy B, Vincke Ph (1987) Pseudoorders: definition, properties and numerical representation. Math Soc Sci 14:263–274CrossRefGoogle Scholar
 Roy B, Słowiński R, Treichel W (1992) Multicriteria programming of water supply systems for countryside. Water Resour Bull 28:13–31CrossRefGoogle Scholar
 Saaty TL (2005) The analytic hierarchy and analytic network processes for the measurement of intangible criteria and for decisionmaking. In: Figueira JR, Greco S, Ehrgott M (eds) Multiple criteria decision analysis: state of the art surveys, Chap 9. Springer, New York, pp 345–408Google Scholar
 Słowiński R (1986) A multicriteria fuzzy linear programming method for water supply system development planning. Fuzzy Sets Syst 19:217–237CrossRefGoogle Scholar
 Słowiński R (ed) (1998) Fuzzy sets in decision analysis, operations research and statistics. The handbooks of fuzzy sets series. Kluwer, BostonGoogle Scholar
 Słowiński R, Teghem J (eds) (1990) Stochastic versus fuzzy approaches to multiobjective mathematical programming under uncertainty. Kluwer, DordrechtGoogle Scholar
 Słowiński R, Greco S, Matarazzo B (2002) Axiomatization of utility, outranking and decisionrule preference models for multiplecriteria classification problems under partial inconsistency with the dominance principle. Control Cybern 31:1005–1035Google Scholar
 Słowiński R, Greco S, Matarazzo B (2009) Rough sets in decision making. In: Meyers RA (ed) Encyclopedia of complexity and systems science. Springer, New York, pp 7753–7786Google Scholar
 Słowiński R, Greco S, Matarazzo B (2012) Rough set and rulebased multicriteria decision aiding. Pesquisa Operacional 32:213–269CrossRefGoogle Scholar
 Steuer RE (1986) Multiple criteria optimization: theory, computation and application. Wiley, New YorkGoogle Scholar
 Szeląg M, Greco S, Słowiński R (2013) Rulebased approach to multicriteria ranking. In: Doumpos M, Grigoroudis E (eds) Multicriteria decision aid and artificial intelligence: links, theory and applications, Chap 6. WileyBlackwell, London, pp 127–160CrossRefGoogle Scholar
 Tsoukiàs A (2007) On the concept of decision aiding process. Ann Oper Res 154:3–27CrossRefGoogle Scholar
 van den Berg J, Kaymak U, van den Bergh WM (2004) Financial markets analysis by using a probabilistic fuzzy modelling approach. Int J Approx Reason 35:291–305CrossRefGoogle Scholar
 Vincke Ph (1982) Preference modelling: a survey and an experiment. In: Brans JP (ed) Operational Research ’81. NorthHolland, Amsterdam, pp 341–353Google Scholar
 von Winterfeldt D, Edwards W (1986) Decision analysis and behavioral research. Cambridge University Press, CambridgeGoogle Scholar
 Wang YJ, Lee HS, Lin K (2003) Fuzzy TOPSIS for multicriteria decisionmaking. Int Math J 3:367–379Google Scholar
 Wang YM, Luo Y, Hua ZS (2008) On the extent analysis method for fuzzy AHP and its applications. Eur J Oper Res 186:735–747CrossRefGoogle Scholar
 Wierzbicki AP (1986) On the completeness and constructiveness of parametric characterization to vector optimization problems. OR Spectr 8:73–87CrossRefGoogle Scholar
 Wierzbicki AP (1999) Reference point approaches. In: Gal T, Stewart TJ, Hanne T (eds) Multicriteria decision makingadvances in MCDM models, algorithms, theory and applications, Chap 9. Kluwer, Dordrecht, pp 91–939Google Scholar
 Wilk Sz, Słowiński R, Michalowski W, Greco S (2005) Supporting triage of children with abdominal pain in the emergency room. Eur J Oper Res 160:696–709CrossRefGoogle Scholar
 Yu W (1992) Aide multicritère à la décision dans le cadre de la problématique du tri: concepts, méthodes et applications. PhD dissertation, Université ParisDauphineGoogle Scholar
 Zadeh LA (1999) From computing with numbers to computing with wordsfrom manipulation of measurements to manipulation of perception. IEEE Trans Circuits Syst I Fundam Theory Appl 45:105–119CrossRefGoogle Scholar
 Zopounidis C, Doumpos M (2000) PREFDIS: a multicriteria decision support system for sorting decision problems. Comput Oper Res 27:779–797CrossRefGoogle Scholar
Copyright information
Open AccessThis article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited.