EURO Journal on Decision Processes

, Volume 1, Issue 1–2, pp 69–97 | Cite as

Questions guiding the choice of a multicriteria decision aiding method

Open Access
Original Article


We formulate some questions that may help an analyst to choose a multicriteria decision aiding method well adapted to the decision context. These questions take into account several aspects of the decision process and of the cooperation between the analyst and the decision maker. We present these questions in a hierarchical order, from the most general and crucial, through other pertinent questions concerning the multicriteria aggregation, to the secondary ones. The initial question is what type of results the method is expected to bring. The next questions concern requirements on preference scales, acquisition of preference information, handling of imperfect knowledge, acceptance of compensation among criteria, and existence of interaction among criteria. The last questions are about intelligibility, axiomatic characterization, and weaknesses of the considered methods. To illustrate these questions, we introduce twelve representative and realistic decision contexts.


Decision process Multicriteria decision aiding Analyst Decision maker 

Mathematics Subject Classification

90B50 91B02 91B06 


We are considering the following situation. In a sufficiently well-defined decision context, a decision process has been started. A manager and/or a ‘task force’ team, called decision maker, is confronted with a problem of making ‘the best possible’ decision. In view of this challenging task, the decision maker has appointed a team of consultants, called the analyst,1 which is expected to clarify the decision situation and to provide some recommendations. We suppose that:
  • The goal of the decision process, as well as possible ways of achieving it, has been discussed sufficiently well to define a set of potential actions (alternatives).

  • The need to account for multiple and somehow conflicting points of view has been recognized and, based on this, a family of criteria or attributes has been outlined.

  • The analyst has acquired a good knowledge of the decision context, and of the possibilities of interactions she may have with the decision maker (or his representative), and other stakeholders, in order to get all necessary information.

Consequently, we suppose that the analyst has arrived at the stage of reflection where she is about to choose the most suitable multicriteria method to be used within the decision process. In our opinion, this method should be seen as a tool for going deeper into the decision problem, for exploring various possibilities, interpreting them, debating and arguing, rather than a tool able to make the decision. We suppose further that the model of preferences used by the method is, at least partially, co-constructed through interaction between the analyst and the decision maker (or his representative). This co-construction should account for the consequences on which the actions will be judged and for value systems related to the decision context. It should also help the analyst to formulate and express the working hypotheses on which her advice will be based. Moreover, once the method has been chosen, the analyst will collaborate with the decision maker (or his representative) to specify certain characteristics (notably the values attributed to the different parameters) of the preference model that the method requires.

It follows from the above that, in our understanding, the method to be chosen is not expected to discover any good approximation of an objectively best decision, taking into account a pre-existing preference system of the decision maker, but rather more modestly, it is expected to provide the decision maker with results that follow from an adopted way of reasoning, consistent with the working hypotheses (Roy 1999, 2010).

This paper presents questions to guide in the choice of a multicriteria decision aiding method. These questions are presented in a hierarchical order. In our opinion, answering these questions in this order may help the analyst choose the most appropriate method for the decision context. In “A number of actual decision contexts”, we present a number of such contexts to illustrate the diversity of decision situations and to demonstrate further how they are conditioning the analyst’s answers to subsequent questions. The first of these questions—which seems to us to be crucial—is analyzed in “A crucial question conditioning the choice of the method by the analyst”. “Five other key questions to choose the right method” presents the first series of key questions which has to be tackled in relation to the first one. In “Secondary questions”, we present another series of questions that may be addressed to the analyst, which, although realistic, seems to be less general than the previous ones. In “Conclusions”, we highlight two difficulties that the analyst can face in her choice of a multicriteria decision aiding method.

It should be emphasized that we do not aim at providing the reader with strict conditions under which such or such method is the most appropriate, but rather at providing guidelines facilitating the choice of a multicriteria decision aiding method well adapted to the specific context of a case study.

A number of actual decision contexts

Here, we present briefly a number of actual decision contexts to which we will refer later on. Most of these concern a real case that one of us has studied. Note that the list of contexts presented below does not pretend to be representative of the use of MCDA methodology in real world decision problems: it only aims at helping the reader to better understand the arguments that are put forward in the second part of the paper.

At the first reading, this section can be skipped. The reader will be sent back to each of these contexts later, while going along the following sections.

Context no. 1: Commuter rail line. In a large urbanized area, the policy makers have decided to improve the public transportation system. To increase access to different zones, especially for residents of a growing suburb, while using the existing infrastructure as much as possible, the decision to build a rail link (tramway) between the suburb and the employment zone has been taken. Providing such a service does not present any major problem in terms of the line’s layout. The difficulties arise, rather, from design issues related to the number and location of stations as well as capacity-related features. A dozen of variants have already been considered, and all the interested parties have already agreed on five general objectives: (1) minimize investment and operating costs, (2) minimize access time to the stations and line haul times along the rail line, (3) improve the compatibility among urban development, employment, and the transportation system, (4) maximize the well-being of the transport users (increase comfort, safety, etc.), (5) avoid environmental disruption as much as possible. More details about this problem context can be found in example 2 of Roy (1996) and in Labbouz et al. (2008).

Context no. 2: Siting of a nuclear power-plant. The public power supply authority of a country planned to build a nuclear power-plant on a seaside. At the time the consultants’ bureau was called to work on this problem, nine potential sites for a technically feasible placement of the power-plant had already been identified. In order to evaluate and compare these sites, six points of view were chosen: (1) the health and security of the population in the surrounding region, (2) the loss of salmonids in streams absorbing the heat from the power-plant, (3) the biological effects on the surrounding region, (4) the socio-economic impact of the installation, (4) the esthetic impact of the power lines, (5) the investment cost and (6) the operating cost of the power-plant. Further details may be found in Keeney and Nair (1976), Keeney and Robillard (1977), and Roy and Bouyssou (1986).

Context no. 3: Location of a municipal waste incineration plant. The Swiss federal law is charging the cantons with responsibility for installing depolluting plants, in particular, municipal waste incineration plants (MWIP). To encourage cantons to build these plants, the Swiss government gives them a subvention for installing a MWIP. This resulted in overcapacity of MWIPs at the country level. In consequence, when at the end of 1990s two neighboring cantons, Vaud and Fribourg, applied to the government for a subsidy for building their own MWIP, the government asked them to co-ordinate their projects and consider using overcapacity from the neighboring cantons, or to extend the MWIP in Vaud instead of building two new MWIPs. To work out a recommendation for the consensus decision, a body representing four stakeholders was established. These were: Swiss Federal Agency for the Environment, Forests and Landscape, cantonal environmental offices of Vaud and Fribourg, and a representative of a MWIP of Geneva having a big overcapacity. This body invited help from two facilitators (the analyst) from the Federal Polytechnic School of Lausanne (D. Bollinger and R. Słowiński). The five basic scenarios were considered:
  • no new MWIP is build, and the waste of Vaud and Fribourg are distributed over existing plants,

  • two MWIPs are built, one in Vaud and another in Fribourg,

  • one MWIP is built, in Vaud only,

  • one MWIP is built, in Fribourg only,

  • one MWIP is built, at the frontier of Vaud and Fribourg.

Considering different ways of distributing the waste, and a possible extension of the existing MWIP in Vaud (TRIDEL in Lausanne), the ‘task force’ team constructed 17 potential actions. The construction of these potential actions was followed by construction of a consistent family of criteria. This family included 20 criteria, covering diversified aspects such as: ecology (3), economy (5), organization (4), law (3), and psychology (5). One can notice a strongly non-compensatory character of such heterogeneous criteria. Moreover, the four stakeholders had different views on the relative importance of each criterion. The construction of actions and criteria required a substantial amount of work by the body of the four stakeholders helped by the facilitators and by an engineers’ office employed for this purpose. More details can be found in Bollinger et al. (1997) and Maystre and Bollinger (1999).

Context no. 4: Line extension of the Paris metro. Due to important development of the metropolitan Paris, the metro lines set up in the past needed to be extended. At the end of 1980s, the Régie Autonome des Transports Parisiens (RATP) elaborated 12 new metro extension projects into the suburbs, which added up to 42.6 km, and consisted of the construction of 38 new stations, doubling the number of stations in the suburbs. The decision about a priority ranking of these projects was not within the purview of the RATP: it was in charge of national and regional agencies responsible for planning and programming the infrastructure. RATP was chartered, however, to elaborate all the technical, financial, economic and social considerations, necessary to pointing up all the priorities which may validly be set up, independently of any value system pertaining to RATP. To clarify the decisions concerning the time-dependent construction of the various segments of such line extensions, the department of Operational Research of the RATP was in charge of their evaluation and, to a possible extent, of their classification, taking into account the following six criteria of which each reflects a specific point of view of one of the stakeholders: (1) the number of residents and jobs served by the project, (2) the number of daily passengers entering the stations on the line concerned, (3) the capital cost of the project per 1 km of line, (4) the internal rate of return, (5) the advisability of the extension with respect to the general organization of the transit systems in the considered sector, and (6) the structuring effects on urban development. In order to fix a performance on a particular criterion for each extension, it was necessary to rely on many estimations of the population affected, cost elements, value of time and discount rate. One had also to adopt prospective views on such issues as urban development, and the behavior of public transport users. It is thus evident that the validity of any such assessment was subject to a margin of error. A report of this study can be found in Roy and Hugonnard (1982).

Context no. 5: Supplier selection. A postal company sought to equip its regional centers with parcel sorting machines. Toward this end, it announced an international tender for commissioning a prototype (it reckoned that this type of machines is not yet quite ready). This tender consisted of specifications and an upper bound on the cost of the basic version of the machine. The selected supplier was to receive the order for supplying all sorting centers. Nine responses to the tender were preselected and evaluated on a set of 12 qualitative and quantitative criteria, such as: quality of workstations, noise pollution, capital cost, operating cost, sorting speed, ease of use, maintenance cost, ease of installation on the spot, possibility of sorting bar-coded parcels, quality of service, confidence in the supplier (Roy and Bouyssou 1993, chapter 8). The quantitative criteria involved very heterogeneous scales. A consulting firm was appointed to help the decision maker that was the board composed of four company directors: technical, financial, commercial, and human resources. The board requested that the consulting firm would carefully take into account the extent to which:
  1. 1.

    The evaluation of responses on some criteria was partially uncertain or even arbitrary.

  2. 2.

    The four directors had different views on the relative importance of each criterion.

Context no. 6: Responses to tenders. A big company is carrying out a considerable part of its research and innovation work by replying and winning some tenders for bids it regularly receives. Replying to a tender needs, however, many months of work, and sometimes calls for starting up research which has high material costs. For this reason, every week, a committee chaired by the sales manager in charge of the tender budget examines the “new business” files received. Each of these files comes from a service within the company, which has suggested replying to a tender it has received. Each of these is treated as a potential action. The committee must decide for each of them whether to accept or refuse and, in the case of acceptance, how much money to allocate to the service in question for developing a response on behalf of the company. Using information in the file, the reply proposal is evaluated with regard to nine criteria that span three structure-providing points of view: (1) chances of being awarded the contract, (2) strategic interest for the company, and (3) economic interest. In order to organize the decision process, the sales manager has asked the research unit of the company to develop a software tool which makes it possible to sort every week the “new business” files and to assigning each file to one of the following four categories:
  • no restriction concerning acceptance,

  • some hesitation concerning acceptance (a doubtful “yes”),

  • hesitation concerning refusal (a doubtful “no”),

  • unhesitating refusal.

To foster a debate within the committee, each of its members gets information about the nine evaluations of each file, and about the category to which each of them was assigned. Any member of the committee may query any item of the information leading to the assignment (assessment of certain risks, allocation of resources other than those requested, etc.). When some items of information are thus modified, new evaluations are immediately recalculated and the resulting reassignment is made known to committee members.

Context no. 7: Management of highway assets. A road network is managed by a central agency that needs to coordinate and control the activities of many local districts spread over a wide geographic area. Considering that the funds required to satisfy the needs of maintenance usually exceed the available resources, the decision about annual budget allocation for routine maintenance is crucial to achieve the best possible efficiency of the road network. In this decision context, there are many decision levels and a hierarchy of stakeholders with different perspectives and specific objectives. The large amount of data makes the decision process very complex. Thus, highway agencies need tools for the coordinated management of their assets that allow interactions between the stakeholders and the analyst in course of the allocation of available funds according to their preferences.

A case study referring to this context is described by Augeri et al. (2011). Specifically, this study concerns the distribution of maintenance resources from a central administration to regional districts and takes into account local agencies’ maintenance needs and the central authority’s goals in a short-term planning period. A network composed of a number of road sections belonging to an Italian highway agency was used as a pilot study for the proposed methodology. The considered maintenance activity concerned the pavement. The family of criteria was composed of 11 very heterogeneous criteria representing the following aspects of evaluation of the road sections:
  • type of distress recorded during the periodical survey,

  • geometric characteristics of the road section,

  • road functional class,

  • intensity of traffic,

  • accident rate.

The road sections were sorted by road experts into four categories of urgency. The task was to develop an intelligible decision model that would reproduce the expert decisions on the road sections and would facilitate prospective decisions about the degree of urgency of the pavement on new road sections.

Context no. 8: Programming of water supply systems for rural areas. The construction of a new water supply system is usually preceded by regional planning which accounts for long-term water resources management and designing of water supply installations. The intermediate stage between planning and designing is called the programming stage. At this stage, a medium-term decision problem is to be considered which involves socio-economic criteria on the one hand and technical criteria on the other hand. In Poland, in the beginning of 1990s, some rural areas, in particular in the East part of the country, did not have water supply systems (WSS). Regional agencies of rural investments were facing then a hard decision problem, that is, how to program the construction of a WSS in a given area, so as to connect water users (understood as topographically compact groups of receivers, like villages, big farms or food processing plants) in a priority order respecting the urgency of their needs. Roy et al. (1992) consider a case study in which this complex task has been decomposed into two problems:
  1. (a)

    setting up a priority order in which water users should be connected to a new WSS, taking into account economic, agricultural, and sociological consequences of the investment,

  2. (b)

    choosing the best variant of technical construction of the regional WSS evaluated from technical and economic viewpoints and from the viewpoint of relationship with the priority order of users coming from problem (a).


The study took into account both purely technical and socio-economic aspects of WSS programming in the form of distinct criteria. A single WSS typically concerns a set of 20–40 users. In problem (a), they were evaluated with respect to the following criteria: water deficiency, farm production potential, function and activity of the user, structure of settlements, water demand, share of water supply installations in all investments concerning the user, possibility of connecting the user to another existing WSS. The data used to calculate user performances on these heterogeneous criteria were affected by some imprecision and indetermination. As to problem (b), there can be hundreds of technical variants, because it involves different types and locations of water sources, types and capacities of system components, and feasible structures of the distribution network. Each feasible variant was characterized by four criteria: (1) investment and (2) operating costs, (3) reliability and (4) a distance between the socio-economic priority order of users, and the precedence order of users connected to the WSS constructed according to a given variant. The last criterion played a coordinating role between both problems of the programming task. The recommended variant was the one which ensured the best compromise between the four criteria.

A similar problem has been considered by Słowiński (1986), however, with respect to development planning of a jointly operated urban water supply and wastewater treatment system in a 20-year planning horizon. This problem has been formulated as a multiobjective LP problem, where the variables were daily water flows on main pipeline connections between sources and users, and daily inflows of wastewater to discharging treatment plants, both of them in consecutive time periods. The objectives were: expansion and operating cost of (separately) water intakes, recycling treatment plants and discharging treatment plants, reliability of water supply, and environmental quality. Because there was no precise data about cost and reliability coefficients, water pollution indices, discount factors and the user’s demands, the experts accepted to specify for each of them an interval of the most possible values that were included in an interval of least possible although realistic values. This corresponded to the definition of trapezoidal fuzzy numbers and lead to a fuzzy multiobjective LP formulation.

Context no. 9: Clinical decision support in emergency room. Abdominal pain in childhood is a highly prevalent symptom caused by organic diseases, psychosocial disturbances and emotional disorders. In many cases, the exact cause is never known. Medical staff must focus on identifying a minority of cases in need of urgent treatment. The child who complains about abdominal pain is initially examined in the emergency room by a medical intern. The possible outcomes of this evaluation are: ‘discharge’, ‘surgical consult’, and ‘in-hospital observation’. A limited number of clinical signs, symptoms and tests, available at the early stage, make such a triage very difficult. To increase the accuracy of the triage, the Children’s Hospital of Eastern Ontario in Ottawa (CHEO) called for a study aiming at developing a clinical decision support system that would assist medical interns in the emergency room. In order to learn diagnostic patterns from past diagnoses made by surgeons, CHEO provided a data set including records of 647 patients with abdominal pain seen during a 3-year period in the emergency room of CHEO. The patients were described by 12 early stage symptoms, called attributes. The data was collected as part of a retrospective chart study, and thus were not complete (10–20 % of missing values). Most of the attributes were nominal (gender, type of pain), some were numerical (age, duration of pain, temperature, white blood cell count), some were binary (vomiting, recent visit in emergency room, muscle guarding, rebound tenderness), and some indicated a location of a condition on patient’s abdomen (location of pain, location of tenderness). According to medical practice, patient records stored in the data set were sorted into three categories: ‘discharge’, ‘surgical consult’, and ‘in-hospital observation’. The focus of the study—which is reported in Michalowski et al. (2003) and Wilk et al. (2005)—was on inducing from the data set an intelligible decision model consistent with the past decisions of specialists, involving the most relevant attributes from among the 12 available attributes. This decision model has been designed to support emergency room staff and has been embedded into a decision support system on a mobile platform, called MET (Mobile Emergency Triage).

Context no. 10: Credit granting. Every day, bank B receives several credit applications from various firms wishing, for example, to put a hotel or a clinic, or to buy machines for some public works. These files are submitted to a credit evaluator whose mission is to consider all of them and decide about the sort of each particular application. This credit evaluator would like to get a help from operational research department of B, by asking it to develop a software tool that would make a preliminary sorting of the incoming files. He wants that the sorting is based on performances of each file on a family of criteria that takes into account a list of viewpoints he provides. These viewpoints cover three main concerns: profitability, the risk of mortgage non-payment, and commercial impact. The family of criteria has to take into account how the application looks like (in particular, what is motivation, requested amount and duration), various characteristics of personality and situation of the applicant (in particular, legal status of the applicant’s enterprise, age of managers, balance sheets from recent years), as well as the history of possible past applications and commercial relations of bank B with the applicant. As soon as a new application for credit arrives, an assistant of the credit evaluator analyzes it in view of defining its performances on the family of criteria. Taking into account an imprecision, or even a lack of some data, which makes the definition of performances highly subjective, the assistant of the credit evaluator can be led to fix not one but two performances per criterion, which correspond to optimistic and pessimistic value, respectively. The credit evaluator requires that on this base each file is assigned to one of the following categories:

Category C1: Files apparently bad that should be rejected after a quick verification.

Category C2: Files rather bad that, for some commercial reasons, could be nevertheless accepted and that have to be transferred to another department.

Category C3: Files rather good that nevertheless need to be completed with some additional information before being carefully examined.

Category C4: Files apparently good that should be accepted after a quick verification.

If necessary, the credit evaluator would like a fifth category to be appended:

Category C5: Atypical files that could hardly be assigned to one of the above four categories.

The multicriteria decision aiding method used to assist the credit evaluator in the above context has been implemented as a computer program which served many years and remained confidential. Even if it was more complex, this method inspired the ELECTRE TRI method (Yu 1992; Roy and Bouyssou 1993).

Context no. 11: Monitoring of risk zones. One is considering here a territory T where some old iron mines exploited using the ‘room and pillar’ caving method present a risk of collapse or ground subsidence causing major damages to surface buildings and infrastructures. T has been partitioned into zones which exhibit homogeneous characteristics of the underground and of the surface infrastructure (segments of highways and other roads, schools, commercial centers, public buildings, apartment buildings, entertainment parks, etc.). For more details, see Merad et al. (2004). A family of criteria has been defined to assess the gravity of risk that a zone may present based on its characteristics. The risk prevention authority responsible for the monitoring of territory T is wishing that each zone is assigned to one of the following categories:

Category C1: Very low risk zones requiring reference leveling (topographic surveys) only.

Category C2: Low risk zones requiring reference and annual leveling.

Category C3: Zones of risk sufficiently high for an in depth investigation including geological boring if necessary.

Category C4: High risk zones requiring long-term continuous monitoring based on recording of underground microseismic activity.

Criteria used for evaluation of the homogeneous zones are grouped under two headings: ‘susceptibility of the mine to collapse’ and ‘surface sensitivity’. The first refers to the ‘probability of rupture’ and the second to both the ‘intensity of the rupture’ and the ‘value and vulnerability of assets’. In the first group, there are criteria such as: mean stress applied on pillars, existence of fault, superimposition of pillars, size and regularity of pillars, and sensitivity of rock to flooding. The second group has criteria such as: depth of the top mined layer, maximum expected subsistence, expected surface deformation, zone extent, and vulnerability of buildings, roads, railways, bridges and various networks (electricity, water, gas). For some criteria the risk grows with the performance, and for the others, it decreases.

This is a complex decision-making problem in which the available information is uncertain (missing information, such as geological data) and imprecise (mining works maps), and in which knowledge is incomplete (e.g., soil-structure interaction). The risk prevention authority would like to be supported by a method permitting identification and sorting of homogeneous zones into the four predefined risk categories.

Context no. 12: Engineering design of a chemical reactor. In the engineering design of a chemical reactor for the production of p-xylene, it is necessary to determine the most appropriate parameters for the efficient production of this compound. It is used as an intermediate in production of various plastics such as: polyesters and polyamides. p-Xylene is to be produced by isomerization of o-xylene over H-modernite catalyst in a flow reactor. A kinetic model of this reaction, defined by a set of differential equations, is a basis for formulation of a multiobjective nonlinear programming problem. The decision (design) variables are: temperature and pressure of the process, reactor volume, feed flow rate and catalyst weight. The feasible values of the decision variables are constrained by technological requirements and by equations following from the kinetic model. The values of the decision variables should give the best compromise between four conflicting objectives. The first objective—reactor volume, to be minimized—represents the designer’s attitude to reduce the size of the chemical installation. It influences both investment and operating costs of the reactor. The second objective—catalyst weight, to be minimized—expresses the designer’s aspiration to reduce the weight of the H-modernite used in the reactor. The third objective—mass production of p-xylene—is to be maximized. The last objective—concentration ratio between p-xylene and o-xylene, to be maximized—is correlated with the quality of the final product and with the level of transformation of o-xylene, i.e. with the efficiency of the production process (Jaszkiewicz et al. 1995). One can notice that the objectives involve heterogeneous scales. Moreover, as the kinetic model of the reaction is an approximation of the real process, the performances of feasible solutions on particular objectives should also be considered as approximate during the search of the best compromise solution.

A crucial question conditioning the choice of the method by the analyst

To make the choice of a multicriteria decision aiding method, the analyst should, in our opinion, start with reflecting on the best or even the only way of answering the following essential question:

Taking into account the context of the decision process, what type(s) of results the method is expected to bring, so as to allow elaboration of relevant answers to questions asked by the decision maker?

The type of results produced is a feature which distinguishes various methods of multicriteria decision aiding. Depending on the decision context, this is not the same type of results that may bring useful information able to guide the decision aiding process in the right way, and to work out some conclusions, or even a recommendation. Moreover, the type of results is conditioning the way in which the analysis is inserted in the decision process. The analyst should keep all this in mind when answering the above question.

A review of multicriteria decision aiding methods (well defined and sufficiently operational to be taken into account by the analyst) led us to distinguish five main types of results that the analyst may want to consider in relation to methods clearly associated with them. Other types of results corresponding to different decision problems could also be considered (Bana e Costa 1996; Tsoukiàs 2007) but, to our knowledge, there is no well-established method that could produce them.

At this stage of reflection, the analyst can hesitate among several types of convenient results so that she can keep more than one type at the moment. Even if she selects only one type, the method producing this type of results may not be unique. Consequently, in many cases, the answer to the above initial question may lead at this stage to a short list of more than one method. The final choice of a particular method will result from the answers to questions formulated in “Five other key questions to choose the right method” and “Secondary questions”.
  1. (a)

    Type 1: A numerical value (utility, score) is assigned to each potential action.


It is possible that this type of result is imposed by the decision maker. This could be the case of contexts: Commuter rail line (1), Siting of a nuclear power-plant (2), and possibly Responses to tenders (6). Many methods can produce this type of results: MAVT (Keeney and Raiffa 1976), MAUT (Dyer 2005), UTA (Jacquet-Lagrèze and Siskos 1982), MACBETH (Bana e Costa et al. 2005), AHP, (Saaty 2005), SMART (Winterfeld and Edwards 1986), TOPSIS (Hwang and Yoon 1981), Choquet Integral (Grabisch and Labreuche 2005), representative value function of UTAGMS (Kadziński et al. 2012), and others.

The application of these methods requires (either in course of applying the method, or even before) that the scale of each evaluation criterion is an interval scale. Sometimes, these scales have to be identical. Constructing such scales calls for interacting with the decision maker (or his representative) in a way which is specific for each of these methods. The analyst should keep this requirement in mind when drawing up a short list of methods.

In some contexts, for example, Location of a municipal waste incineration plant (3), Line extension of the Paris metro (4), or Supplier selection (5), one should take into account some viewpoints for which the definition of associated criteria, as well as the data necessary for defining performances of each action on these criteria, can involve some arbitrariness, uncertainty or, more generally, some indetermination. Such an indetermination can be handled using probabilistic or fuzzy modeling, however, this modeling can also be arbitrary to some extent [see, e.g., Fuzzy AHP (Wang et al. 2008) and Fuzzy TOPSIS (Wang et al. 2003)]. This difficulty can be bypassed through sensitivity analysis, when there is only one or two viewpoints that need the handling of the impact of this indetermination. It can happen, however, that the diversity and importance of the sources of indetermination make it difficult to assign a numerical value (or even a small interval) to each of the potential actions. For this reason, the analyst may give up the idea of requesting this type of results.
  1. (b)

    Type 2: The set of actions is ranked (without associating a numerical value to each of them) as a complete or partial weak order.


This type of result can be considered only if the set A of potential actions is known a priori. It is not convenient when the potential actions are examined as they arrive (week after week, month after month, etc.); this is the case of such contexts as Clinical decision support in emergency room (9), or Credit granting (10). It seems well adapted to such contexts as Line extension of the Paris metro (4), or Programming of water supply systems for rural areas (8).

The methods relevant here are: ELECTRE III, IV (Figueira et al. 2005, 2013), PROMETHEE I and II (Brans and Mareschal 2005), all Robust Ordinal Regression methods (Greco et al. 2010b) producing necessary and possible rankings, like UTAGMS (Greco et al. 2008c), GRIP (Figueira et al. 2009b), Extreme Ranking Analysis (Greco et al. 2012), RUTA (Kadziński et al. 2013), ELECTREGKMS and PROMETHEEGKMS (Greco et al. 2011), and moreover, the Dominance-based Rough Set Approach to ranking (Greco et al. 2001; Słowiński et al. 2009; Szeląg et al. 2013), and Machine Learning approach (Dembczyński et al. 2010).

Note, moreover, that the results of type 1 and 2 are well adapted to the case when the expected result is a list of k-best actions that are diverse enough and should be analyzed further by the decision maker.
  1. (c)

    Type 3: A subset of actions, as small as possible, is selected in view of a final choice of one or, at first, few actions.


As in the case of type 2, this type of result is not convenient when the set A of potential actions is not known a priori. It is also not convenient in the contexts like Line extension of the Paris metro (4), Management of highway assets (7), Programming of water supply systems for rural areas (8), Clinical decision support in emergency room (9), or Monitoring of risk zones (11). In these contexts, many potential actions, and not only one, are intended for joint execution. The type of result considered here is convenient when the potential actions are modeled as alternatives, i.e. such that a joint execution of any two of them is excluded. This is the case of such contexts as Commuter rail line (1), Siting of a nuclear power-plant (2), Location of a municipal waste incineration plant (3), Supplier selection (5), or Engineering design of a chemical reactor (12).

This type of results is produced directly by such methods as: ELECTRE I and IS (Figueira et al. 2005, 2013), PROMETHEE V (Brans and Mareschal 2005), Rubis (Bisdorff et al. 2008). The methods of multiobjective optimization (Branke et al. 2008) also lead to this type of results, however, they are applied when actions of the set A are vectors of variables subject to some mathematical programming constraints. These methods will also be considered later for type 5 of results.

Note that this type of result is also relevant to the case of a multicriteria choice of the best portfolio of objects with cardinality and cost constraints. In this case, the set A is composed of alternatives that represent those combinations of objects which respect cardinality and cost constraints. The methods designed for this case are discussed, e.g., by Liesiö et al. (2008), Metaxiotis and Liagkouras (2012) and Greco et al. (2013a).

Remark, moreover, that the methods considered for type 1 and type 2 of results can also be used in this case: the top ranked actions can be seen as result of type 3. Thus, when the set A is defined as a set of alternatives, the analyst can put on the short list quite a few methods.
  1. (d)

    Type 4: Each action is assigned to one or several categories, given that the set of categories has been defined a priori.


It is possible that this type of results is imposed by the decision maker. This could be the case of such contexts as: Responses to tenders (6), Management of highway assets (7), Clinical decision support in emergency room (9), Credit granting (10), or Monitoring of risk zones (11). This type of results is particularly well adapted to the contexts where the set A is not defined a priori, like Clinical decision support in emergency room (9) and Credit granting (10). It can also be convenient to presort when a large number of potential actions have been listed at the starting point of the decision process (Jaszkiewicz and Ferhat 1999). This could happen in such contexts as: Commuter rail line (1), or Siting of a nuclear power-plant (2). Such a presorting is also used in some interactive multiobjective optimization procedures, where it concerns a number of non-dominated solutions proposed for evaluation to the decision maker in each dialogue phase (Greco et al. 2008a).

Quite various methods provide this type of results. Let us mention those that rely on Dominance-based Rough Set Approach (Greco et al. 2001, 2002a, 2005; Błaszczyński et al. 2007; Dembczyński et al. 2009; Słowiński et al. 2009), UTADIS (Devaud et al. 1980), PREFDIS (Zopounidis and Doumpos 2000), UTADISGMS (Greco et al. 2010a), ELECTRE TRI-B (initially ELECTRE TRI) (Figueira et al. 2005; Yu 1992), ELECTRE TRI-C (Almeida-Dias et al. 2010), ELECTRE TRI-NC (Almeida-Dias et al. 2012), filtering method (Perny 1998), PROAFTN (Belacel 2000), TRINOMFC (Léger and Martel 2002), PAIRCLASS (Doumpos and Zopounidis 2004), THESEUS (Fernandez and Navarro 2011), among others. These methods are distinguished by many features, in particular by: ordered or non-ordered categories, the way of defining the categories, the hypotheses and logical foundations of the assignment procedure, the nature of requested preference information. One can observe, however, a lack of sorting methods that take into account some additional constraints on the categories, like a balanced composition of the categories (for example, men and women). Recently, however, a sorting method called DIS-CARD has been proposed (Kadziński and Słowiński 2012), which takes into account desired cardinalities of the categories. Another tentative of handling constrained sorting problems has been made by Mousseau et al. (2003).
  1. (e)

    Type 5: A subset of potential actions enjoying some remarkable properties is provided to serve as a base in the following stage of the decision aiding process.


This type of result may be required when the set of alternatives A contains a very large number of actions (more than one hundred). This is, for example, the case when the actions are defined by vectors of variables subject to some mathematical programming constraints. Then, it may be interesting to get a restricted subset of A, called A′, composed of actions enjoying some remarkable properties, and then replace A by A′ at later stages of the decision aiding process. In multiobjective optimization, the set A′ is a set of non-dominated actions (also called efficient or Pareto-optimal solutions) or an approximation of this set. Mathematical foundations of the completeness and constructiveness of parametric characterization of the set of non-dominated actions has been given by Wierzbicki (1986). From a practical point of view, evolutionary algorithms appeared to be particularly effective in finding a good approximation of the set of non-dominated actions in multiobjective optimization (Deb 2008).

This type of result is particularly welcome in interactive multiobjective optimization, where the set A′, being a complete set of non-dominated actions or its (approximate) representation, is a base for an interactive procedure leading to some best compromise actions. Interactive procedures are composed of two alternating phases: computation phase and dialogue phase. In the computation phase, one or several non-dominated actions are found in A′ and presented to the decision maker. Then, in the dialogue phase, the decision maker is criticizing the proposed actions unless one of them is completely satisfactory. In the latter case the procedure stops. Otherwise, the critical evaluation of proposed actions is used as preference information to guide the search of one or several non-dominated actions belonging to A′ in the next computation phase, with the intention of better fitting the decision maker’s preferences. The way of using the set A′ in this procedure is distinguishing two major categories of interactive procedures:
  1. (1)

    procedures based on exploration of A′,

  2. (2)

    procedures based on progressive contraction of A′.


Typical examples of category (1) are procedures using reference points defined by aspiration levels in the criteria space. The reference points are projected onto the set A′ in order to find the ‘closest’ non-dominated actions to be proposed to the decision maker. The projection is done using Chebyshev-like achievement (scalarizing) functions. Changing the reference point, one can browse the whole set A′. The reference point approaches have been described by Wierzbicki (1999). Some implementations of the projection principle give to the user an impression of driving a vehicle over the non-dominated set—this is, for example, the case of Pareto race (Korhonen and Wallenius 1988) or NIMBUS method (Miettinen 1999).

Interactive procedures of category (2) present to the decision maker in each dialogue phase a sample of non-dominated actions picked from a gradually reduced region of the set A′. A typical example is the method of Choo and Atkins (1980). The interesting subregion of the set A′ is often delimited by a polyhedral cone with the origin in a reference point, oriented toward the non-dominated set. The spread of this cone is controlled by interval values of weights assigned to criteria in the weighted Chebyshev achievement function, like in the cone contraction method (Steuer 1986). In the ‘Light Beam Search’ method (Jaszkiewicz and Słowiński 1999), the spread of the cone is defined by an outranking relation between a non-dominated action, called middle point, resulting from projection of a reference point onto the set A′, and its neighborhood actions, such that the subregion of A′ is composed of non-dominated actions which are not worse than the middle point, i.e. outrank the middle point. In the robust cone contraction method (Kadziński and Słowiński 2012), the spread of the cone with the origin in a reference point is defined by directions of the isoquants of all weighted Chebyshev achievement functions compatible with pairwise comparisons of some non-dominated actions from the current set A′, specified by the decision maker. In successive iterations, each new pairwise comparison contracts the cone which is zooming on a subset of non-dominated actions of greatest interest for the decision maker. It is also worth mentioning the NEMO method which combines an evolutionary multiobjective optimization with robust ordinal regression in an interactive procedure (Branke et al. 2010); this combination allows speeding up convergence to the most preferred subregion of the set A′.

Note that, independently of category (1) or (2), the final result of interactive multiobjective optimization methods is of type 3.

The context fully adequate to type 5 of results is Engineering design of a chemical reactor (12). It may also be useful in Programming of water supply systems for rural areas (8), in part (b) concerning the choice of the best variant of technical construction of the regional WSS, and, particularly, in development planning of a jointly operated urban water supply and wastewater treatment system, formulated as a multiobjective fuzzy LP problem.

Five other key questions to choose the right method

The analyst is advised to answer these questions while considering successively various methods short-listed in reply to the initial question about the type of expected results. The way these questions are ordered below does not suggest any priority in answering them. The decision context, in which the analyst plays her role, and the methods she has short-listed, can influence the order of examination of these questions.
Question 1a

Do the original performance scales have all required properties for a rightful application of the considered method?

Some methods (in particular those mentioned in “A crucial question conditioning the choice of the method by the analyst” in point (a)) cannot handle directly the evaluation criteria whose performances are located on verbal scales, or even numerical but purely ordinal scales. When the answer to question 1a is “no”, the analyst has to check if it is possible to code or transform in a meaningful way the original scales, such that the properties of scales required by the considered method are satisfied. The analyst can carry out this check by looking for an additional information, especially in the course of interaction with the decision maker (or his representative). In order to decide about keeping or rejecting the considered method, the analyst will have to examine to what extent the transformations are arbitrary, and how much the numerical coding of performances masks their meaning. If among the short-listed methods there are some methods that do not require this type of transformation, she will have to assess the trade-off between the advantage of keeping the original scales (for interpretation of results and organization of a debate about them) and the inconvenience that may have these alternative methods.

In the case study referring to the context of Siting of a nuclear power-plant (2), Keeney and Nair (1976) had to recode the original performances such that the selected method (MAUT) could be applied in a meaningful way. Roy and Bouyssou (1986) have shown that it would have been possible to avoid this recoding if ELECTRE III had been used, taking into account imperfect knowledge through discrimination thresholds without invoking distributions of probability. In cases considered within the contexts of Management of highway assets (7), and Clinical decision support in emergency room (9), the analyst came to conclusion that it is not possible to make a meaningful recoding of original performances into a common scale, which would permit to use, e.g., the Sugeno integral (Grabisch and Labreuche 2005), and thus she selected the DRSA method based on rough sets (Augeri et al. 2011; Wilk et al. 2005; Słowiński et al. 2009). This method is using a preference model in the form of a set of logical “if…, then…” decision rules that express conditions on particular criteria in terms of their original scales (see question 1d).
Question 1b

Is it simple or hard (even impossible) to get preference information that the method requires?

In order to make the results provided by a short-listed method pertinent for decision aiding, it is necessary to adjust some of its characteristics so as to take into account in the best possible way the preferences of the decision maker; these are usually some wishes underlying the value system of the decision maker. The analyst has thus to acquire what is called preference information. This information takes different forms for various methods: ordering of criteria, ordering of some actions, acceptable trade-offs, pairwise comparisons of some actions, assignment of some actions to categories, comparisons of some actions with respect to intensity of preference, assessment of lotteries, specification of the relative importance of criteria, presence of veto, etc. To acquire this information, the analyst has to interact with the decision maker (or with his representative), in view of co-constructing the model of preferences that the considered method exploits to work out expected results. A key issue is to organize this interaction such that the analyst is able to elaborate meaningful results. This implies that the interaction protocol or the software tool involved should be compatible with the way in which the analyst has been inserted in the decision process, with the way of reasoning of the inquired people, and with their meaning of useful results. This protocol or software tool has to ensure, moreover, an intelligibility and a traceability of the impact of the preference information on the results. If such an interaction appears to be impossible, the considered method has to be rejected. In all cases the analyst has to assess the part of arbitrariness that the acquired information may contain. She has to remember that it will be necessary to analyze its impact on the results provided by the method (sensitivity analysis, robustness concern).

In the case study Line extension of the Paris metro (4), the impossibility of acquiring preference information about relative importance of criteria led to elaboration of a new method: ELECTRE IV (Figueira et al. 2005). In cases considered within the contexts of Management of highway assets (7), and Clinical decision support in emergency room (9), direct elicitation of preference information about values of parameters of a preference model (in particular, the relative importance weights of criteria) also appeared impossible, which put forward the DRSA method based on rough sets. DRSA accepts sorting examples as input preference information (Greco et al. 2005). Remark that sorting examples, i.e. assignments of some well-known actions to decision categories, does not need as much cognitive effort from the decision maker as direct specification of preference model parameters required by many other methods.

On the other hand, in the case study Engineering design of a chemical reactor (12), it was natural to select a method taking into account the preference information expressed in terms of aspiration levels on the four objectives related to: reactor cost, production cost, volume of production, and efficiency. These aspiration levels define a reference point in the objective space, which can be projected onto the non-dominated set, indicating a candidate for the compromise action, together with its neighborhood that can then be explored by the decision maker (Jaszkiewicz and Słowiński 1999).
Question 1c

Should the part of imprecision, uncertainty or indetermination in the definition of performances be taken into account, and if so, in what way?

It is rare that the performances of the considered actions can be evaluated on each criterion without any ambiguity. The way in which a criterion is modeling preferences related to a specific point of view, or the role that an attribute is supposed to play, can be ill-determined or contain a part of arbitrariness. Some data used to construct criteria can be imprecise or defined in an ambiguous way. When it is important to take into account such an imperfect knowledge, the analyst should examine carefully the possibility of handling this knowledge by the short-listed methods.

We note that, when performance data are ambiguous, this can cause inconsistency in the indirect preference information given by the decision maker in the form of decision examples (sorting examples or pairwise comparisons). In case of inconsistency, the rough set concept is useful for discerning certain from possible knowledge in reasoning about ordinal data (Słowiński et al. 2012).

Let us observe, moreover, that, independently of the method, it is always possible to handle the imperfect knowledge:
  • Either by sensitivity or robustness analysis of results provided by the method. This approach becomes quickly too onerous when the sources of imperfect knowledge affect more than one or two criteria, because this sensitivity or robustness analysis has often to be combined with a similar analysis taking into account imperfect preference information (partial, inconsistent, vague, etc.; see question 1b).

  • Or, in an indirect way, by modeling the imperfect knowledge using probability or possibility distribution of actions’ performances on considered criteria. In probabilistic case, the action’s performance on a criterion is set to an expected value of probabilistic distribution. In possibilistic case, it is set to a “mean value” of fuzzy number representing possibilistic distribution (Dubois and Prade 1987). This approach admits, however, that it is possible to model the indetermination in probabilistic or possibilistic way on the base of a relatively objective assessment. A frequent use of the Gaussian distribution in the probabilistic approach, and of the triangular fuzzy numbers in the possibilistic approach, is often missing a sound foundation, given that the performances can vary in a rather narrow interval and that extending it for very low probabilities or possibilities is rather arbitrary. To validate a probabilistic approach, the analyst can sometimes be tempted to ask the decision maker (or his representative) to compare some lotteries in view of revealing his perception of preference or indifference. Proceeding in this way, the analyst is led to translate the indetermination of performances into the terms of preference information (see point 1b), which involves an attitude toward risk. These questions can, however, confuse the decision maker.

The methods that take into account pseudo-criteria are able to deal with imperfect knowledge about performances through the use of discrimination thresholds (Roy and Vincke 1987): indifference and preference thresholds. In order to assign values to these thresholds, the analyst has to investigate what sources of imprecision, uncertainty and indetermination affect the performance of an action on the considered criterion. These sources can originate from imperfect representation of the specific viewpoint by this criterion as well as from imperfect knowledge of data to be used for definition of the performance. Being conscious of these sources, the analyst should be able to assess the smallest difference between two performances which, when growing, becomes significant for preference of the action with the better evaluation over the action with the worse evaluation (this is the preference threshold), as well as the greatest difference between two performances which, when decreasing, becomes insignificant and leads to indifference of the two actions (this is the indifference threshold). These two values are not necessarily equal. The analyst must be convinced that the values assigned to these thresholds are appropriate for a right handling of the imperfect knowledge, or using other words, for giving to the comparison of two performances the meaning it deserves (for more details, see Roy and Figueira 2013).

Among the methods short-listed by the analyst, there can be methods which have been developed for the handling of a particular form of imperfect knowledge. Taking into account that reasoning based on imperfect knowledge can lead to uncertain conclusions, three kinds of uncertainty can be distinguished (Zadeh 1999): (1) uncertainty following from a random change of some variables, called veristic, which can be modeled by probability, (2) uncertainty following from subjective judgments, called possibilistic, which can be modeled by fuzzy sets, and (3) uncertainty caused by granularity of information, called inconsistency, which can be modeled by rough sets. The theories standing behind these concepts of uncertainty are: (1) probability theory, (2) possibility theory (Dubois and Prade 1988) and fuzzy set theory (Słowiński 1998), and (3) rough set theory (Pawlak 1991; Greco et al. 2001). Sometimes the uncertainty is more complex and needs hybridization of the above theories, e.g., probabilistic-fuzzy (van den Berg et al. 2004), fuzzy-rough (Greco et al. 2008b), or probabilistic-rough (Kotłowski et al. 2008). Some comparative studies have been carried out between stochastic and fuzzy approaches on the ground of multiobjective optimization (Słowiński and Teghem 1990).

If some methods accepting pseudo-criteria have been short-listed together with some methods specialized in handling a particular type of imperfect knowledge, the analyst must check which one of them will fit the context under study.

In the case studies Supplier selection (5), Programming of water supply systems for rural areas (8), Credit granting (10), Monitoring of risk zones (11), and Engineering design of a chemical reactor (12), the handling of imprecision, uncertainty and indetermination in the definition of performances of actions on some criteria was crucial. The methods applied in these studies dealt with imperfect knowledge about performances through application of pseudo-criteria. In the development planning of a jointly operated urban water supply and wastewater treatment system, considered in context (8), imperfect knowledge about cost and reliability coefficients, water pollution indices, discount factors and the user’s demands was modeled by fuzzy numbers; in consequence, the problem has been formulated as a multiobjective fuzzy LP problem. Finally, in the decision contexts Management of highway assets (7) and Clinical decision support in emergency room (9), the crucial problem related to imperfect knowledge was the inconsistency in the indirect preference information given in the form of sorting examples; for this reason the rough set approach (DRSA) has been used in these studies.
Question 1d

Is the compensation of bad performances on some criteria by good ones on other criteria acceptable?

In the context of multicriteria decision aiding methods, compensation means that the method offers possibilities of the following type. Let a be an action strictly preferred to another action b, both having the same performances on all but one criterion i on which b is significantly worse. One says that the method offers a possibility of compensation if improving one or more performances of b on other criteria than i it is possible to define an action c indifferent to a. These improvements compensate the bad performance of b compared to a on the i-th criterion. Such possibilities of compensation are offered in many ways by additive methods. In a lexicographic method, they exist only if the i-th criterion is not the most important. In ELECTRE type methods, they exist under extremely restrictive conditions.

On one hand, the methods that rely on aggregation of criteria into a synthetic criterion assigning a numerical value (utility, score) to each potential action use extensively and systematically this kind of compensation. On the other hand, methods that rely on multicriteria aggregation involving the concepts of concordance and discordance in view of elaborating outranking relations allow a very limited compensation in particular conditions. Moreover, methods that rely on the concept of rough set represent preferences in terms of “if…, then…” decision rules which do not admit any compensation. In the condition part of these rules there is a conjunction of elementary conditions concerning either performances of a single action on a subset of criteria (in case of sorting) or difference of performances for pairs of actions on a subset of criteria (in case of choice or ranking); for example, denoting the set of identifiers of criteria by I, criteria (of the gain type) by g i , g j , and actions by a, b, the rules have the following syntax (Greco et al. 2005):
  • in case of sorting:

    “if g i1(a) ≥ r i1 and g i2(a) ≥ r i2 and… g ik (a) ≥ r ik , then a is assigned to category t or better”,

    “if g j1(a) ≤ r j1 and g j2(a) ≤ r j2 and… g jh (a) ≤ r jh , then a is assigned to category t or worse”,

    where r i1r i2,…, r ik and r j1r j2,…, r jh are some threshold performances on criteria i1,…, ikj1,…, jh ∈ I, found during induction of rules from preference information given in the form of sorting examples and structured using the rough set concept,

  • in case of choice and ranking:

    “if g i1(a) ≥ r i1 and g i1(b) ≤ s i2 and… g ik (a) ≥ r ik and g ik (b) ≤ s ik , then a is outranking b”,

    “if g j1(a) ≤ r j1 and g j1(b) ≥ s j2 and… g jh (a) ≤ r jh and g jh (b) ≥ s jh , then a is not outranking b”,

    where r i1r i2, …,r ik r j1r j2,…, r jh and s i1s i2, …, s ik s j1s j2,…, s jh are some threshold performances on criteria i1,…, ikj1, …, jh ∈ I, found during induction of rules from preference information given in the form of pairwise comparisons of some actions and structured using the rough set concept.

We note that each rule is a scenario of a causal relationship between performances on a subset of criteria and a comprehensive judgment. The rules are non-compensatory aggregators that do not convert ordinal scales of criteria into a richer (interval or ratio) scale.

There are also other methods using non-compensatory aggregation, such as the lexicographic method and the method using the Sugeno integral. A comparison of these methods at an axiomatic level has been done by Greco et al. (2003, 2004), and by Słowiński et al. (2002).

Among the twelve contexts described in “A number of actual decision contexts”, Siting of a nuclear power-plant (2), Programming of water supply systems for rural areas (8), but only in part (b) concerning the choice of the best variant of technical construction of the regional WSS, and Engineering design of a chemical reactor (12), are those in which the compensation of a very bad performance on one criterion by a series of good performances on other criteria seems conceivable. In all other contexts, the decision maker would rather be reluctant to accept such compensation, and thus the analyst would be obliged to eliminate compensatory methods.
Question 1e

Is it necessary to take into account some forms of interaction among criteria?

A great majority of methods available nowadays, do not account for any form of interaction. Let us remind that interaction is a complex concept (Roy 1996, chapter 10; Roy 2009). For this reason, in general, the analyst is interested in designing the family of criteria so that any interaction among these criteria is excluded. If such a design appears impossible, the analyst and the decision maker (or his representative) should examine together the forms of interactions that should be handled in course of a multicriteria aggregation. This examination can be done either a priori or a posteriori; in the former case, identification of the form of interaction to be handled by the preference model permits to select a method designed for this form of interaction; in the latter case, the examination may resort to analysis of compatibility of preference information, provided in the form of sorting examples or pairwise comparisons of some actions, with the preference model that accounts for interaction or not. Then, considering all short-listed methods, the analyst will hesitate between:
  • a special form of MAUT, involving either a decomposition and graphical representation of additive value functions under ceteris paribus assumption, like GAI-networks (Gonzales and Perny 2005) and UCP-networks (Boutilier et al. 2001), or on one hand, a multilinear value function (Keeney and Raiffa 1976), and on the other hand, Choquet or Sugeno integral (Grabisch and Labreuche 2005),

  • the robust ordinal regression methods with an additive utility function augmented by components accounting for positive and negative synergy of pairs of criteria (Greco et al. 2013b),

  • ELECTRE method designed to handle interactions (Figueira et al. 2009a), and

  • the methods using a set of “if…, then…” decision rules as a preference model, since this model based on a very simple syntax of rules is able to handle the most complex interactions (Greco et al. 2002b, 2004; Słowiński et al. 2002).

None of the cases considered in “A number of actual decision contexts” required a priori recognition of some form of interaction among criteria. However, in the case studies referring to contexts: Management of highway assets (7) and Clinical decision support in emergency room (9), the rough set approach has been used, which is able to handle interactions through decision rules, if such interactions would appear in the preference information.

Secondary questions

Before making the final choice, especially if hesitating between various methods, the analyst may consider the following secondary questions:
Question 2a

Is the method able to satisfy properly the needs of comprehension from the part of stakeholders involved in the decision process?

The needs of comprehension can come not only from the decision maker (or his representative) but also from other stakeholders. If they are not properly satisfied, they can compromise a good insertion of the analyst in the decision process. Thus, she must try to assess up to what level of detail she should explain the way of functioning of the tool, which is the method. Depending on the level at which the requests are placed, she must also assess if those making the requests are ready to devote enough time for listening to the explanation. In many cases, what matters is the possibility of explaining the link that exists between information and data provided from one side, and the results obtained by the method from the other side, without entering into details of the method. As an example, let us mention the ‘even swaps’ method which typically provides non-technical explanation of this link for an additive preference model (Hammond et al. 1998).

This question was considered when choosing the methods in the contexts: Responses to tenders (6), Credit granting (10) and Monitoring of risk zones (11). In these cases, the needs of comprehension of the selected method were satisfied by showing on some well-chosen practical examples the type of results these methods produce. An answer to this question also influenced the selection of the method using a preference model in the form of “if…, then…” decision rules in the contexts of Management of highway assets (7), and Clinical decision support in emergency room (9)—intelligibility and traceability of the feedback between the preference information and the recommendations given by rules were convincing arguments for choosing in these contexts the DRSA method based on rough sets (Słowiński et al. 2009).
Question 2b

Is an axiomatic characterization of the method available, and if so, is it acceptable in the considered decision context?

This characterization, if available, provides a set of axioms that justify the application of the method for every analyst and decision maker who find this set of axioms adequate to the considered decision context. A good example of the case where the preferences expressed by the decision maker are not compatible with a set of axioms underlying a multicriteria aggregation procedure is the example of mayor’s preferences presented by Vincke (1982). An axiomatic characterization can serve as a scientific backing to the theorist playing the role of the analyst but she should not overestimate its importance. The scientific guarantee brought by this characterization has to be put into perspective of the following reasoning:
  1. 1.

    Suppose first that the analyst accepts the hypothesis that the decision maker has in his mind a relatively well-defined system of preferences, and that the analyst wants to know if every axiom characterizing the short-listed method is satisfied by this pre-existing system of preferences. A classic way to answer this question is to involve the decision maker (in practice, the person being inquired is rarely the decision maker; usually, this is somebody who represents the decision maker—sometimes a member of the ‘task force’ team) in a series of choice situations, and to ask him for precise answers. The decision maker is not necessarily familiar with these situations. He may find them artificial and thus unrealistic. In consequence, the answers given by the decision maker should be interpreted cautiously. Nevertheless, if in a given situation an answer violated clearly the considered axiom, the analyst would be allowed to conclude that the process which determines the decision maker’s preferences with respect to this choice situation violates the axiom in question. If, however, such a violation would not happen for any presented choice situation, the analyst could not be certain if in other choice situations the axiom would not be violated. The analyst should also take into account the fact that the process which determines the answer of the decision maker in choice situation n can be influenced by the way in which he was led to think and answer in n − 1 previous situations. Interrogation protocols which do not escape the trap of this influence obscure the conditions in which the analyst can check if a set of axioms conforms with a system of preferences that pre-exists in the decision maker’s mind (Roy 2010).

  2. 2.

    Let us suppose now that the analyst is not interested in knowing if the decision maker has in his mind a system of preferences concordant with the considered set of axioms, but, more modestly, if he accepts each axiom as a working hypothesis. Such a question can be posed directly to the decision maker only if the axioms are sufficiently simple and well understandable by him. For any other axiom, the analyst will have to present as realistic situations as possible and check the decision maker’s reaction to these situations. They should be conceived in a way permitting to show the decision maker that accepting the axiom implies one type of answers, and rejecting it implies another type. Also here, the way of conceiving the situations and the underlying questions are not neutral. In consequence, the responses being obtained have to be seen as constructs rather than a reflection of an objective reality. It follows that the analyst is influencing more or less consciously the acceptance or rejection of an axiom as a working hypothesis.

  3. 3.

    Accepting one by one the axioms from a certain set does not ensure that the set would be accepted in total by the decision maker. Taking into account all axioms of a certain set jointly may show some emerging phenomena that the decision maker cannot see when analyzing sequentially these axioms. It would be thus wrong to suggest that accepting each axiom from a set would imply the validity of all results following from the totality of the set of axioms. The wrong conclusion follows from the confusion between two levels of logics: the one of parts, and the one of the whole. This probably explains why the answers obtained by Maurice Allais to the choice situations presented to the adherents of the utility theory (and, in particular, to many of its founders) appeared to be discordant with the results following from the set of axioms on which this theory is based (Allais 1953; Allais and Hagen 1979).


The above considerations should not lead to believe that the axiomatic work is not useful. On the contrary, it can provide a better understanding and insight to the analyst on how the method actually behaves and how it compares to other methods at the axiomatic level. Such an analysis permitted, for example, to compare the capacity of preference representation of a general utility function and three of its special cases: associative operator, Sugeno integral and ordered weighted maximum on one hand, and a set of rough set decision rules on the other hand (Greco et al. 2004). The formal proof that the decision-rule aggregation (preference) model is the most general among the known aggregation functions is a useful conclusion of this study.

As to the influence of this question on the choice of a multicriteria decision aiding method in the twelve contexts described in “A number of actual decision contexts”, we can say that this influence was not observed. Perhaps with the exception of the way the case of Siting of a nuclear power-plant (2) was treated by Keeney and Nair (1976) and Keeney and Robillard (1977).
Question 2c

Can the weak points of the method affect the final choice?

All methods of decision aiding have some weak points. The analyst should know them if she is familiar with the method. She has to examine the possible impact of these weak points in the considered decision context.

An example of the weak point of many methods that involve pairwise comparisons of actions to build a preference model is known under the name of “rank reversal”. This weakness can compromise a method that one would like to use in a decision context where the set of actions may be modified incrementally; this may be the case of ranking an unstable set of actions. Another weak point could be the calculation time if it would be incompatible with the conditions of using the method in the considered decision context; for example, too long calculations between dialogue phases in a multiobjective optimization method.

The weak points of some popular methods have been clearly shown and discussed in various publications, e.g.: for ELECTRE methods in Figueira et al. (2013), for AHP in Bana e Costa and Vansnick (2008) and in Bouyssou et al. (2000, point 6.3.2), for TOPSIS in Martel and Roy (2006), for MAUT in McCord and de Neufville (1983), for methods based on Choquet integral in Roy (2009) and in Greco et al. (2013b), and for methods based on Sugeno integral in Słowiński et al. (2002).

In all the considered decision contexts, the week points of the selected methods have been examined to feel reassured about the final choice.


In this paper, we have reviewed the questions which, in our view, should be answered by an analyst before choosing the method to be used in a decision context. We have proposed a hierarchy of questions which, although quite general, may appear incomplete or inadequate in some specific cases.

After answering all these questions, the analyst can face two difficulties:
  1. (a)

    Her answers to the questions lead her to select a method which she does not control well enough or for which there is no software implementation available. If the analyst does not have enough time to learn the method, or if she is not able to make or order its software implementation, then she will be obliged to disregard this method and reconsider her answers to these questions.

  2. (b)

    Taking into account her answers to the questions, there is no appropriate method. If the analyst can afford (competence, time and financing), she can try to design and implement an appropriate method. Otherwise, she must revisit her answers to these questions, and accept some less satisfactory answers with the aim of permitting to find a method.


The content of the questions, and the diversity of answers that can be given with respect to the decision context, lead us to the conclusion that it is not possible to conceive a family of criteria which would permit a multicriteria formulation of the problem of choosing a multicriteria decision aiding method. The few attempts known from the literature do not seem to be a success (Ozernoy 1988, 1992; Guitouni and Martel 1998). The literature confirms that we are not alone to claim that the choice of an appropriate method is one of the most difficult problems to which the analyst is confronted in multicriteria decision aiding (Belton and Stewart 2002).

The fact that in real world contexts the answers to the nine questions formulated above do not boil down to terms “yes” or “no”, as well as the fact that the short-listed methods are usually more or less adequate to the considered context, explain why so many methods have been developed, and why it is difficult to compare one method to another in an insightful way. Unfortunately, many researchers are tempted to compare different multicriteria decision aiding methods by basing their conclusions mainly on comparison of results obtained by these methods. The arguments enunciated in this article highlight the fact that such a comparison is ill-founded.


  1. 1.

    To avoid the complex expression ‘he/she’ when talking about the roles of the analyst and the decision maker, we are using the pronoun ‘he’ for the decision maker, and ‘she’ for the analyst.



The second author wishes to acknowledge financial support from the Polish National Science Centre, Grant No. N N519 441939.


  1. Allais M (1953) Le comportement de l’homme rationnel devant le risque-critique des postulats et axiomes de l’école américaine. Econometrica 21:503–546CrossRefGoogle Scholar
  2. Allais M, Hagen O (eds) (1979) Expected utility hypotheses and the Allais Paradox. Theory and Decision Library, Reidel, DordrechtGoogle Scholar
  3. Almeida-Dias J, Figueira JR, Roy B (2010) ELECTRE TRI-C: a multiple criteria sorting method based on characteristic reference actions. Eur J Oper Res 204:565–580CrossRefGoogle Scholar
  4. Almeida-Dias J, Figueira JR, Roy B (2012) A multiple criteria sorting method where each category is characterized by several reference actions: the ELECTRE TRI-NC method. Eur J Oper Res 217:567–579CrossRefGoogle Scholar
  5. Augeri MG, Colombrita R, Greco S, Lo Certo A, Matarazzo B, Słowiński R (2011) Dominance-based rough set approach to budget allocation in highway maintenance activities. J Infrastruct Syst 17:75–85CrossRefGoogle Scholar
  6. Bana e Costa CA (1996) Les problématiques de l’aide à la décision : vers l’enrichissement de la trilogie choix-tri-rangement. RAIRO/Recherche Opérationnelle 30:191–216Google Scholar
  7. Bana e Costa CA, Vansnick J-C (2008) A critical analysis of the eigenvalue method used to derive priorities in AHP. Eur J Oper Res 187:1422–1428CrossRefGoogle Scholar
  8. Bana e Costa CA, De Corte J-M, Vansnick J-C (2005) On the mathematical foundations of MACBETH. In: Figueira JR, Greco S, Ehrgott M (eds) Multiple criteria decision analysis: state of the art surveys, Chap 10. Springer, New York, pp 409–442Google Scholar
  9. Belacel N (2000) Multicriteria assignment method PROAFTN: methodology and medical applications. Eur J Oper Res 125:175–183CrossRefGoogle Scholar
  10. Belton V, Stewart TJ (2002) Multiple criteria decision analysis: an integrated approach. Kluwer, BostonCrossRefGoogle Scholar
  11. Bisdorff R, Meyer P, Roubens M (2008) Rubis: a bipolar-valued outranking method for the best choice decision problem. 4OR 6:143–165CrossRefGoogle Scholar
  12. Błaszczyński J, Greco S, Słowiński R (2007) Multi-criteria classification: a new scheme for application of dominance-based decision rules. Eur J Oper Res 181:1030–1044CrossRefGoogle Scholar
  13. Bollinger D, Maystre LY, Słowiński R (1997) Un cas concret de la contribution de l’aide multicritère à la décision à la coordination intercantonale pour l’incinération des déchets urbains. In: Proceedings of the 45th meeting of the European working group on multicriteria aid for Decisions, Čelákovice, Czech Republic, pp 5–70Google Scholar
  14. Boutilier C, Bacchus F, Brafman RI (2001) UCP-networks: a directed graphical representation of conditional utilities. In: Proceedings of the 17th annual conference on uncertainty in artificial intelligence, pp 56–64Google Scholar
  15. Bouyssou D, Marchant T, Pirlot M, Perny P, Tsoukiàs A, Vincke Ph (2000) Evaluation and decision models. A critical perspective. Kluwer, BostonGoogle Scholar
  16. Branke J, Deb K, Miettinen K, Słowiński R (eds) (2008) Multiobjective optimization: interactive and evolutionary approaches. State-of-the-art survey series, LNCS, vol 5252. Springer, BerlinGoogle Scholar
  17. Branke J, Greco S, Słowiński R, Zielniewicz P (2010) Interactive evolutionary multiobjective optimization driven by robust ordinal regression. Bull Pol Acad Sci Techn Sci 58:347–358Google Scholar
  18. Brans JP, Mareschal B (2005) PROMETHEE methods. In: Figueira JR, Greco S, Ehrgott M (eds) Multiple criteria decision analysis: state of the art surveys, Chap 5. Springer, New York, pp 163–195Google Scholar
  19. Choo EU, Atkins DR (1980) An interactive algorithm for multicriteria programming. Comput Oper Res 7:81–87CrossRefGoogle Scholar
  20. Deb K (2008) Introduction to evolutionary multiobjective optimization. In: Branke J, Deb K, Miettinen K, Słowiński R (eds) Multiobjective optimization: interactive and evolutionary approaches, Chap 3. State-of-the-art survey series, LNCS, vol 5252. Springer, Berlin, pp 59–96Google Scholar
  21. Dembczyński K, Greco S, Słowiński R (2009) Rough set approach to multiple criteria classification with imprecise evaluations and assignments. Eur J Oper Res 198:626–636CrossRefGoogle Scholar
  22. Dembczyński K, Kotłowski W, Słowiński R, Szeląg M (2010) Learning of rule ensembles for multiple attribute ranking problems. In: Fürnkranz J, Hüllermeier E (eds) Preference learning. Springer, Berlin, pp 217–247CrossRefGoogle Scholar
  23. Devaud JM, Groussaud G, Jacquet-Lagrèze E (1980) UTADIS: Une méthode de construction de fonctions d’utilité additives rendant compte de jugements globaux. In: Proceedings of the European working group on MCDA. Bochum, GermanyGoogle Scholar
  24. Doumpos M, Zopounidis C (2004) A multicriteria classification approach based on pairwise comparisons. Eur J Oper Res 158:378–389CrossRefGoogle Scholar
  25. Dubois D, Prade H (1987) The mean value of a fuzzy number. Fuzzy Sets Syst 24:279–300CrossRefGoogle Scholar
  26. Dubois D, Prade H (1988) Possibility theory. Plenum Press, New YorkCrossRefGoogle Scholar
  27. Dyer J (2005) MAUT-multiattribute utility theory. In: Figueira JR, Greco S, Ehrgott M (eds) Multiple criteria decision analysis: state of the art surveys, Chapt 7. Springer, New York, pp 265–292Google Scholar
  28. Fernandez ER, Navarro J (2011) A new approach to multi-criteria sorting based on fuzzy outranking relations: the THESEUS method. Eur J Oper Res 213:405–413CrossRefGoogle Scholar
  29. Figueira JR, Mousseau V, Roy B (2005) ELECTRE methods. In: Figueira JR, Greco S, Ehrgott M (eds) Multiple criteria decision analysis: state of the art surveys, Chap 4. Springer, New York, pp 133–162Google Scholar
  30. Figueira JR, Greco S, Roy B (2009a) ELECTRE methods with interaction between criteria: an extension of the concordance index. Eur J Oper Res 199:496–505CrossRefGoogle Scholar
  31. Figueira JR, Greco S, Słowiński R (2009b) Building a set of additive value functions representing a reference preorder and intensities of preference: GRIP method. Eur J Oper Res 195:460–486CrossRefGoogle Scholar
  32. Figueira JR, Greco S, Roy B, Słowiński R (2013) An overview of ELECTRE methods and their recent extensions. J Multi-Criteria Decis Anal 20:61–85Google Scholar
  33. Gonzales C, Perny P (2005) GAI networks for decision making under certainty. In: Brafman R, Junker U (eds) Proceedings of the 19th international joint conference on artificial intelligence—workshop on advances in preference handling, pp 100–105Google Scholar
  34. Grabisch M, Labreuche C (2005) Fuzzy measures and integrals in MCDA. In: Figueira JR, Greco S, Ehrgott M (eds) Multiple criteria decision analysis: state of the art surveys, Chap 14. Springer, New York, pp 563–608Google Scholar
  35. Greco S, Matarazzo B, Słowiński R (2001) Rough sets theory for multicriteria decision analysis. Eur J Oper Res 129:1–47CrossRefGoogle Scholar
  36. Greco S, Matarazzo B, Słowiński R (2002a) Rough sets methodology for sorting problems in presence of multiple attributes and criteria. Eur J Oper Res 138:247–259CrossRefGoogle Scholar
  37. Greco S, Matarazzo B, Słowiński R (2002b) Preference representation by means of conjoint measurement and decision rule model. In: Bouyssou D, Jacquet-Lagrèze E, Perny P, Słowiński R, Vanderpooten D, Vincke Ph (eds) Aiding decisions with multiple criteria-essays in honor of Bernard Roy. Kluwer, Boston, pp 263–313CrossRefGoogle Scholar
  38. Greco S, Matarazzo B, Słowiński R (2003) The axiomatic basis of multicriteria non-compensatory preferences. In: Proceedings of the 4th international workshop on preferences and decisions. Trento, pp 81–86Google Scholar
  39. Greco S, Matarazzo B, Słowiński R (2004) Axiomatic characterization of a general utility function and its particular cases in terms of conjoint measurement and rough-set decision rules. Eur J Oper Res 158:271–292CrossRefGoogle Scholar
  40. Greco S, Matarazzo B, Słowiński R (2005) Decision rule approach. In: Figueira JR, Greco S, Ehrgott M (eds) Multiple criteria decision analysis: state of the art surveys, Chap 13. Springer, New York, pp 507–562Google Scholar
  41. Greco S, Matarazzo B, Słowiński R (2008a) Dominance-based rough set approach to interactive multiobjective optimization. In: Branke J, Deb K, Miettinen K, Słowiński R (eds) Multiobjective optimization: interactive and evolutionary approaches, Chap 5. State-of-the-art surveys, LNCS, vol 5252. Springer, Berlin, pp 121–155Google Scholar
  42. Greco S, Matarazzo B, Słowiński R (2008b) Fuzzy set extensions of the dominance-based rough set approach. In: Bustince H, Herrera F, Montero J (eds) Fuzzy sets and their extensions: representation, aggregation and model, Chap III. 3. Studies in fuzziness and soft computing, vol 220. Springer, Berlin, pp 239–261CrossRefGoogle Scholar
  43. Greco S, Mousseau V, Słowiński R (2008c) Ordinal regression revisited: multiple criteria ranking using a set of additive value functions. Eur J Oper Res 191:416–436CrossRefGoogle Scholar
  44. Greco S, Mousseau V, Słowiński R (2010a) Multiple criteria sorting with a set of additive value functions. Eur J Oper Res 207:1455–1470CrossRefGoogle Scholar
  45. Greco S, Słowiński R, Figueira J, Mousseau V (2010b) Robust ordinal regression. In: Ehrgott M, Figueira J, Greco S (eds) Trends in multiple criteria decision analysis, Chap 9. Springer, New York, pp 241–283CrossRefGoogle Scholar
  46. Greco S, Kadziński M, Mousseau V, Słowiński R (2011) ELECTREGKMS: robust ordinal regression for outranking methods. Eur J Oper Res 214:118–135CrossRefGoogle Scholar
  47. Greco S, Kadziński M, Słowiński R (2012) Extreme ranking analysis in robust ordinal regression. OMEGA 40:488–501CrossRefGoogle Scholar
  48. Greco S, Matarazzo B, Słowiński R (2013a) Beyond Markowitz with multiple criteria decision aiding. J Bus Econ 83:29–60Google Scholar
  49. Greco S, Mousseau V, Słowiński R (2013b) UTAGMS-INT: robust ordinal regression of value functions handling interacting criteria. Eur J Oper Res (to appear)Google Scholar
  50. Guitouni A, Martel J-M (1998) Tentative guidelines to help choosing an appropriate MCDA method. Eur J Oper Res 109:501–521CrossRefGoogle Scholar
  51. Hammond JS, Keeney RL, Raiffa H (1998) Even swaps: a rational method for making trade-offs. Harv Bus Rev 76:137–149Google Scholar
  52. Hwang CL, Yoon KS (1981) Multiple attribute decision making: methods and applications. Springer, BerlinCrossRefGoogle Scholar
  53. Jacquet-Lagrèze E, Siskos Y (1982) Assessing a set of additive utility functions for multicriteria decision making: the UTA method. Eur J Oper Res 10:151–164CrossRefGoogle Scholar
  54. Jaszkiewicz A, Ferhat AB (1999) Solving multiple criteria choice problems by interactive trichotomy segmentation. Eur J Oper Res 113:271–280CrossRefGoogle Scholar
  55. Jaszkiewicz A, Słowiński R (1999) The ‘Light Beam Search’ approach—an overview of methodology and applications. Eur J Oper Res 113:300–314CrossRefGoogle Scholar
  56. Jaszkiewicz A, Słowiński R, Gawałek A (1995) The ‘Light Beam Search’ over a non-dominated set and its application in Polish chemical industry. Cent Eur J Oper Res Econ 3:157–169Google Scholar
  57. Kadziński M, Słowiński R (2012) Interactive robust cone contraction method for multiple objective optimization problems. Int J Inf Technol Decis Mak 11:327–357CrossRefGoogle Scholar
  58. Kadziński M, Greco S, Słowiński R (2012) Selection of a representative value function in robust multiple criteria ranking and choice. Eur J Oper Res 217:541–553CrossRefGoogle Scholar
  59. Kadziński M, Greco S, Słowiński R (2013) RUTA: a framework for assessing and selecting additive value functions on the basis of rank related requirements. OMEGA 41:735–751CrossRefGoogle Scholar
  60. Keeney RL, Nair K (1976) Evaluating potential nuclear power plant sites in the Pacific Northwest using decision analysis, IIASA Professional Paper no. 76-1; also in: Bell DE, Keeney RL, Raiffa H (eds) (1977), Conflicting Objectives in Decisions, Chapter 14, Wiley, and in: Keeney RL (1980), Siting Energy Facilities, Chapter 3, Academic Press, New YorkGoogle Scholar
  61. Keeney RL, Raiffa H (1976) Decisions with multiple objectives: preferences and value tradeoffs. Wiley, New YorkGoogle Scholar
  62. Keeney RL, Robillard GA (1977) Assessing and evaluating environmental impacts at proposed nuclear power plant sites. J Environ Econ Manag 4:153–166CrossRefGoogle Scholar
  63. Korhonen P, Wallenius J (1988) Pareto race. Nav Res Log 35:615–623CrossRefGoogle Scholar
  64. Kotłowski W, Dembczyński K, Greco S, Słowiński R (2008) Stochastic dominance-based rough set model for ordinal classification. Inf Sci 178:4019–4037CrossRefGoogle Scholar
  65. Labbouz S, Roy B, Diab Y, Christen M (2008) Implementing a public transport line. Multi-criteria decision-making method that facilitate concertation. Oper Res Int J 8:5–31CrossRefGoogle Scholar
  66. Léger J, Martel J-M (2002) A multicriteria assignment procedure for a nominal sorting problematic. Eur J Oper Res 138:349–364CrossRefGoogle Scholar
  67. Liesiö J, Mild P, Salo A (2008) Robust portfolio modeling with incomplete cost information and project interdependencies. Eur J Oper Res 190:679–695CrossRefGoogle Scholar
  68. Martel J-M, Roy B (2006) Analyse de la signifiance de diverses procédures d’agrégation multicritère. INFOR 44:191–214Google Scholar
  69. Maystre LY, Bollinger D (1999) Aide à la négotiation multicritère: pratique et conseils, Chapitre 4, Niveau de centralisation de la gestion des déchets en Suisse Occidentale. Presses polytechniques et universitaires romandes, LausanneGoogle Scholar
  70. McCord M, de Neufville R (1983) Fundamental deficiency of expected utility decision analysis. In: French S, Hartley R, Thomas LC, White DJ (eds) Multi-objective decision making. Academic Press, New York, pp 279–305Google Scholar
  71. Merad M, Verdel T, Kouniali S, Roy B (2004) Use of multi-criteria-aids for risk zoning and management of a large area subjected to mining induced hazards. Tunn Undergr Space Technol 19:125–138CrossRefGoogle Scholar
  72. Metaxiotis K, Liagkouras K (2012) Multiobjective evolutionary algorithms for portfolio management: a comprehensive literature review. Expert Syst Appl 39:11685–11698CrossRefGoogle Scholar
  73. Michalowski W, Rubin S, Słowiński R, Wilk Sz (2003) Mobile clinical support system for pediatric emergencies. Decis Support Syst 36:161–176CrossRefGoogle Scholar
  74. Miettinen K (1999) Nonlinear multiobjective optimization. Kluwer, BostonGoogle Scholar
  75. Mousseau V, Dias LC, Figueira J (2003) On the notion of category size in multiple criteria sorting models. Cahier du LAMSADE 205, Université Paris-Dauphine, ParisGoogle Scholar
  76. Ozernoy VM (1988) Some issues in designing an expert system for multiple criteria decision making. Acta Psychol 68:237–253CrossRefGoogle Scholar
  77. Ozernoy VM (1992) Choosing the ‘best’ multiple criteria decision-making method. INFOR 30:159–171Google Scholar
  78. Pawlak Z (1991) Rough sets: theoretical aspects of reasoning about data. Kluwer, DordrechtGoogle Scholar
  79. Perny P (1998) Multicriteria filtering methods based on concordance/non-discordance principles. Ann Oper Res 80:137–167CrossRefGoogle Scholar
  80. Roy B (1996) Multicriteria methodology for decision aiding. Kluwer, DordrechtCrossRefGoogle Scholar
  81. Roy B (1999) Decision-aiding today: what should we expect? In: Gal T, Stewart TJ, Hanne T (eds) Multicriteria decision making-advances in MCDM models, algorithms, theory and applications, Chap 1. Kluwer, Dordrecht, pp 1-1–1-35Google Scholar
  82. Roy B (2009) A propos de la signification des dépendances entre critères: quelle place et quells modes de prise en compte pour l’aide à la décision? RAIRO Recherche Opérationnelle 43:255–275CrossRefGoogle Scholar
  83. Roy B (2010) Two conceptions of decision aiding. Int J Multicrit Dec Mak 1:74–79CrossRefGoogle Scholar
  84. Roy B, Bouyssou D (1986) Comparison of two decision-aid models applied o a nuclear power plant siting example. Eur J Oper Res 25:200–215CrossRefGoogle Scholar
  85. Roy B, Bouyssou D (1993) Aide Multicritère à la Décision: Méthodes et Cas. Economica, ParisGoogle Scholar
  86. Roy B, Figueira J (2013) Discriminating thresholds as a tool to cope with imperfect knowledge in multicriteria decision aiding: theoretical results and practical issues. Cahier du LAMSADE no. 328, Université Paris DauphineGoogle Scholar
  87. Roy B, Hugonnard JC (1982) Ranking of suburban line extension project on the Paris metro system by a multicriteria method. Transp Res 16A:301–312Google Scholar
  88. Roy B, Vincke Ph (1987) Pseudo-orders: definition, properties and numerical representation. Math Soc Sci 14:263–274CrossRefGoogle Scholar
  89. Roy B, Słowiński R, Treichel W (1992) Multicriteria programming of water supply systems for countryside. Water Resour Bull 28:13–31CrossRefGoogle Scholar
  90. Saaty TL (2005) The analytic hierarchy and analytic network processes for the measurement of intangible criteria and for decision-making. In: Figueira JR, Greco S, Ehrgott M (eds) Multiple criteria decision analysis: state of the art surveys, Chap 9. Springer, New York, pp 345–408Google Scholar
  91. Słowiński R (1986) A multicriteria fuzzy linear programming method for water supply system development planning. Fuzzy Sets Syst 19:217–237CrossRefGoogle Scholar
  92. Słowiński R (ed) (1998) Fuzzy sets in decision analysis, operations research and statistics. The handbooks of fuzzy sets series. Kluwer, BostonGoogle Scholar
  93. Słowiński R, Teghem J (eds) (1990) Stochastic versus fuzzy approaches to multiobjective mathematical programming under uncertainty. Kluwer, DordrechtGoogle Scholar
  94. Słowiński R, Greco S, Matarazzo B (2002) Axiomatization of utility, outranking and decision-rule preference models for multiple-criteria classification problems under partial inconsistency with the dominance principle. Control Cybern 31:1005–1035Google Scholar
  95. Słowiński R, Greco S, Matarazzo B (2009) Rough sets in decision making. In: Meyers RA (ed) Encyclopedia of complexity and systems science. Springer, New York, pp 7753–7786Google Scholar
  96. Słowiński R, Greco S, Matarazzo B (2012) Rough set and rule-based multicriteria decision aiding. Pesquisa Operacional 32:213–269CrossRefGoogle Scholar
  97. Steuer RE (1986) Multiple criteria optimization: theory, computation and application. Wiley, New YorkGoogle Scholar
  98. Szeląg M, Greco S, Słowiński R (2013) Rule-based approach to multicriteria ranking. In: Doumpos M, Grigoroudis E (eds) Multicriteria decision aid and artificial intelligence: links, theory and applications, Chap 6. Wiley-Blackwell, London, pp 127–160CrossRefGoogle Scholar
  99. Tsoukiàs A (2007) On the concept of decision aiding process. Ann Oper Res 154:3–27CrossRefGoogle Scholar
  100. van den Berg J, Kaymak U, van den Bergh W-M (2004) Financial markets analysis by using a probabilistic fuzzy modelling approach. Int J Approx Reason 35:291–305CrossRefGoogle Scholar
  101. Vincke Ph (1982) Preference modelling: a survey and an experiment. In: Brans JP (ed) Operational Research ’81. North-Holland, Amsterdam, pp 341–353Google Scholar
  102. von Winterfeldt D, Edwards W (1986) Decision analysis and behavioral research. Cambridge University Press, CambridgeGoogle Scholar
  103. Wang YJ, Lee HS, Lin K (2003) Fuzzy TOPSIS for multi-criteria decision-making. Int Math J 3:367–379Google Scholar
  104. Wang YM, Luo Y, Hua ZS (2008) On the extent analysis method for fuzzy AHP and its applications. Eur J Oper Res 186:735–747CrossRefGoogle Scholar
  105. Wierzbicki AP (1986) On the completeness and constructiveness of parametric characterization to vector optimization problems. OR Spectr 8:73–87CrossRefGoogle Scholar
  106. Wierzbicki AP (1999) Reference point approaches. In: Gal T, Stewart TJ, Hanne T (eds) Multicriteria decision making-advances in MCDM models, algorithms, theory and applications, Chap 9. Kluwer, Dordrecht, pp 9-1–9-39Google Scholar
  107. Wilk Sz, Słowiński R, Michalowski W, Greco S (2005) Supporting triage of children with abdominal pain in the emergency room. Eur J Oper Res 160:696–709CrossRefGoogle Scholar
  108. Yu W (1992) Aide multicritère à la décision dans le cadre de la problématique du tri: concepts, méthodes et applications. PhD dissertation, Université Paris-DauphineGoogle Scholar
  109. Zadeh LA (1999) From computing with numbers to computing with words-from manipulation of measurements to manipulation of perception. IEEE Trans Circuits Syst I Fundam Theory Appl 45:105–119CrossRefGoogle Scholar
  110. Zopounidis C, Doumpos M (2000) PREFDIS: a multicriteria decision support system for sorting decision problems. Comput Oper Res 27:779–797CrossRefGoogle Scholar

Copyright information

© The Author(s) 2013

Open AccessThis article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited.

Authors and Affiliations

  1. 1.LAMSADE, University of Paris DauphineParisFrance
  2. 2.Institute of Computing SciencePoznań University of TechnologyPoznańPoland
  3. 3.Systems Research InstitutePolish Academy of SciencesWarsawPoland

Personalised recommendations