When a public administration wishes to implement policies, there is a previous need of comparing different options to assess their social attractiveness. A fair policy assessment process should consider the ethical obligation of taking a plurality of social values, perspectives and interests into account; there is no doubt that Impact Assessment (IA) is then multidimensional in nature. For example, the European Commission current practice on IA considers three main objectives i.e. efficiency, effectiveness (including proportionality) and coherence and it is based on the assessment of various broad impacts such as economic, environmental and social ones. In empirical assessment of public policies and publicly provided goods, Multiple Criteria Decision Analysis is an appropriate policy tool, since it allows taking into account a wide range of assessment criteria (e.g. environmental impact, income distribution, social inclusion, and so on) and not simply profit maximization, as a private economic agent might do. This paper deals with the importance of using appropriate multi-criteria mathematical aggregation rules to guarantee consistency and transparency of results. An illustrative example dealing with a recent EC IA on modernising VAT for cross-border B2C e-Commerce is presented too.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price includes VAT (USA)
Tax calculation will be finalised during checkout.
All data used are contained in the manuscript.
Arrow's axiom of "the independence of irrelevant alternatives" states that the choice made in a given set of alternatives A depends only on the ordering made with respect to the alternatives in that set. Alternatives outside A (irrelevant since the choice must be made within A) should not affect the choice inside A. Empirical experience does not generally support this axiom. The issue of the independence of irrelevant alternatives is particularly important and tricky when pair-wise comparisons are used. To clarify this point, the reader may imagine a football championship. To determine the winner all the teams have to compete pair-wise. Then we need to know the performance of each team with respect to all the others, e.g., how many times a given team won, lost or was even. By using this information, we can finally determine who won the championship. Now imagine that when the championship is about to end and the team X is going to win (e.g. Barcelona), a new team Y is created (e.g. in Madrid). Would it be acceptable to allow this new team Y to play directly with X? Would the supporters of team X accept that if Y wins, then Y will also win the championship? Of course not! This example seems to give a clear answer to our problem, but if instead of ranking football teams, our problem is to evaluate the performance of universities, the answer is not that straightforward anymore. Let us assume that a study is almost finalized, and university A is going to be top ranked; however, the study team discovers that an important university institutionZ was not present in the original data set. Now the question is: can we just compare A with Z or do we have to make all the pairwise comparisons again? Now the answer is less clear-cut. Moreover, it might happen that the ranking at time T (without Z) ranks university A better than B and that at time T + 1 (when Z is considered in the pair-wise comparisons) B is ranked better than A just because Z is taken into consideration! Can this result be acceptable? To answer this question in a definitive manner is very controversial. What we can say for sure is that if pair-wise comparisons are used, it has to be accepted the assumption that the irrelevant alternative Z (irrelevant for the evaluation between A and B) can indeed change the relative evaluation of A and B. This phenomenon is called “rank reversal”.
One should note that this qualitative approach is more a rule than an exception. Although the majority of European Commission IA studies is based on a broad MCE framework, only a few uses quantification (in the form of a mathematical aggregation rule) in ranking the considered options. In 17 EC IAs examined (SWD(2016) 392 final, SWD(2016) 152 final, SWD(2017) 31 final, SWD(2016) 211 final, SWD(2016) 434 final, SWD(2016) 410 final, SWD(2016) 418 final, SWD(2016) 173 final, SWD(2016) 303 final, SWD(2017) 26 final, SWD(2016) 315 final, SWD(2016) 193 final, SWD(2016) 468 final, SWD(2016) 17 final, SWD(2015) 135 final, SWD(2017) 202 final, SWD(2016) 379 final), only one (SWD(2016) 211 final) uses quantification in the form of a MCE mathematical aggregation rule. All the other ones are based on a qualitative analysis of the various impacts, like the one described here.
Arrow, K.J.: Social choice and individual values, 2d edn. Wiley, New york (1963)
Arrow, K.J., Raynaud, H.: Social choice and multicriterion decision making. Press, Cambridge, M.I.T (1986)
Azzini, I., Munda, G.: – A new approach for identifying the kemeny median ranking. Eur. J. Oper. Res. 281, 388–401 (2020)
Bäcklund, A.-K.: Impact assessment in the European commission—a system with multiple objectives. Environ. Sci. Policy 12, 1077–1087 (2009)
Bell Michelle, L., Hobbs, B.F., Ellis, H.: The use of multi-criteria decision-making methods in the integrated assessment of climate change: implications for IA practitioners. In Socio-Econom. Plann. Sci. 37(4), 289–316 (2003)
Bozeman, B., Pandey, S.K.: Public management decision making: effects of decision content. Public Adm. Rev. 64(5), 553–565 (2004)
Brans, J.P., Mareschal, B., Vincke, Ph.: How to select and how to rank projects. the PROMETHEE method. Eur. J. Oper. Res. 24, 228–238 (1986)
Figueira, J., Greco, S. and Ehrgott, M. (eds.) (2016) Multiple-criteria decision analysis. State of the art surveys. Springer International Series in Operations Research and Management Science, New york.
Hinloopen, E., Nijkamp, P.: Qualitative multiple criteria choice analysis, the dominant regime method. Qual. Quant. 24, 37–56 (1990)
Ishizaka, I., Nemery, P.: Multi-criteria decision analysis: methods and software. John Wiley & Sons (2013)
Kemeny, J.: Mathematics without numbers. Daedalus 88, 571–591 (1959)
Moulin, H.: Axioms of co-operative decision-making. Econometric Society Monographs, Cambridge University Press, Cambridge (1988)
Munda, G.: Multicriteria evaluation in a fuzzy environment. Physica-Verlag, Contributions to Economics Series, Heidelberg (1995)
Munda, G.: “Social multi-criteria evaluation (SMCE)”: methodological foundations and operational consequences. Eur. J. Oper. Res. 158(3), 662–677 (2004)
Munda, G.: Social multi-criteria evaluation for a sustainable economy. Springer, Heidelberg, New york (2008)
Munda, G., Nardo, M.: Non-compensatory/non-linear composite indicators for ranking countries: a defensible setting. Appl. Econ. 41, 1513–1523 (2009)
Munda, G.: Intensity of preference and related uncertainty in non-compensatory aggregation rules. Theor. Decis. 73(4), 649–669 (2012)
Munda, G.: Beyond welfare economics: some methodological issues. J. Econom. Methodol. 23(2), 185–202 (2016)
Nijkamp, P., Rietveld, P., Voogd, H.: Multicriteria Evaluation in Physical Planning. North-Holland, Amsterdam (1990)
Podinovskii, V.V.: Criteria importance theory. Math. Soc. Sci. 27, 237–252 (1994)
Roy, B.: Multicriteria methodology for decision analysis. Kluwer, Dordrecht (1996)
Roberts, F.S.: Measurement theory with applications to decision making, utility and the social sciences. Addison-Wesley, London (1979)
Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., Tarantola, S.: Global sensitivity analysis. John Wiley & Sons, The Primer, England (2008)
Saltelli, A., Guimarães, P.A., van der Sluijs, J.P., Funtowicz, S.: What do I make of your latinorum? sensitivity auditing of mathematical modelling. Int. J. Foresight Innov. Policy 9(2–4), 213–234 (2013)
Sen, A.: The Idea of justice. The Belknap Press of Harvard University Press, Cambridge, Massachusetts (2009)
Sieber, S., Domínguez, I.P.: Impact assessment of agri-food policies in Europe: methods, tools and applications. J. Policy Model. 33(1), 1–6 (2011)
Torriti, J.: Impact assessment and the liberalization of the EU energy markets: evidence-based policy-making or policy-based evidence-making? J. Common Market Stud. 48, 4 (2010)
Vansnick J. C. (1990) Measurement theory and decision aid- in Bana e Costa C.A. (ed.)- Readings in multiple criteria decision aid, Springer-Verlag, Berlin, pp. 81–100.
Vincke, Ph.: Multicriteria decision aid. Wiley, New york (1992)
This article has been developed in the context of the institutional activities of the Competence Centre on Modelling at the Joint Research Centre. The views expressed are purely those of the writer and may not in any circumstance be regarded as stating an official position of the European Commission.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Munda, G. Qualitative reasoning or quantitative aggregation rules for impact assessment of policy options? A multiple criteria framework. Qual Quant (2021). https://doi.org/10.1007/s11135-021-01267-8
- Public policy
- Multiple Criteria Decision Analysis
- Robustness analysis
- Ex-ante impact assessment
- Qualitative and quantitative information