Skip to main content

Qualitative reasoning or quantitative aggregation rules for impact assessment of policy options? A multiple criteria framework


When a public administration wishes to implement policies, there is a previous need of comparing different options to assess their social attractiveness. A fair policy assessment process should consider the ethical obligation of taking a plurality of social values, perspectives and interests into account; there is no doubt that Impact Assessment (IA) is then multidimensional in nature. For example, the European Commission current practice on IA considers three main objectives i.e. efficiency, effectiveness (including proportionality) and coherence and it is based on the assessment of various broad impacts such as economic, environmental and social ones. In empirical assessment of public policies and publicly provided goods, Multiple Criteria Decision Analysis is an appropriate policy tool, since it allows taking into account a wide range of assessment criteria (e.g. environmental impact, income distribution, social inclusion, and so on) and not simply profit maximization, as a private economic agent might do. This paper deals with the importance of using appropriate multi-criteria mathematical aggregation rules to guarantee consistency and transparency of results. An illustrative example dealing with a recent EC IA on modernising VAT for cross-border B2C e-Commerce is presented too.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Data availability

All data used are contained in the manuscript.


  1. 1.

    Arrow's axiom of "the independence of irrelevant alternatives" states that the choice made in a given set of alternatives A depends only on the ordering made with respect to the alternatives in that set. Alternatives outside A (irrelevant since the choice must be made within A) should not affect the choice inside A. Empirical experience does not generally support this axiom. The issue of the independence of irrelevant alternatives is particularly important and tricky when pair-wise comparisons are used. To clarify this point, the reader may imagine a football championship. To determine the winner all the teams have to compete pair-wise. Then we need to know the performance of each team with respect to all the others, e.g., how many times a given team won, lost or was even. By using this information, we can finally determine who won the championship. Now imagine that when the championship is about to end and the team X is going to win (e.g. Barcelona), a new team Y is created (e.g. in Madrid). Would it be acceptable to allow this new team Y to play directly with X? Would the supporters of team X accept that if Y wins, then Y will also win the championship? Of course not! This example seems to give a clear answer to our problem, but if instead of ranking football teams, our problem is to evaluate the performance of universities, the answer is not that straightforward anymore. Let us assume that a study is almost finalized, and university A is going to be top ranked; however, the study team discovers that an important university institutionZ was not present in the original data set. Now the question is: can we just compare A with Z or do we have to make all the pairwise comparisons again? Now the answer is less clear-cut. Moreover, it might happen that the ranking at time T (without Z) ranks university A better than B and that at time T + 1 (when Z is considered in the pair-wise comparisons) B is ranked better than A just because Z is taken into consideration! Can this result be acceptable? To answer this question in a definitive manner is very controversial. What we can say for sure is that if pair-wise comparisons are used, it has to be accepted the assumption that the irrelevant alternative Z (irrelevant for the evaluation between A and B) can indeed change the relative evaluation of A and B. This phenomenon is called “rank reversal”.

  2. 2.

    One should note that this qualitative approach is more a rule than an exception. Although the majority of European Commission IA studies is based on a broad MCE framework, only a few uses quantification (in the form of a mathematical aggregation rule) in ranking the considered options. In 17 EC IAs examined (SWD(2016) 392 final, SWD(2016) 152 final, SWD(2017) 31 final, SWD(2016) 211 final, SWD(2016) 434 final, SWD(2016) 410 final, SWD(2016) 418 final, SWD(2016) 173 final, SWD(2016) 303 final, SWD(2017) 26 final, SWD(2016) 315 final, SWD(2016) 193 final, SWD(2016) 468 final, SWD(2016) 17 final, SWD(2015) 135 final, SWD(2017) 202 final, SWD(2016) 379 final), only one (SWD(2016) 211 final) uses quantification in the form of a MCE mathematical aggregation rule. All the other ones are based on a qualitative analysis of the various impacts, like the one described here.


  1. Arrow, K.J.: Social choice and individual values, 2d edn. Wiley, New york (1963)

    Google Scholar 

  2. Arrow, K.J., Raynaud, H.: Social choice and multicriterion decision making. Press, Cambridge, M.I.T (1986)

    Google Scholar 

  3. Azzini, I., Munda, G.: – A new approach for identifying the kemeny median ranking. Eur. J. Oper. Res. 281, 388–401 (2020)

    Article  Google Scholar 

  4. Bäcklund, A.-K.: Impact assessment in the European commission—a system with multiple objectives. Environ. Sci. Policy 12, 1077–1087 (2009)

    Article  Google Scholar 

  5. Bell Michelle, L., Hobbs, B.F., Ellis, H.: The use of multi-criteria decision-making methods in the integrated assessment of climate change: implications for IA practitioners. In Socio-Econom. Plann. Sci. 37(4), 289–316 (2003)

    Article  Google Scholar 

  6. Bozeman, B., Pandey, S.K.: Public management decision making: effects of decision content. Public Adm. Rev. 64(5), 553–565 (2004)

    Article  Google Scholar 

  7. Brans, J.P., Mareschal, B., Vincke, Ph.: How to select and how to rank projects. the PROMETHEE method. Eur. J. Oper. Res. 24, 228–238 (1986)

    Article  Google Scholar 

  8. Figueira, J., Greco, S. and Ehrgott, M. (eds.) (2016) Multiple-criteria decision analysis. State of the art surveys. Springer International Series in Operations Research and Management Science, New york.

  9. Hinloopen, E., Nijkamp, P.: Qualitative multiple criteria choice analysis, the dominant regime method. Qual. Quant. 24, 37–56 (1990)

    Article  Google Scholar 

  10. Ishizaka, I., Nemery, P.: Multi-criteria decision analysis: methods and software. John Wiley & Sons (2013)

    Book  Google Scholar 

  11. Kemeny, J.: Mathematics without numbers. Daedalus 88, 571–591 (1959)

    Google Scholar 

  12. Moulin, H.: Axioms of co-operative decision-making. Econometric Society Monographs, Cambridge University Press, Cambridge (1988)

    Book  Google Scholar 

  13. Munda, G.: Multicriteria evaluation in a fuzzy environment. Physica-Verlag, Contributions to Economics Series, Heidelberg (1995)

    Book  Google Scholar 

  14. Munda, G.: “Social multi-criteria evaluation (SMCE)”: methodological foundations and operational consequences. Eur. J. Oper. Res. 158(3), 662–677 (2004)

    Article  Google Scholar 

  15. Munda, G.: Social multi-criteria evaluation for a sustainable economy. Springer, Heidelberg, New york (2008)

    Book  Google Scholar 

  16. Munda, G., Nardo, M.: Non-compensatory/non-linear composite indicators for ranking countries: a defensible setting. Appl. Econ. 41, 1513–1523 (2009)

    Article  Google Scholar 

  17. Munda, G.: Intensity of preference and related uncertainty in non-compensatory aggregation rules. Theor. Decis. 73(4), 649–669 (2012)

    Article  Google Scholar 

  18. Munda, G.: Beyond welfare economics: some methodological issues. J. Econom. Methodol. 23(2), 185–202 (2016)

    Article  Google Scholar 

  19. Nijkamp, P., Rietveld, P., Voogd, H.: Multicriteria Evaluation in Physical Planning. North-Holland, Amsterdam (1990)

    Google Scholar 

  20. Podinovskii, V.V.: Criteria importance theory. Math. Soc. Sci. 27, 237–252 (1994)

    Article  Google Scholar 

  21. Roy, B.: Multicriteria methodology for decision analysis. Kluwer, Dordrecht (1996)

    Book  Google Scholar 

  22. Roberts, F.S.: Measurement theory with applications to decision making, utility and the social sciences. Addison-Wesley, London (1979)

    Google Scholar 

  23. Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., Tarantola, S.: Global sensitivity analysis. John Wiley & Sons, The Primer, England (2008)

    Google Scholar 

  24. Saltelli, A., Guimarães, P.A., van der Sluijs, J.P., Funtowicz, S.: What do I make of your latinorum? sensitivity auditing of mathematical modelling. Int. J. Foresight Innov. Policy 9(2–4), 213–234 (2013)

    Article  Google Scholar 

  25. Sen, A.: The Idea of justice. The Belknap Press of Harvard University Press, Cambridge, Massachusetts (2009)

    Book  Google Scholar 

  26. Sieber, S., Domínguez, I.P.: Impact assessment of agri-food policies in Europe: methods, tools and applications. J. Policy Model. 33(1), 1–6 (2011)

    Article  Google Scholar 

  27. Torriti, J.: Impact assessment and the liberalization of the EU energy markets: evidence-based policy-making or policy-based evidence-making? J. Common Market Stud. 48, 4 (2010)

    Article  Google Scholar 

  28. Vansnick J. C. (1990) Measurement theory and decision aid- in Bana e Costa C.A. (ed.)- Readings in multiple criteria decision aid, Springer-Verlag, Berlin, pp. 81–100.

  29. Vincke, Ph.: Multicriteria decision aid. Wiley, New york (1992)

    Google Scholar 

Download references


This article has been developed in the context of the institutional activities of the Competence Centre on Modelling at the Joint Research Centre. The views expressed are purely those of the writer and may not in any circumstance be regarded as stating an official position of the European Commission.

Author information



Corresponding author

Correspondence to Giuseppe Munda.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Munda, G. Qualitative reasoning or quantitative aggregation rules for impact assessment of policy options? A multiple criteria framework. Qual Quant (2021).

Download citation


  • Public policy
  • Multiple Criteria Decision Analysis
  • Robustness analysis
  • Ex-ante impact assessment
  • Qualitative and quantitative information

JEL Classification

  • A12
  • C44
  • D04