Skip to main content
SpringerLink
Log in

Comparing MCDA Aggregation Methods in Constructing Composite Indicators Using the Shannon-Spearman Measure

  • Published:
Social Indicators Research Aims and scope Submit manuscript

Abstract

Composite indicators have been increasingly recognized as a useful tool for performance monitoring, benchmarking comparisons and public communication in a wide range of fields. The usefulness of a composite indicator depends heavily on the underlying data aggregation scheme where multiple criteria decision analysis (MCDA) is commonly used. A problem in this application is the determination of an appropriate MCDA aggregation method. Of the many criteria for comparing MCDA methods, the Shannon-Spearman measure (SSM) is one that compares alternative MCDA aggregation methods in constructing composite indicators based on the information loss concept. This paper assesses the effectiveness of the SSM using Monte Carlo approach-based uncertain analysis and variance-based sensitivity analysis techniques. It is found that most of the variation in the SSM arises from the uncertainty in choosing an aggregation method. Therefore, the SSM can be considered as an effective measure for comparing MCDA aggregation methods in constructing composite indicators. We also use the SSM to evaluate five MCDA aggregation methods in constructing composite indicators and present the findings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. Although there are many different interpretations of criteria weights in MCDA (e.g., Choo et al. 1999), the most commonly used ones in constructing CIs appear to be the “trade-off ratios” and “coefficients of importance” (Munda 2005; Nardo et al. 2005). In this study the criteria weights are assumed to be exogenous and will not be discussed in detail.

  2. There is not a universally agreed upon definition of “validity.” Different researchers/analysts may interpret it differently, such as predictive validity, estimative validity, methodological validity, construct validity and convergent validity (Hobbs 1986). The various definitions of “validity” arise from the different purposes of MCDA methods in application. In this study, the validity of SSM refers to whether most of its variation can be explained by the change in the MCDA aggregation method.

References

  • Booysen, F. (2002). An overview and evaluation of composite indices of development. Social Indicators Research, 59, 115–151. doi:10.1023/A:1016275505152.

    Article  Google Scholar 

  • Chan, K., Tarantola, S., Saltelli, A., & Sobol’, I. M. (2000). Variance based methods. In A. Saltelli, K. Chan, & E. Scott (Eds.), Sensitivity analysis (pp. 167–197). New York: Wiley.

    Google Scholar 

  • Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring the efficiency of decision making units. European Journal of Operational Research, 2, 429–444. doi:10.1016/0377-2217(78)90138-8.

    Article  Google Scholar 

  • Cherchye, L., Lovell, C. A. K., Moesen, W., & van Puyenbroeck, T. (2007a). One market, one number? A composite indicator assessment of EU internal market dynamics. European Economic Review, 51, 749–779. doi:10.1016/j.euroecorev.2006.03.011.

    Article  Google Scholar 

  • Cherchye, L., Moesen, W., Rogge, N., & van Puyenbroeck, T. (2007b). An introduction to ‘benefit of the doubt’ composite indicators. Social Indicators Research, 82, 111–145. doi:10.1007/s11205-006-9029-7.

    Article  Google Scholar 

  • Cherchye, L., Moesen, W., Rogge, N., van Puyenbroeck, T., Saisana, M., Saltelli, A., et al. (2008). Creating composite indicators with DEA and robustness analysis: The case of the Technology Achievement Index. Journal of the Operational Research Society, 59, 239–251.

    Article  Google Scholar 

  • Choo, E. U., Schoner, B., & Wedley, W. C. (1999). Interpretation of criteria weights in multicriteria decision making. Computers & Industrial Engineering, 37, 527–541. doi:10.1016/S0360-8352(00)00019-X.

    Article  Google Scholar 

  • Despotis, D. K. (2005a). A reassessment of the human development index via data envelopment analysis. The Journal of the Operational Research Society, 56, 96–980. doi:10.1057/palgrave.jors.2601927.

    Google Scholar 

  • Despotis, D. K. (2005b). Measuring human development via data envelopment analysis: The case of Asia and the Pacific. Omega, 33, 385–390. doi:10.1016/j.omega.2004.07.002.

    Article  Google Scholar 

  • Diakoulaki, D., Mavrotas, G., & Papayannakis, L. (1995). Determining objective weights in multiple criteria problems: The CRITIC method. Computers & Operations Research, 22, 763–770. doi:10.1016/0305-0548(94)00059-H.

    Article  Google Scholar 

  • Diaz-Balteiro, L., & Romero, C. (2004). In search of a natural systems sustainability index. Ecological Economics, 49, 401–405. doi:10.1016/j.ecolecon.2004.02.005.

    Article  Google Scholar 

  • Duckstein, L., Gershon, M., & Mcaniff, R. (1982). Model selection in multiobjective decision making for river basin planning. Advances in Water Resources, 5, 178–184. doi:10.1016/0309-1708(82)90040-9.

    Article  Google Scholar 

  • Ebert, U., & Welsch, H. (2004). Meaningful environmental indices: A social choice approach. Journal of Environmental Economics and Management, 47, 270–283. doi:10.1016/j.jeem.2003.09.001.

    Article  Google Scholar 

  • Efron, B., & Tibshirani, R. (1993). An introduction to the Bootstrap. New York: Chapman and Hall.

    Google Scholar 

  • Esty, D. C., Levy, M. A., Srebotnjak, T., & de Sherbinin, A. (2005). 2005 environmental sustainability index: Benchmarking national environmental stewardship. New Haven, Conn: Yale Center for Environmental Law & Policy.

    Google Scholar 

  • Figueira, J., Greco, S., & Ehrgott, M. (Eds.). (2005). Multiple criteria decision analysis: State of the art surveys. New York: Springer.

    Google Scholar 

  • Gershon, M., & Duckstein, L. (1984). A procedure for selection of a multiobjective technique with application to water and mineral resources. Applied Mathematics and Computation, 14, 245–271. doi:10.1016/0096-3003(84)90024-9.

    Article  Google Scholar 

  • Guitouni, A., & Martel, J. M. (1998). Tentative guidelines to help choosing an appropriate MCDA method. European Journal of Operational Research, 109, 501–521. doi:10.1016/S0377-2217(98)00073-3.

    Article  Google Scholar 

  • Hobbs, B. F. (1986). What can we learn from experiments in multiobjective decision analysis? IEEE Transactions on Systems, Man and Cybernetics, 16, 384–394. doi:10.1109/TSMC.1986.4308970.

    Article  Google Scholar 

  • Hobbs, B. F., Chankong, V., Hamadeh, W., & Stakhiv, E. Z. (1992). Does choice of multicriteria method matter? An experiment in water resources planning. Water Resources Research, 28, 1767–1779. doi:10.1029/92WR00712.

    Article  Google Scholar 

  • Hoskins, B.L., & Mascherini, M. Measuring active citizenship through the development of a composite indicator. Social Indicators Research, in press. doi:10.1007/s11205-008-9271-2.

  • Lee, G. K. L., & Chan, E. H. W. (2008). The analytic hierarchy process (AHP) approach for assessment of urban renewal proposals. Social Indicators Research, 89, 155–168. doi:10.1007/s11205-007-9228-x.

    Article  Google Scholar 

  • Lun, G., Holzer, D., Tappeiner, G., & Tappeiner, U. (2006). The stability of rankings derived from composite indicators: Analysis of the “Il Sole 24 Ore” quality of life report. Social Indicators Research, 77, 307–331. doi:10.1007/s11205-005-4505-z.

    Article  Google Scholar 

  • Munda, G. (2005). “Measuring sustainability”: A multi-criterion framework. Environment, Development and Sustainability, 7, 117–134. doi:10.1007/s10668-003-4713-0.

    Article  Google Scholar 

  • Munda, G., & Nardo, M. (2003). On the methodological foundations of composite indicators used for ranking countries. Paper presented at the First OECD/JRC Workshop on Composite Indicators of Country performance, Ispra, JRC.

  • Murias, P., de Miguel, J. C., & Rodriguez, D. (2008). A composite indicator for university quality assessment: The case of Spanish higher education system. Social Indicators Research, 89, 129–146. doi:10.1007/s11205-007-9226-z.

    Article  Google Scholar 

  • Murias, P., Martinez, F., & de Miguel, C. (2006). An economic wellbeing index for the Spanish provinces: A data envelopment analysis approach. Social Indicators Research, 77, 395–417. doi:10.1007/s11205-005-2613-4.

    Article  Google Scholar 

  • Nardo, M., Paisana, M., Saltelli, A., Tarantola, S., Hoffman, A., & Giovannini, E. (2005). Handbook on constructing composite indicators: Methodology and user Guide. OECD Statistics Working Paper 2005/3, OECD Statistics Directorate.

  • Olson, D. L. (2001). Comparison of three multicriteria methods to predict known outcomes. European Journal of Operational Research, 130, 576–587. doi:10.1016/S0377-2217(99)00416-6.

    Article  Google Scholar 

  • Saisana, M., Saltelli, A., & Tarantola, S. (2005). Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators. Journal of the Royal Statistical Society. Series A (General), 168, 307–323.

    Google Scholar 

  • Salminen, P., Hokkanen, J., & Lahdelma, R. (1998). Comparing multicriteria methods in the context of environmental problems. European Journal of Operational Research, 104, 485–496. doi:10.1016/S0377-2217(96)00370-0.

    Article  Google Scholar 

  • Saltelli, A. (2007). Composite indicators between analysis and advocacy. Social Indicators Research, 81, 65–77. doi:10.1007/s11205-006-0024-9.

    Article  Google Scholar 

  • Saltelli, A., Tarantola, S., Campolongo, F., & Ratto, M. (2004). Sensitivity analysis in practice: A guide to assessing scientific models. New York: Wiley.

    Google Scholar 

  • Sinha, B. K., & Shah, K. R. (2003). On some aspects of data integration techniques with environmental applications. Environmetrics, 14, 409–416. doi:10.1002/env.595.

    Article  Google Scholar 

  • Triantaphyllou, E. (2000). Multi-criteria decision making methods: A comparative study. Boston: Kluwer Academic Publishers.

    Google Scholar 

  • United Nations (UN). (2001). Human development report. Oxford: Oxford University Press.

    Google Scholar 

  • Yoon, K. P., & Hwang, C. L. (1995). Multiple attribute decision making: An introduction. Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Zaim, O., Fare, R., & Grosskopf, S. (2001). An economic approach to achievement and improvement indexes. Social Indicators Research, 56, 91–118. doi:10.1023/A:1011837827659.

    Article  Google Scholar 

  • Zanakis, S. H., Solomon, A., Wishart, N., & Dublish, S. (1998). Multi-attribute decision making: A simulation comparison of select methods. European Journal of Operational Research, 107, 507–529. doi:10.1016/S0377-2217(97)00147-1.

    Article  Google Scholar 

  • Zeleny, M. (1982). Multiple criteria decision making. New York: McGraw-Hill.

    Google Scholar 

  • Zhou, P., & Ang, B. W. (2008). Indicators for assessing sustainability performance. In K. B. Misra (Ed.), Handbook of performability engineering (pp. 905–918). London: Springer-Verlag.

    Google Scholar 

  • Zhou, P., Ang, B. W., & Poh, K. L. (2006). Comparing aggregating methods for constructing the composite environmental index: An objective measure. Ecological Economics, 59, 305–311. doi:10.1016/j.ecolecon.2005.10.018.

    Article  Google Scholar 

  • Zhou, P., Ang, B. W., & Poh, K. L. (2007a). A mathematical programming approach to constructing composite indicators. Ecological Economics, 62, 291–297. doi:10.1016/j.ecolecon.2006.12.020.

    Article  Google Scholar 

  • Zhou, P., Poh, K. L., & Ang, B. W. (2007b). A non-radial DEA approach to measuring environmental performance. European Journal of Operational Research, 178, 1–9. doi:10.1016/j.ejor.2006.04.038.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Economics and Management, Nanjing University of Aeronautics and Astronautics, 29 Yudao Street, 210016, Nanjing, People’s Republic of China

    P. Zhou

  2. Department of Industrial and Systems Engineering, National University of Singapore, 10 Kent Ridge Crescent, Singapore, 119260, Singapore

    B. W. Ang

Authors
  1. P. Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. B. W. Ang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to P. Zhou.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zhou, P., Ang, B.W. Comparing MCDA Aggregation Methods in Constructing Composite Indicators Using the Shannon-Spearman Measure. Soc Indic Res 94, 83–96 (2009). https://doi.org/10.1007/s11205-008-9338-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11205-008-9338-0

Keywords

Search

Navigation