Expert Panel Opinion and Global Sensitivity Analysis for Composite Indicators

  • M. Saisana
  • A. Saltelli
Part of the Lecture Notes in Computational Science and Engineering book series (LNCSE, volume 62)


Composite indicators aggregate multi-dimensional processes into simplified concepts often aiming at underpinning the development of data-driven narratives for policy consumption. Due to methodological issues, doubts are often raised about the robustness of the composite indicators and the significance of the associated policy messages. In this paper we use expert panel information (derived from budget allocation and analytic hierarchy process) on the relative importance of the underlying indicators included in a composite indicator and run in tandem uncertainty and sensitivity analysis to gain useful insights during the process of composite indicators building. We discuss the extent to which variance-based sensitivity analysis may increase transparency or make policy inference more defensible by using the United Nation’s Technology Achievement Index as an illustration.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bandura R. (2005) Measuring country performance and state behavior: a survey of composite indices, Background paper prepared for “The new public finance: responding to global challenges”, United Nations Development Programme, http://www. thenewpublicfinance. org
  2. 2.
    Brand D. A., M. Saisana, L. A. Rynn, F. Pennoni, A. B. Lowenfels (2007) Comparative analysis of alcohol control policies in 30 countries, PLoS Medicine, 4, 752-759, www. plosmedicine. org.
  3. 3.
    Canadian Council on Learning (2007) The 2007 Composite Learning Index: Helping Communities Improve their Quality of Life, Ottawa, pp. 40, www. ccl- cca. ca.
  4. 4.
    Chan K., S. Tarantola, A. Saltelli, I. M. Sobol’ (2000) Variance based methods. In Sensitivity Analysis (eds. A. Saltelli, K. Chan, M. Scott), pp. 167-197. New York: Wiley.Google Scholar
  5. 5.
    Cherchye L, K. Lovell, W. Moesen, T. Van Puyenbroeck (2007) One market, one number? A composite indicator assessment of EU internal market dynamics, European Economic Review, 51, 749-779.Google Scholar
  6. 6.
    Cherchye L., W. Moesen, N. Rogge, T. Van Puyenbroeck, M. Saisana, A. Saltelli, R. Liska, S. Tarantola (2007) Creating composite indicators with data envelopment analysis and robustness analysis: the case of the technology achievement index, Journal of Operational Research Society, online 27/06/07, doi: 10. 1057/ palgrave. jors. 2602445Google Scholar
  7. 7.
    Cox D., R. Fitzpatrick, A. Fletcher, S. Gore, D. Spiegelhalter, D. Jones (1992) Quality-of-life assessment: can we keep it simple? J. R. Statist. Soc. 155(3), 353-393.CrossRefGoogle Scholar
  8. 8.
    EC-JRC (2007) Information server on composite indicators, http://farmweb.  jrc. cec. eu. int/ci/ provided by the Joint Research Centre of the European Commission.
  9. 9.
    EPA (2001) Draft guidance on the development, evaluation, and application of regulatory environmental models. Council for regulatory environmental model- ing (CREM), http://cfpub. epa. gov/crem/cremlib. cfm whitepapers.
  10. 10.
    Esty D. C., M. A. Levy, T. Srebotnjak, A. de Sherbinin, C. H. Kim, B. Anderson (2006) Pilot 2006 Environmental Performance Index. New Haven: Yale Center for Environmental Law & Policy.Google Scholar
  11. 11.
    Esty D. C., M. Levy, T. Srebotnjak, A. de Sherbinin (2005) 2005 Environmental Sustainability Index: Benchmarking National Environmental Stewardship. New Haven: Yale Center for Environmental Law & Policy.Google Scholar
  12. 12.
    Freudenberg M. (2003) Composite indicators of country performance: a critical assessment. Report DSTI/IND(2003)5, OECD, Paris.Google Scholar
  13. 13.
    Funtowicz, S. O., J. R. Ravetz (1990) Uncertainty and Quality in Science for Policy. Dordrecht, NL: Kluwer Academic Publishers.Google Scholar
  14. 14.
    Gough C., N. Castells, S. Funtowicz (1998) Integrated Assessment: an emerging methodology for complex issues, Journal Environmental Modeling and Assess- ment, 3, 19-29.CrossRefGoogle Scholar
  15. 15.
    Höeg P. (1995) Borderliners. Seal Books Publisher.Google Scholar
  16. 16.
    Homma T., A. Saltelli (1996) Importance measures in global sensitivity analysis of model output. Reliability Engineering and System Safety, 52(1), 1-17.CrossRefGoogle Scholar
  17. 17.
    Kennedy P. (2007) A Guide to Econometrics, Fifth edition. Blackwell Publishing.Google Scholar
  18. 18.
    Leamer E. (1978) Specification Searches: Ad hoc Inferences with Nonexperimen-tal Data. New York: Wiley.Google Scholar
  19. 19.
    Moldan B., S. Billharz, R. Matravers (1997) Sustainability Indicators: Report of the Project on Indicators of Sustainable Development. SCOPE 58. Chichester and New York: Wiley.Google Scholar
  20. 20.
    Munda G. (2004) Social multi-criteria evaluation (SMCE): methodological foun- dations and operational consequences, European Journal of Operational Re- search, 158/3, 662-677.MATHCrossRefMathSciNetGoogle Scholar
  21. 21.
    Munda G., M. Nardo (2005) Non-compensatory composite indicators for ranking Countries: A defensible setting, European Commission, Joint Research Centre, IPSC, Italy, EUR 21833 EN.Google Scholar
  22. 22.
    Nardo M., M. Saisana, A. Saltelli, S. Tarantola, A. Hoffman, E. Giovannini (2005) Handbook on constructing composite indicators: methodology and users guide, OECD-JRC joint publication, OECD Statistics Working Paper, STD/DOC(2005)3, JT00188147, pp. 108.Google Scholar
  23. 23.
    Nardo M., M. Saisana, A. Saltelli, S. Tarantola (2005) Tools for Composite Indicators Building, European Commission, Joint Research Centre, IPSC, Italy, EUR 21682 EN, pp. 131.Google Scholar
  24. 24.
    Oreskes N., K. Shrader-Frechette, K. Belitz (1994) Verification, validation, and confirmation of numerical models in the earth sciences. Science 263, 641-646.CrossRefGoogle Scholar
  25. 25.
    Rosen, R. (1991) Life Itself, Columbia University Press, New York.Google Scholar
  26. 26.
    Saaty T. L. (1980) The Analytic Hierarchy Process, New York: McGraw-Hill.MATHGoogle Scholar
  27. 27.
    Saaty R. W. (1987) The analytic hierarchy process - what it is and how it is used. Mathematical Modelling, 9, 161-176.MATHCrossRefMathSciNetGoogle Scholar
  28. 28.
    Saisana M. (2007) Robustness issues & Critical Assessment of the 2007 Composite Learning Index, Working paper, at http://composite-indicators. jrc.  ec. europa. eu/Document/CLI2007%20Report%20on%20Validation v1.0. pdf
  29. 29.
    Saisana M., S. Tarantola (2002) State-of-the-art report on current methodologies and practices for composite indicator development, European Commission, Joint Research Centre, IPSC, Italy, EUR 20408 EN.Google Scholar
  30. 30.
    Saisana M., A. Saltelli, S. Tarantola (2005) Uncertainty and sensitivity analy- sis techniques as tools for the analysis and validation of composite indicators, Journal of the Royal Statistical Society A, 168(2), 307-323.MATHCrossRefMathSciNetGoogle Scholar
  31. 31.
    Saltelli A. (2002) Making best use of model valuations to compute sensitivity indices. Computer Physics Communications, 145, 280-297.MATHCrossRefGoogle Scholar
  32. 32.
    Saltelli A. (2006) Composite indicators between analysis and advocacy. Social Indicators Research, doi 10. 1007/s11205-006-0024-9.Google Scholar
  33. 33.
    Saltelli A., S. Tarantola (2002) On the relative importance of input factors in mathematical models: safety assessment for nuclear waste disposal, Journal of American Statistical Association, 97(459), 702-709.MATHCrossRefMathSciNetGoogle Scholar
  34. 34.
    Saltelli A., K. Chan, M. Scott (2000) Sensitivity Analysis, Probability and Sta-tistics series, New York: Wiley.Google Scholar
  35. 35.
    Saltelli A., S. Tarantola, F. Campolongo (2000) Sensitivity analysis as an ingre- dient of modelling. Statistical Science, 15, 377-395.CrossRefMathSciNetGoogle Scholar
  36. 36.
    Saltelli A., S. Tarantola, F. Campolongo, M. Ratto (2004) Sensitivity Analysis in Practice, a Guide to Assessing Scientific Models. New York: Wiley. SIM- LAB: software for sensitivity analysis at http://www. jrc. cec. eu. int/uasa/prj-sa- soft. asp.
  37. 37.
    Saltelli A., M. Ratto, F. Campolongo, J. Carboni, D. Gabelli, M. Saisana, S. Tarantola, T. Andres (2007) Global Sensitivity Analysis: Gauging the Worth of Scientific Models, New York: Wiley.Google Scholar
  38. 38.
    Sharpe A. (2004) Literature Review of Frameworks for Macro-indicators, Centre for the Study of Living Standards, Ottawa, CAN.Google Scholar
  39. 39.
    Sobol’ I. M. (1967) On the distribution of points in a cube and the approxi- mate evaluation of integrals. USSR Computational Mathematics and Physics, 7, 86-112.CrossRefMathSciNetGoogle Scholar
  40. 40.
    Sobol’ I. M. (1976) Uniformly distributed sequences with an additional uniform property. Zh. V&ymacr;chisl. Mat. mat. Fiz. 16, 1332-1337. English translation: U. S. S. R. Comput. Maths. Math. Phys. 16, 236-242.CrossRefMathSciNetGoogle Scholar
  41. 41.
    Sobol’ I. M. (1993) Sensitivity analysis for non-linear mathematical models. Mathematical Modelling & Computational Experiment 1, 407-414.MathSciNetGoogle Scholar
  42. 42.
    Tarantola S., M. Saisana, A. Saltelli, F. Schmiedel, N. Leapman (2002) Sta- tistical techniques and participatory approaches for the composition of the European Internal Market Index 1992-2001, European Commission, Joint Research Centre, IPSC, Italy, EUR 20547 EN.Google Scholar
  43. 43.
    United Nations (2001) Human Development Report, United Kingdom: Oxford University Press.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • M. Saisana
    • 1
  • A. Saltelli
    • 1
  1. 1.European Commission-Joint Research CentreIspraItaly

Personalised recommendations