Skip to main content
SpringerLink
Log in
Book cover

Computational Methods in Transport: Verification and Validation pp 251–275Cite as

Expert Panel Opinion and Global Sensitivity Analysis for Composite Indicators

  • Conference paper

Part of the Lecture Notes in Computational Science and Engineering book series (LNCSE,volume 62)

Summary

Composite indicators aggregate multi-dimensional processes into simplified concepts often aiming at underpinning the development of data-driven narratives for policy consumption. Due to methodological issues, doubts are often raised about the robustness of the composite indicators and the significance of the associated policy messages. In this paper we use expert panel information (derived from budget allocation and analytic hierarchy process) on the relative importance of the underlying indicators included in a composite indicator and run in tandem uncertainty and sensitivity analysis to gain useful insights during the process of composite indicators building. We discuss the extent to which variance-based sensitivity analysis may increase transparency or make policy inference more defensible by using the United Nation’s Technology Achievement Index as an illustration.

Keywords

  • Analytic Hierarchy Process
  • Weighting Scheme
  • Input Factor
  • Composite Indicator
  • Global Sensitivity Analysis

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Bandura R. (2005) Measuring country performance and state behavior: a survey of composite indices, Background paper prepared for “The new public finance: responding to global challenges”, United Nations Development Programme, http://www. thenewpublicfinance. org

  2. Brand D. A., M. Saisana, L. A. Rynn, F. Pennoni, A. B. Lowenfels (2007) Comparative analysis of alcohol control policies in 30 countries, PLoS Medicine, 4, 752-759, www. plosmedicine. org.

  3. Canadian Council on Learning (2007) The 2007 Composite Learning Index: Helping Communities Improve their Quality of Life, Ottawa, pp. 40, www. ccl- cca. ca.

  4. Chan K., S. Tarantola, A. Saltelli, I. M. Sobol’ (2000) Variance based methods. In Sensitivity Analysis (eds. A. Saltelli, K. Chan, M. Scott), pp. 167-197. New York: Wiley.

    Google Scholar 

  5. Cherchye L, K. Lovell, W. Moesen, T. Van Puyenbroeck (2007) One market, one number? A composite indicator assessment of EU internal market dynamics, European Economic Review, 51, 749-779.

    Google Scholar 

  6. Cherchye L., W. Moesen, N. Rogge, T. Van Puyenbroeck, M. Saisana, A. Saltelli, R. Liska, S. Tarantola (2007) Creating composite indicators with data envelopment analysis and robustness analysis: the case of the technology achievement index, Journal of Operational Research Society, online 27/06/07, doi: 10. 1057/ palgrave. jors. 2602445

    Google Scholar 

  7. Cox D., R. Fitzpatrick, A. Fletcher, S. Gore, D. Spiegelhalter, D. Jones (1992) Quality-of-life assessment: can we keep it simple? J. R. Statist. Soc. 155(3), 353-393.

    CrossRef  Google Scholar 

  8. EC-JRC (2007) Information server on composite indicators, http://farmweb.  jrc. cec. eu. int/ci/ provided by the Joint Research Centre of the European Commission.

  9. EPA (2001) Draft guidance on the development, evaluation, and application of regulatory environmental models. Council for regulatory environmental model- ing (CREM), http://cfpub. epa. gov/crem/cremlib. cfm whitepapers.

  10. Esty D. C., M. A. Levy, T. Srebotnjak, A. de Sherbinin, C. H. Kim, B. Anderson (2006) Pilot 2006 Environmental Performance Index. New Haven: Yale Center for Environmental Law & Policy.

    Google Scholar 

  11. Esty D. C., M. Levy, T. Srebotnjak, A. de Sherbinin (2005) 2005 Environmental Sustainability Index: Benchmarking National Environmental Stewardship. New Haven: Yale Center for Environmental Law & Policy.

    Google Scholar 

  12. Freudenberg M. (2003) Composite indicators of country performance: a critical assessment. Report DSTI/IND(2003)5, OECD, Paris.

    Google Scholar 

  13. Funtowicz, S. O., J. R. Ravetz (1990) Uncertainty and Quality in Science for Policy. Dordrecht, NL: Kluwer Academic Publishers.

    Google Scholar 

  14. Gough C., N. Castells, S. Funtowicz (1998) Integrated Assessment: an emerging methodology for complex issues, Journal Environmental Modeling and Assess- ment, 3, 19-29.

    CrossRef  Google Scholar 

  15. Höeg P. (1995) Borderliners. Seal Books Publisher.

    Google Scholar 

  16. Homma T., A. Saltelli (1996) Importance measures in global sensitivity analysis of model output. Reliability Engineering and System Safety, 52(1), 1-17.

    CrossRef  Google Scholar 

  17. Kennedy P. (2007) A Guide to Econometrics, Fifth edition. Blackwell Publishing.

    Google Scholar 

  18. Leamer E. (1978) Specification Searches: Ad hoc Inferences with Nonexperimen-tal Data. New York: Wiley.

    Google Scholar 

  19. Moldan B., S. Billharz, R. Matravers (1997) Sustainability Indicators: Report of the Project on Indicators of Sustainable Development. SCOPE 58. Chichester and New York: Wiley.

    Google Scholar 

  20. Munda G. (2004) Social multi-criteria evaluation (SMCE): methodological foun- dations and operational consequences, European Journal of Operational Re- search, 158/3, 662-677.

    CrossRef  MATH  MathSciNet  Google Scholar 

  21. Munda G., M. Nardo (2005) Non-compensatory composite indicators for ranking Countries: A defensible setting, European Commission, Joint Research Centre, IPSC, Italy, EUR 21833 EN.

    Google Scholar 

  22. Nardo M., M. Saisana, A. Saltelli, S. Tarantola, A. Hoffman, E. Giovannini (2005) Handbook on constructing composite indicators: methodology and users guide, OECD-JRC joint publication, OECD Statistics Working Paper, STD/DOC(2005)3, JT00188147, pp. 108.

    Google Scholar 

  23. Nardo M., M. Saisana, A. Saltelli, S. Tarantola (2005) Tools for Composite Indicators Building, European Commission, Joint Research Centre, IPSC, Italy, EUR 21682 EN, pp. 131.

    Google Scholar 

  24. Oreskes N., K. Shrader-Frechette, K. Belitz (1994) Verification, validation, and confirmation of numerical models in the earth sciences. Science 263, 641-646.

    CrossRef  Google Scholar 

  25. Rosen, R. (1991) Life Itself, Columbia University Press, New York.

    Google Scholar 

  26. Saaty T. L. (1980) The Analytic Hierarchy Process, New York: McGraw-Hill.

    MATH  Google Scholar 

  27. Saaty R. W. (1987) The analytic hierarchy process - what it is and how it is used. Mathematical Modelling, 9, 161-176.

    CrossRef  MATH  MathSciNet  Google Scholar 

  28. Saisana M. (2007) Robustness issues & Critical Assessment of the 2007 Composite Learning Index, Working paper, at http://composite-indicators. jrc.  ec. europa. eu/Document/CLI2007%20Report%20on%20Validation v1.0. pdf

  29. Saisana M., S. Tarantola (2002) State-of-the-art report on current methodologies and practices for composite indicator development, European Commission, Joint Research Centre, IPSC, Italy, EUR 20408 EN.

    Google Scholar 

  30. Saisana M., A. Saltelli, S. Tarantola (2005) Uncertainty and sensitivity analy- sis techniques as tools for the analysis and validation of composite indicators, Journal of the Royal Statistical Society A, 168(2), 307-323.

    CrossRef  MATH  MathSciNet  Google Scholar 

  31. Saltelli A. (2002) Making best use of model valuations to compute sensitivity indices. Computer Physics Communications, 145, 280-297.

    CrossRef  MATH  Google Scholar 

  32. Saltelli A. (2006) Composite indicators between analysis and advocacy. Social Indicators Research, doi 10. 1007/s11205-006-0024-9.

    Google Scholar 

  33. Saltelli A., S. Tarantola (2002) On the relative importance of input factors in mathematical models: safety assessment for nuclear waste disposal, Journal of American Statistical Association, 97(459), 702-709.

    CrossRef  MATH  MathSciNet  Google Scholar 

  34. Saltelli A., K. Chan, M. Scott (2000) Sensitivity Analysis, Probability and Sta-tistics series, New York: Wiley.

    Google Scholar 

  35. Saltelli A., S. Tarantola, F. Campolongo (2000) Sensitivity analysis as an ingre- dient of modelling. Statistical Science, 15, 377-395.

    CrossRef  MathSciNet  Google Scholar 

  36. Saltelli A., S. Tarantola, F. Campolongo, M. Ratto (2004) Sensitivity Analysis in Practice, a Guide to Assessing Scientific Models. New York: Wiley. SIM- LAB: software for sensitivity analysis at http://www. jrc. cec. eu. int/uasa/prj-sa- soft. asp.

  37. Saltelli A., M. Ratto, F. Campolongo, J. Carboni, D. Gabelli, M. Saisana, S. Tarantola, T. Andres (2007) Global Sensitivity Analysis: Gauging the Worth of Scientific Models, New York: Wiley.

    Google Scholar 

  38. Sharpe A. (2004) Literature Review of Frameworks for Macro-indicators, Centre for the Study of Living Standards, Ottawa, CAN.

    Google Scholar 

  39. Sobol’ I. M. (1967) On the distribution of points in a cube and the approxi- mate evaluation of integrals. USSR Computational Mathematics and Physics, 7, 86-112.

    CrossRef  MathSciNet  Google Scholar 

  40. Sobol’ I. M. (1976) Uniformly distributed sequences with an additional uniform property. Zh. V&ymacr;chisl. Mat. mat. Fiz. 16, 1332-1337. English translation: U. S. S. R. Comput. Maths. Math. Phys. 16, 236-242.

    CrossRef  MathSciNet  Google Scholar 

  41. Sobol’ I. M. (1993) Sensitivity analysis for non-linear mathematical models. Mathematical Modelling & Computational Experiment 1, 407-414.

    MathSciNet  Google Scholar 

  42. Tarantola S., M. Saisana, A. Saltelli, F. Schmiedel, N. Leapman (2002) Sta- tistical techniques and participatory approaches for the composition of the European Internal Market Index 1992-2001, European Commission, Joint Research Centre, IPSC, Italy, EUR 20547 EN.

    Google Scholar 

  43. United Nations (2001) Human Development Report, United Kingdom: Oxford University Press.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. European Commission-Joint Research Centre, Ispra, Italy

    M. Saisana & A. Saltelli

Authors
  1. M. Saisana
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. A. Saltelli
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Lawrence Livermore National Laboratory, 7000 East Ave., Livermore, CA, 94550-9234, USA

    Frank Graziani

Rights and permissions

Reprints and Permissions

Copyright information

© 2008 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Saisana, M., Saltelli, A. (2008). Expert Panel Opinion and Global Sensitivity Analysis for Composite Indicators. In: Graziani, F. (eds) Computational Methods in Transport: Verification and Validation. Lecture Notes in Computational Science and Engineering, vol 62. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-77362-7_11

Download citation

search

Navigation