Composite indicators aggregate multi-dimensional processes into simplified concepts often aiming at underpinning the development of data-driven narratives for policy consumption. Due to methodological issues, doubts are often raised about the robustness of the composite indicators and the significance of the associated policy messages. In this paper we use expert panel information (derived from budget allocation and analytic hierarchy process) on the relative importance of the underlying indicators included in a composite indicator and run in tandem uncertainty and sensitivity analysis to gain useful insights during the process of composite indicators building. We discuss the extent to which variance-based sensitivity analysis may increase transparency or make policy inference more defensible by using the United Nation’s Technology Achievement Index as an illustration.
- Analytic Hierarchy Process
- Weighting Scheme
- Input Factor
- Composite Indicator
- Global Sensitivity Analysis
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, access via your institution.
Tax calculation will be finalised at checkout
Purchases are for personal use onlyLearn about institutional subscriptions
Unable to display preview. Download preview PDF.
Bandura R. (2005) Measuring country performance and state behavior: a survey of composite indices, Background paper prepared for “The new public finance: responding to global challenges”, United Nations Development Programme, http://www. thenewpublicfinance. org
Brand D. A., M. Saisana, L. A. Rynn, F. Pennoni, A. B. Lowenfels (2007) Comparative analysis of alcohol control policies in 30 countries, PLoS Medicine, 4, 752-759, www. plosmedicine. org.
Canadian Council on Learning (2007) The 2007 Composite Learning Index: Helping Communities Improve their Quality of Life, Ottawa, pp. 40, www. ccl- cca. ca.
Chan K., S. Tarantola, A. Saltelli, I. M. Sobol’ (2000) Variance based methods. In Sensitivity Analysis (eds. A. Saltelli, K. Chan, M. Scott), pp. 167-197. New York: Wiley.
Cherchye L, K. Lovell, W. Moesen, T. Van Puyenbroeck (2007) One market, one number? A composite indicator assessment of EU internal market dynamics, European Economic Review, 51, 749-779.
Cherchye L., W. Moesen, N. Rogge, T. Van Puyenbroeck, M. Saisana, A. Saltelli, R. Liska, S. Tarantola (2007) Creating composite indicators with data envelopment analysis and robustness analysis: the case of the technology achievement index, Journal of Operational Research Society, online 27/06/07, doi: 10. 1057/ palgrave. jors. 2602445
Cox D., R. Fitzpatrick, A. Fletcher, S. Gore, D. Spiegelhalter, D. Jones (1992) Quality-of-life assessment: can we keep it simple? J. R. Statist. Soc. 155(3), 353-393.
EC-JRC (2007) Information server on composite indicators, http://farmweb. jrc. cec. eu. int/ci/ provided by the Joint Research Centre of the European Commission.
EPA (2001) Draft guidance on the development, evaluation, and application of regulatory environmental models. Council for regulatory environmental model- ing (CREM), http://cfpub. epa. gov/crem/cremlib. cfm whitepapers.
Esty D. C., M. A. Levy, T. Srebotnjak, A. de Sherbinin, C. H. Kim, B. Anderson (2006) Pilot 2006 Environmental Performance Index. New Haven: Yale Center for Environmental Law & Policy.
Esty D. C., M. Levy, T. Srebotnjak, A. de Sherbinin (2005) 2005 Environmental Sustainability Index: Benchmarking National Environmental Stewardship. New Haven: Yale Center for Environmental Law & Policy.
Freudenberg M. (2003) Composite indicators of country performance: a critical assessment. Report DSTI/IND(2003)5, OECD, Paris.
Funtowicz, S. O., J. R. Ravetz (1990) Uncertainty and Quality in Science for Policy. Dordrecht, NL: Kluwer Academic Publishers.
Gough C., N. Castells, S. Funtowicz (1998) Integrated Assessment: an emerging methodology for complex issues, Journal Environmental Modeling and Assess- ment, 3, 19-29.
Höeg P. (1995) Borderliners. Seal Books Publisher.
Homma T., A. Saltelli (1996) Importance measures in global sensitivity analysis of model output. Reliability Engineering and System Safety, 52(1), 1-17.
Kennedy P. (2007) A Guide to Econometrics, Fifth edition. Blackwell Publishing.
Leamer E. (1978) Specification Searches: Ad hoc Inferences with Nonexperimen-tal Data. New York: Wiley.
Moldan B., S. Billharz, R. Matravers (1997) Sustainability Indicators: Report of the Project on Indicators of Sustainable Development. SCOPE 58. Chichester and New York: Wiley.
Munda G. (2004) Social multi-criteria evaluation (SMCE): methodological foun- dations and operational consequences, European Journal of Operational Re- search, 158/3, 662-677.
Munda G., M. Nardo (2005) Non-compensatory composite indicators for ranking Countries: A defensible setting, European Commission, Joint Research Centre, IPSC, Italy, EUR 21833 EN.
Nardo M., M. Saisana, A. Saltelli, S. Tarantola, A. Hoffman, E. Giovannini (2005) Handbook on constructing composite indicators: methodology and users guide, OECD-JRC joint publication, OECD Statistics Working Paper, STD/DOC(2005)3, JT00188147, pp. 108.
Nardo M., M. Saisana, A. Saltelli, S. Tarantola (2005) Tools for Composite Indicators Building, European Commission, Joint Research Centre, IPSC, Italy, EUR 21682 EN, pp. 131.
Oreskes N., K. Shrader-Frechette, K. Belitz (1994) Verification, validation, and confirmation of numerical models in the earth sciences. Science 263, 641-646.
Rosen, R. (1991) Life Itself, Columbia University Press, New York.
Saaty T. L. (1980) The Analytic Hierarchy Process, New York: McGraw-Hill.
Saaty R. W. (1987) The analytic hierarchy process - what it is and how it is used. Mathematical Modelling, 9, 161-176.
Saisana M. (2007) Robustness issues & Critical Assessment of the 2007 Composite Learning Index, Working paper, at http://composite-indicators. jrc. ec. europa. eu/Document/CLI2007%20Report%20on%20Validation v1.0. pdf
Saisana M., S. Tarantola (2002) State-of-the-art report on current methodologies and practices for composite indicator development, European Commission, Joint Research Centre, IPSC, Italy, EUR 20408 EN.
Saisana M., A. Saltelli, S. Tarantola (2005) Uncertainty and sensitivity analy- sis techniques as tools for the analysis and validation of composite indicators, Journal of the Royal Statistical Society A, 168(2), 307-323.
Saltelli A. (2002) Making best use of model valuations to compute sensitivity indices. Computer Physics Communications, 145, 280-297.
Saltelli A. (2006) Composite indicators between analysis and advocacy. Social Indicators Research, doi 10. 1007/s11205-006-0024-9.
Saltelli A., S. Tarantola (2002) On the relative importance of input factors in mathematical models: safety assessment for nuclear waste disposal, Journal of American Statistical Association, 97(459), 702-709.
Saltelli A., K. Chan, M. Scott (2000) Sensitivity Analysis, Probability and Sta-tistics series, New York: Wiley.
Saltelli A., S. Tarantola, F. Campolongo (2000) Sensitivity analysis as an ingre- dient of modelling. Statistical Science, 15, 377-395.
Saltelli A., S. Tarantola, F. Campolongo, M. Ratto (2004) Sensitivity Analysis in Practice, a Guide to Assessing Scientific Models. New York: Wiley. SIM- LAB: software for sensitivity analysis at http://www. jrc. cec. eu. int/uasa/prj-sa- soft. asp.
Saltelli A., M. Ratto, F. Campolongo, J. Carboni, D. Gabelli, M. Saisana, S. Tarantola, T. Andres (2007) Global Sensitivity Analysis: Gauging the Worth of Scientific Models, New York: Wiley.
Sharpe A. (2004) Literature Review of Frameworks for Macro-indicators, Centre for the Study of Living Standards, Ottawa, CAN.
Sobol’ I. M. (1967) On the distribution of points in a cube and the approxi- mate evaluation of integrals. USSR Computational Mathematics and Physics, 7, 86-112.
Sobol’ I. M. (1976) Uniformly distributed sequences with an additional uniform property. Zh. V&ymacr;chisl. Mat. mat. Fiz. 16, 1332-1337. English translation: U. S. S. R. Comput. Maths. Math. Phys. 16, 236-242.
Sobol’ I. M. (1993) Sensitivity analysis for non-linear mathematical models. Mathematical Modelling & Computational Experiment 1, 407-414.
Tarantola S., M. Saisana, A. Saltelli, F. Schmiedel, N. Leapman (2002) Sta- tistical techniques and participatory approaches for the composition of the European Internal Market Index 1992-2001, European Commission, Joint Research Centre, IPSC, Italy, EUR 20547 EN.
United Nations (2001) Human Development Report, United Kingdom: Oxford University Press.
Editors and Affiliations
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Saisana, M., Saltelli, A. (2008). Expert Panel Opinion and Global Sensitivity Analysis for Composite Indicators. In: Graziani, F. (eds) Computational Methods in Transport: Verification and Validation. Lecture Notes in Computational Science and Engineering, vol 62. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-77362-7_11
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-77361-0
Online ISBN: 978-3-540-77362-7