Abstract
Composite indicators (CIs) are often used for benchmarking countries' performance, but they frequently stir controversies about the unavoidable subjectivity in their construction. Data Envelopment Analysis helps to overcome some key limitations, as it does not need any prior information on either the normalization of sub-indicators or on an agreed unique set of weights. Still, subjective decisions remain, and such modelling uncertainty propagates onto countries' CI scores and rankings. Uncertainty and sensitivity analysis are therefore needed to assess the robustness of the final outcome and to analyse how much each source of uncertainty contributes to the output variance. The current paper reports on these issues, using the Technology Achievement Index as an illustration.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Atkinson A, Cantillon B, Marlier E and Nolan B (2002). Social Indicators. The EU and Social Inclusion. Oxford University Press: Oxford.
Banker RD, Chang H and Cooper WW (1996). Simulation studies of efficiency, returns to scale and misspecification with nonlinear functions in DEA. Ann Opns Res 66: 233–253.
Box G, Hunter W and Hunter J (1978). Statistics for Experimenters. John Wiley: New York.
Cazals C, Florens JP and Simar L (2002). Nonparametric frontier estimation: A robust approach. J Econ 106: 1–25.
Charnes A and Cooper WW (1985). Preface to topics in data envelopment analysis. Ann Opns Res 2: 59–94.
Charnes A, Cooper WW and Rhodes E (1978). Measuring the efficiency of decision making units. Eur J Opl Res 2: 429–444.
Cherchye L and Kuosmanen T (2006). Benchmarking Sustainable Development: A Synthetic Meta-Index Approach. In: McGillivray M. and Clarke M. (eds). Understanding Human Well-being. United Nations University Press: Tokyo, pp. 139–168.
Cherchye L, Moesen W and Van Puyenbroeck T (2004). Legitimately diverse yet comparable: On synthesizing social inclusion performance in the EU. J Common Market Stud 42: 919–955.
Cherchye L, Knox Lovell CA, Moesen W and Van Puyenbroeck T (2007). One market, one number? A composite indicator assessment of EU internal market dynamics. Eur Econ Rev 51: 749–779.
Conover WJ (1980). Practical Nonparametric Statistics. John Wiley: New York.
Cooper WW, Seiford L and Tone K (2000). Data Envelopment Analysis. Kluwer Academic Publishers: Boston.
Cooper WW, Shanling L, Seiford LM and Zhu J (2004). Sensitivity analysis in DEA. In: Cooper W.W., Seiford L.M. and Zhu J. (eds). Handbook on Data Envelopment Analysis. Kluwer Academic Publishers: Boston, pp. 75–97.
Desai M, Fukuda-Parr S, Johansson C and Sagasti F (2002). Measuring the technology achievement of nations and the capacity to participate in the network age. J Human Dev 3: 95–122.
Despotis DK (2005). A reassessment of the human development index via data envelopment analysis. J Opl Res Soc 56: 969–980.
EPA (2004). Council for Regulatory Environmental Modeling, ‘Draft Guidance on the Development, Evaluation, and Application of Regulatory Environmental Models’, http://www.epa.gov/osp/crem/library/CREM%20Guidance%20Draft%2012_03.pdf.
European Commission (2004). The EU Economy Review 2004. European Economy, vol. 6, Office for Official Publications of the EC: Luxembourg.
European Commission (2005). European innovation scoreboard 2005—Comparative analysis of innovation performance, http://trendchart.cordis.lu/scoreboards/scoreboard2005/pdf/EIS%202005.pdf.
Freudenberg M (2003). Composite indicators of country performance: A critical assessment, STI Working Paper 2003/16, OECD, Paris.
Homma T and Saltelli A (1996). Importance measures in global sensitivity analysis of nonlinear models. Reliab Eng Syst Safety 52: 1–17.
Kao C and Tung HT (2005). Data envelopment analysis with common weights: The compromise solution approach. J Opl Res Soc 56: 1196–1203.
Mahlberg B and Obersteiner M (2001). Remeasuring the HDI by data envelopment analysis. IIASA interim report IR-01-069, Luxemburg.
Munda G and Nardo M (2003). On the methodological foundations of composite indicators used for ranking countries. Mimeo, Universitat Autonoma de Barcelona.
Nardo M, Saisana M, Saltelli A and Tarantola S (2005a). Tools for composite indicators building. EUR 21682 EN, European Commission-Joint Research Centre.
Nardo M, Saisana M, Saltelli A, Tarantola S, Hoffman A and Giovannini E (2005b). Handbook on constructing composite indicators: methodology and user's guide, OECD Statistics Working Paper, JT00188147.
Saisana M, Saltelli A and Tarantola S (2005). Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators. J Roy Stat Soc 168: 307–323.
Saltelli A (2002). Making best use of model evaluations to compute sensitivity indices. Comput Phys Commu 145: 280–297.
Saltelli A, Chan K and Scott M (2000). Sensitivity Analysis. Probability and Statistics Series. John Wiley & Sons: New York.
Saltelli A, Tarantola S, Campolongo F and Ratto M (2004). Sensitivity Analysis in Practice. A Guide to Assessing Scientific Models. Wiley: England.
Simar L (2003). Detecting outliers in frontier models: A simple approach. J Prod Anal 20: 391–424.
Simar L and Wilson P (1998). Sensitivity of efficiency scores: How to bootstrap in Nonparametric frontier models. Mngt Sci 44: 49–61.
Sobol IM (1967). On the distribution of points in a cube and the approximate evaluation of integrals. USSR Comput Math Phys 7: 86–112.
Storrie D and Bjurek H (2000). Benchmarking European labour market performance with efficiency frontier techniques, CELMS Discussion paper, Göteborg University.
Thanassoulis E, Portela MC and Allen R (2004). Incorporating value judgments in DEA. In: Cooper W.W., Seiford L.M. and Zhu J. (eds). Handbook on Data Envelopment Analysis. Kluwer Academic Publishers: Dordrecht, pp. 99–138.
United Nations (2001). Annex 2.1: The technology achievement index: A new measure of countries' ability to participate in the network age. Human Development Report, Oxford University Press: United Kingdom, http://www.undp.org.
Valdmanis V (1992). Sensitivity analysis for DEA models: An empirical example using public vs. NFP hospitals. J Public Econ 48: 185–205.
Wilson PW (1995). Detecting influential observations in data envelopment analysis. J Prod Anal 6: 27–46.
Wong Y-HB and Beasley JE (1990). Restricting weight flexibility in data envelopment analysis. J Opl Res Soc 47: 136–150.
Acknowledgements
We thank two anonymous referees for insightful comments and suggestions. This paper is an offshoot of the KEI-project (contract no 502529) that is part of priority 8 of the policy-orientated research under the European Commission's Sixth Framework Programme (see http://kei.publicstatistics.net/). Laurens Cherchye thanks the Fund for Scientific Research-Flanders (FWO-Vlaanderen) for his postdoctoral fellowship.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Cherchye, L., Moesen, W., Rogge, N. et al. Creating composite indicators with DEA and robustness analysis: the case of the Technology Achievement Index. J Oper Res Soc 59, 239–251 (2008). https://doi.org/10.1057/palgrave.jors.2602445
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1057/palgrave.jors.2602445