An Introduction to ‘Benefit of the Doubt’ Composite Indicators

Abstract

Despite their increasing use, composite indicators remain controversial. The undesirable dependence of countries’ rankings on the preliminary normalization stage, and the disagreement among experts/stakeholders on the specific weighting scheme used to aggregate sub-indicators, are often invoked to undermine the credibility of composite indicators. Data envelopment analysis may be instrumental in overcoming these limitations. One part of its appeal in the composite indicator context stems from its invariance to measurement units, which entails that a normalization stage can be skipped. Secondly, it fills the informational gap in the ‘right’ set of weights by generating flexible ‘benefit of the doubt’-weights for each evaluated country. The ease of interpretation is a third advantage of the specific model that is the main focus of this paper. In sum, the method may help to neutralize some recurring sources of criticism on composite indicators, allowing one to shift the focus to other, and perhaps more essential stages of their construction.

Appendices

Appendix 1: The Technology Achievement Index and Expert Opinion on Weights

The United Nations’ TAI index is developed to capture country performances in creating, adapting and using global technological innovations. Desai et al. (2002) define it as a composite indicator of technological progress that ranks countries on a comparative global scale. The TAI focuses on achievements in four dimensions: creating new technology, diffusing recent innovations, diffusing existing technologies which are still basic inputs to the industrial and the network age and building a human skill base for technology creation and adaptation. Eight sub-indicators capture these dimensions (with two sub-indicators for each dimension): the number of patents granted per 1,000,000 people, the receipt of royalties in US $ per 1000 inhabitants; the number of internet hosts per 1000 people, the exports of high and medium technology products as a share of total goods exports; the number of telephone lines per 1000 people (expressed in logarithms), electricity consumption per capita in kWh (also in logs); the mean years of schooling, and the gross enrolment ratio of tertiary students in science, mathematics and engineering. The eight selected sub-indicators all are ‘goods’ so that higher values reflect better performance. For extensive explanations on the sub-indicators we refer to Desai et al. (2002). The list immediately shows the different units of measurement across sub-indicators, a recurring issue in the construction of composite indicators. In the calculations of the actual TAI, data are first normalized to overcome this problem. We deviate from this common practice in the main text by aggregating the original data. In the original TAI the UN uses equal weights to aggregate the sub-indicators.

Nardo et al. collected opinion from 21 experts about TAI weighting schemes. The weights defined in Table A1 were obtained by using the so-called Budget allocation method.¹⁵ This is a participatory method in which experts have to distribute a budget of 100 points over the sub-indicators allocating more to what they regard to be the more important sub-indicators. It is this information we use to illustrate some possible pie share bounds.

Table A1 Budget allocation weights for the Technology Achievement Index

Appendix 2: Summary Tables for (Benefit of the Doubt) Composite Indicator Values

The following table(s) A2 recapture the original TAI-values for 23 countries, and adds to this the index values as provided by the ‘full flexibility’ benefit of the doubt model (4)–(5a)–(5b); a model with ordinal sub-indicator share restrictions (8) added; one with relative sub-indicator share restrictions (9).

The second part of the table adds the proportional pie share restrictions (11), and its counterpart (12), putting bounds on the four categories. One interesting idea, that we have not pursued in this paper, is checking the robustness of the country rankings (or scores) to this different scenario’s by uncertainty/sensitivity analysis (see e.g. Nardo et al. (2005), or Saisana et al. (2005)).

Table A2 CI values and country rankings following different scenarios (exact scenarios explained in the main text)