Abstract
The article criticizes poor state performance (PSP) indexes, that is to say, cross-national data sets that mark or rank contemporary states according to their performance. In particular, I claim that current indexes provide very little genuine information about performance orderings. The criticism focuses on ‘structural’ PSP problems: those that cannot be circumvented, have no obvious solution, and generally stem from the very nature of the exercise. I suggest that there is a generalized failure to acknowledge, let alone solve, in all three stages of index building – conceptualization, codification/operationalization, aggregation – fundamental issues including defining, dealing with intrinsic ambiguity, and with lack of complete order in the informational domain (that is, the database).
Cet article est une critique des indicateurs de mauvaise performance de l′État, c′est à dire des ensembles de données nationales transversales qui classent les États contemporains en fonction de leurs performances. En particulier, je soutiens que les indicateurs utilisés actuellement fournissent très peu de réelles informations concernant les classifications des performances. Notre critique est centrée sur les problèmes «structurels» qui caractérisent ces indicateurs: ceux qui sont inévitables, n′ont pas de solutions évidentes, et résultent en général de la nature même de l′exercice. Je considère qu′on néglige généralement de reconnaître, et par conséquent de résoudre – lors des trois phases de construction des indicateurs (conceptualisation, codification/opérationalisation, et agrégation) – certains problèmes fondamentaux liés, notamment, à l′ambiguïté intrinsèque au domaine informationnel et au manque d′ordre qui caractérise ce dernier.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
I use this umbrella term because there is a huge terminological dispersion in the literature.
In the context of this article, ‘simple’ should not be taken as synonymous of ‘bad’; rather the contrary.
There are some exceptions (such as the BTI), but this many variables-several aggregation steps situation is the standard one.
In reality, there are many more issues, as defining state and statehood is not a trivial task. But these are beyond the scope of this article.
It is important to note, however, that different instances of state fragility or failure can produce events that involve similar policy challenges (for example, developed countries will face a wave of migration and so on).
Examples from Polity are useful to invoke for several reasons. In many ways, it is an exemplary database, but a number of PSP exercises include democracy as defined by Polity as a dimension of performance, associating it with state strength, with obvious conceptual problems as a result (see Inkeles, 1993; Sørensen, 1993).
See the State Failure's codebook at http://globalpolicy.gmu.edu/pitf/.
‘Other aspects of plural democracy, such as the rule of law, systems of checks and balances, freedom of the press, and so on are means to, or specific manifestations of, these general principles. We do not include coded data on civil liberties’ (see www.systemicpeace.org/inscr/p4manualv2009.pdf).
Note that precision and objectivity are different from each other (and from correctness).
This threshold, additionally, may vary from country to country.
The correlation is high but not perfect. For example, European centrally planned economies in the 1960s had high intermediate levels of development and strong states, but produced almost no data for researchers.
The other way is imputation of missing data. There are already very sophisticated imputation techniques, which are, however, rarely used in PSP indexes, many of the latter simply delete cases with missing data, which introduces serious distortions. At the same time, it is important to note that a certain percentage of missing data, the use of imputation techniques falls sharply.
For example, the World Bank's GNI categorical data are extremely useful in this respect. Note that here we are dealing with intervals: we know that country A's GNI per capita is between 0 and 200. See http://data.worldbank.org/about/country-classifications/country-and-lending-groups or http://data.worldbank.org/node/207.
The issue of whether the record is accurate is a wholly different problem.
Polity has advanced in this regard, introducing a variable about the quality of data (see Polity IV Project: Dataset Users’ Manual, p. 31. See http://www.systemicpeace.org/inscr/p4manualv2009.pdf).
Given the complexity of the problem involved, this is a very good practice.
And of course it need not be money. The underlying counting units are von Neumann utilities, but with a series of additional – and also very reasonable – assumptions, the observable numeraire becomes in effect money.
There are other aggregation functions that do not beg the substitution assumption.
For example, THE consumer, THE producer, THE politician, THE bus customer. We can suppose that, ceteris paribus, THE bus customer prefers a cheaper than a more expensive ride, and a shorter than a longer trip; the genuine exceptions are few, and contrived. On this subject, see Przeworski (2004, p. 86), for whom ‘the theory [“rational choice” or “strategic action”] works only if we can identify classes of individuals in some structure of conflict and plausibly attribute to them some objectives’. To put it differently, the political economy approach works only when it is imbued with sociology. This is why it is hard to say anything about ‘individuals’ or even ‘voters’. They are heterogeneous. Some want one thing, some want another … The more sociology we can build into theories, the greater the benefit of the economic approach’.
For example, the higher the mark the better.
There are Binomial (N,2), of these N being the total number of cases. This is the last ‘formula’ I plug in; the rest of the discussion proceeds strictly verbally.
Note that this interval representation does not assume that there is a numeraire that makes dimensions interchangeable; its assumption, instead, is that the most relevant characteristics of each case are captured by its extreme values. I believe that in many cases – for example, PSP databases – this assumption is much more reasonable than the numeraire one.
This is not true for comparable cases (owing to the restriction of monotonicity).
References
Al-Awadhi, S. and Garthwaite, P. (2006) Quantifying expert opinion for modelling fauna habitat distributions. Computational Statistics 21 (1): 121–140.
Bates, R. (2008) When Things Fell Apart: State Failure in Late-Century Africa. New York: Cambridge University Press.
Beliakov, G., Pradera, A. and Calvo, T. (2007) Aggregation Functions. A Guide For Practitioners. New York: Springer.
Bossert, W. and Peters, H. (2000) Multi-attribute decision-making in individual and social choice. Mathematical Social Sciences 40: 327–339.
Bouyssou, D., Marchant, T., Pirlot, M., Perny, P., Tsoukias, A. and Vincke, P. (2000) Evaluation and Desicion Models. A Critial Perspective. Massachusetts: Kluwer Academic Publishers.
Bouyssou, D. and Perny, P. (1992) Ranking methods for valued preference relations: A characterization of a method based on entering and leaving flows. European Journal of Operational Research 61: 186–194.
Bouyssou, D. and Vansnick, J. (1986) Noncompensatory and generalized noncompensatory preference structures. Theory and Decision 21: 251–266.
Briscoe, I. (2009) isis – Europe, http://www.isis-europe.org/pdf/2008_artrel_227_esr42-eufragilestates.pdf, accessed June 2009.
Bustince, H., Herrera, F. and Montero, J. (2008) Fuzzy Sets and Their Extensions. Representation, Aggregation and Models. New York: Springer.
Cammack, D., Mcleod, D., Menocal, A. and Christiansen, K. (2006) Donors and Fragile States Agenda: A Survey of Current Thinking and Practice-report Submitted to the Japan International Cooperation Agenda. London: ODI-JICA.
Carvalho, S. (2006) Engaging with Fragile States: An IEG Review of World Bank Support to Low-income Countries Under Stress. Washington DC: Independent Evaluation Group, World Bank.
Christenson, M., Dabelko, G.D., Esty, D.C. and Parris, T.M. (2000) State Failure Task Force Report: Phase III Findings. McLean, VA: Science Applications International Corporation, http://globalpolicy.gmu.edu/pitf/SFTF%20Phase%20III%20Report%20Final.pdf, accessed 30 September 2000.
Di John, J. (2010) The concept, causes and consequences of failed states: A critical review of the literature and agenda for research with specific reference to Sub-Saharan Africa. European Journal of Development Research 22 (1): 10–30.
Dorff, R. (2000) Addressing the challenges of failed states. Paper presented at a conference on failed state; 7–10 April, Florence, Italy.
Ehrgott, M. and Gandibleux, X. (2002) Multiple Criteria Optimization, State of the art annotated bibliographic survey. Massachusetts: Kluwer Academic Publishers.
Goldstone, J. et al (2010) A global model for forecasting political instability. American Journal of Political Science 54 (1): 190–208.
Goldstone, J.A., et al in consultation with Christenson, M. Dabelko, G.D. Esty, D.C. and Parris, T.M. (2000) State Failure Task Force Report: Phase III Findings. McLean, VA: Science Applications International Corporation.
Gros, J.G. (1996) Toward a taxonomy of failed state in the new world order: Decaying Somalia, Liberia, Rwanda, and Haiti. Third World Quarterly 17 (3): 455–471.
Gutierrez, F. and González, A. (2009) Force and ambiguity: Evaluating sources for cross-natioanl research – The case of military intervention. Crisis States Working Papers (Crisis State Research Center) 50: 1–35.
Gutierrez, F. and Argoty, A. (2010) Order preserving functions from finite posets to R: How many are there? In: F. Gutierrez, D. Buitrago, A. González and C. Lozano (eds.) Measuring Poor State Performance. Problems, Perspectives, and Paths Ahead. London: LSE UKAID, pp. 136–146.
Gutiérrez, F., González, A., Buitrago, D. and Lozano, C. (2010) Measuring poor state performance: Problems, perspectives and paths ahead. Paper presented to the workshop on ‘Measuring State Performance’; 20–21 May, London School of Economics and Political Science, London, United Kingdom.
Helman, G. and Ratner, S. (1993) Saving failed statess. Foreign Policy 89: 3–20.
Hiroshi, S. (2009) Rough Sets, Fuzzy Sets, Data Mining and Granular Computing. India: Springer.
Inkeles, A. (1993) On Measuring Democracy: Its Consequences and Concomitants. New Jersey: The state university.
Izenman, A.J. (2008) Modern Multivariate Statistical Techniques. Philadelphia, PA: Springer.
Kahraman, C. (2008) Fuzzy Multy-critera Decision Making. Theory and Applications with Recent Developments. Istanbul: Springer.
King, G. and Langche, Z. (2000) Improving quantitative studies of international conflict: A conjecture. American Political Science Review 94 (1): 21–36.
King, G. and Langche, Z. (2001) Improving forecasts of state failure. World Politics 53: 623–658.
Lebart, L. (1995) Multivariate Descriptive Statistical Analysis. Correspondence Analysis and Related Techniques for Large Matrices. New York: John Wiley & Sons.
Lootsma, F. (1997) Fuzzy Logic for Planning and Decision Making. London: Kluwer Academic Publishers.
Marples, D. (2004) The Collapse of the Soviet Union, 1985–1991. London: Longman.
Munck, G. and Verkuilen, J. (2002) Conceptualizing and measuring democracy evaluating alternative indice. Comparative Political Studies 35 (1): 5–34.
North, D., Wallis, J. and Weingast, B. (2009) Violence and Social Orders. A Conceptual Framework for Interpreting Recorded Human History. New York: Cambridge University Press.
OECD/DAC. (2007) Fragile States: Policy Commitment and Principles for Good International Engagement in Fragile States and Situations: DAC High. Paris: OECD/DAC (Organisation for Economic Co-operation and Development/Development Assistance Committee).
Przeworski, A. (2004) States and Markets. A Primer in Political Economy. New York: Cambridge University Press.
Rice, S. and Stewar, P. (2009) Index on State Weakness in the Developing World. Washington: The Brookings Institution.
Rosenau, J. (2006) The Study of World Politics Globalization and Governance. New York: Routledge.
Sartori, G. (1970) Concept misinformation in comparative politics. American Political Science Review 64 (4): 1033–1053.
Skocpol, T. (1984) Los estados y las revoluciones sociales. Un análisis comparativo de Francia, Rusia y China. Mexico: Fondo de Cultura Económico.
Sørensen, G. (1993) Democracy, authoritarianism and state strength. The European Journal of Development Research 5 (1): 6–34.
Taylor, D. (2008) Security expert says new US administration must develop fragile states. Voices of America, http://www.voanews.com/english/archive/2008-11/Security-Expert-says-New-US-Administration-must-Develop-Fragile-States-PART-2-of-5.cfm?CFID=304592121&CFTOKEN=22882106&jsessionid=8430d822b05914a49a8a686e356711325247, accessed August 2009.
Tilly, C. (1992) Coerción, capital y los Estados europeos (1990–1990). Madrid, Spain: Alianza.
USAID. (2005) Fragile states strategy. Washington, DC: USAID. Document no. PD-ACA-999.
Vreeland, J.R. (2008) Research note: A problem with polity – Unpacking anocracy. Journal of Conflict Resolution 52 (3): 401–425.
World Bank. (2006) Engaging with Fragile States. An IEG Review of World Bank Support to Low Income Countries under Stress. Washington: World Bank.
Zanaky, S., Solomon, A., Wishart, N. and Dublish, S. (1998) Multi-attribute decision making. A simulation comparison of select methods. European Journal of Operational Research 107 (3): 507–529.
Zartman, W. (1995) Collapsed States: The Disintegration and Restoration of Legitimate Authority. London: Lynne Rienner Publishers.
Acknowledgements
I present here results of the Crisis States Research Programme, sponsored by DFID. I wish to thank James Putzel, the director of the program, and my colleague Andrea González, for their support and contributions to the reflection developed here. Thanks also to the anonymous reviewers and the editor, whose comments helped improve substantially a previous version of this article.
Author information
Authors and Affiliations
Additional information
†Researcher at the Instituto de Estudios Políticos y Relaciones Internacionales – Universidad Nacional de Colombia. He presents here the results of research conducted under the auspices of the LSE Crisis States Research Centre, funded by UK aid from the Department for International Development.
Appendices
Appendix A
See Table A1.
Appendix B
Examples of Aggregation Functions
LICUS
Fund For Peace
After the sum has been done, then the following If-Then rules are used:
If 
then Alert
If 
then Danger
If 
then Moderate danger
If 
then Sustainable state
Rights and permissions
About this article
Cite this article
Sanín, F. Evaluating State Performance: A Critical View of State Failure and Fragility Indexes. Eur J Dev Res 23, 20–42 (2011). https://doi.org/10.1057/ejdr.2010.53
Published:
Issue Date:
DOI: https://doi.org/10.1057/ejdr.2010.53