Skip to main content

Evaluating State Performance: A Critical View of State Failure and Fragility Indexes

The European Journal of Development Research volume 23, pages 20–42 (2011)Cite this article

Abstract

The article criticizes poor state performance (PSP) indexes, that is to say, cross-national data sets that mark or rank contemporary states according to their performance. In particular, I claim that current indexes provide very little genuine information about performance orderings. The criticism focuses on ‘structural’ PSP problems: those that cannot be circumvented, have no obvious solution, and generally stem from the very nature of the exercise. I suggest that there is a generalized failure to acknowledge, let alone solve, in all three stages of index building – conceptualization, codification/operationalization, aggregation – fundamental issues including defining, dealing with intrinsic ambiguity, and with lack of complete order in the informational domain (that is, the database).

Cet article est une critique des indicateurs de mauvaise performance de l′État, c′est à dire des ensembles de données nationales transversales qui classent les États contemporains en fonction de leurs performances. En particulier, je soutiens que les indicateurs utilisés actuellement fournissent très peu de réelles informations concernant les classifications des performances. Notre critique est centrée sur les problèmes «structurels» qui caractérisent ces indicateurs: ceux qui sont inévitables, n′ont pas de solutions évidentes, et résultent en général de la nature même de l′exercice. Je considère qu′on néglige généralement de reconnaître, et par conséquent de résoudre – lors des trois phases de construction des indicateurs (conceptualisation, codification/opérationalisation, et agrégation) – certains problèmes fondamentaux liés, notamment, à l′ambiguïté intrinsèque au domaine informationnel et au manque d′ordre qui caractérise ce dernier.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

Figure 1
Figure 2
Figure 3

Notes

  1. I use this umbrella term because there is a huge terminological dispersion in the literature.

  2. In the context of this article, ‘simple’ should not be taken as synonymous of ‘bad’; rather the contrary.

  3. There are some exceptions (such as the BTI), but this many variables-several aggregation steps situation is the standard one.

  4. In reality, there are many more issues, as defining state and statehood is not a trivial task. But these are beyond the scope of this article.

  5. It is important to note, however, that different instances of state fragility or failure can produce events that involve similar policy challenges (for example, developed countries will face a wave of migration and so on).

  6. Examples from Polity are useful to invoke for several reasons. In many ways, it is an exemplary database, but a number of PSP exercises include democracy as defined by Polity as a dimension of performance, associating it with state strength, with obvious conceptual problems as a result (see Inkeles, 1993; Sørensen, 1993).

  7. See the State Failure's codebook at http://globalpolicy.gmu.edu/pitf/.

  8. ‘Other aspects of plural democracy, such as the rule of law, systems of checks and balances, freedom of the press, and so on are means to, or specific manifestations of, these general principles. We do not include coded data on civil liberties’ (see www.systemicpeace.org/inscr/p4manualv2009.pdf).

  9. Note that precision and objectivity are different from each other (and from correctness).

  10. This threshold, additionally, may vary from country to country.

  11. The correlation is high but not perfect. For example, European centrally planned economies in the 1960s had high intermediate levels of development and strong states, but produced almost no data for researchers.

  12. The other way is imputation of missing data. There are already very sophisticated imputation techniques, which are, however, rarely used in PSP indexes, many of the latter simply delete cases with missing data, which introduces serious distortions. At the same time, it is important to note that a certain percentage of missing data, the use of imputation techniques falls sharply.

  13. For example, the World Bank's GNI categorical data are extremely useful in this respect. Note that here we are dealing with intervals: we know that country A's GNI per capita is between 0 and 200. See http://data.worldbank.org/about/country-classifications/country-and-lending-groups or http://data.worldbank.org/node/207.

  14. The issue of whether the record is accurate is a wholly different problem.

  15. Polity has advanced in this regard, introducing a variable about the quality of data (see Polity IV Project: Dataset Users’ Manual, p. 31. See http://www.systemicpeace.org/inscr/p4manualv2009.pdf).

  16. Given the complexity of the problem involved, this is a very good practice.

  17. And of course it need not be money. The underlying counting units are von Neumann utilities, but with a series of additional – and also very reasonable – assumptions, the observable numeraire becomes in effect money.

  18. There are other aggregation functions that do not beg the substitution assumption.

  19. For example, THE consumer, THE producer, THE politician, THE bus customer. We can suppose that, ceteris paribus, THE bus customer prefers a cheaper than a more expensive ride, and a shorter than a longer trip; the genuine exceptions are few, and contrived. On this subject, see Przeworski (2004, p. 86), for whom ‘the theory [“rational choice” or “strategic action”] works only if we can identify classes of individuals in some structure of conflict and plausibly attribute to them some objectives’. To put it differently, the political economy approach works only when it is imbued with sociology. This is why it is hard to say anything about ‘individuals’ or even ‘voters’. They are heterogeneous. Some want one thing, some want another … The more sociology we can build into theories, the greater the benefit of the economic approach’.

  20. For example, the higher the mark the better.

  21. There are Binomial (N,2), of these N being the total number of cases. This is the last ‘formula’ I plug in; the rest of the discussion proceeds strictly verbally.

  22. Note that this interval representation does not assume that there is a numeraire that makes dimensions interchangeable; its assumption, instead, is that the most relevant characteristics of each case are captured by its extreme values. I believe that in many cases – for example, PSP databases – this assumption is much more reasonable than the numeraire one.

  23. This is not true for comparable cases (owing to the restriction of monotonicity).

References

  • Al-Awadhi, S. and Garthwaite, P. (2006) Quantifying expert opinion for modelling fauna habitat distributions. Computational Statistics 21 (1): 121–140.

    Article  Google Scholar 

  • Bates, R. (2008) When Things Fell Apart: State Failure in Late-Century Africa. New York: Cambridge University Press.

    Book  Google Scholar 

  • Beliakov, G., Pradera, A. and Calvo, T. (2007) Aggregation Functions. A Guide For Practitioners. New York: Springer.

    Google Scholar 

  • Bossert, W. and Peters, H. (2000) Multi-attribute decision-making in individual and social choice. Mathematical Social Sciences 40: 327–339.

    Article  Google Scholar 

  • Bouyssou, D., Marchant, T., Pirlot, M., Perny, P., Tsoukias, A. and Vincke, P. (2000) Evaluation and Desicion Models. A Critial Perspective. Massachusetts: Kluwer Academic Publishers.

    Book  Google Scholar 

  • Bouyssou, D. and Perny, P. (1992) Ranking methods for valued preference relations: A characterization of a method based on entering and leaving flows. European Journal of Operational Research 61: 186–194.

    Article  Google Scholar 

  • Bouyssou, D. and Vansnick, J. (1986) Noncompensatory and generalized noncompensatory preference structures. Theory and Decision 21: 251–266.

    Article  Google Scholar 

  • Briscoe, I. (2009) isis – Europe, http://www.isis-europe.org/pdf/2008_artrel_227_esr42-eufragilestates.pdf, accessed June 2009.

  • Bustince, H., Herrera, F. and Montero, J. (2008) Fuzzy Sets and Their Extensions. Representation, Aggregation and Models. New York: Springer.

    Book  Google Scholar 

  • Cammack, D., Mcleod, D., Menocal, A. and Christiansen, K. (2006) Donors and Fragile States Agenda: A Survey of Current Thinking and Practice-report Submitted to the Japan International Cooperation Agenda. London: ODI-JICA.

    Google Scholar 

  • Carvalho, S. (2006) Engaging with Fragile States: An IEG Review of World Bank Support to Low-income Countries Under Stress. Washington DC: Independent Evaluation Group, World Bank.

    Book  Google Scholar 

  • Christenson, M., Dabelko, G.D., Esty, D.C. and Parris, T.M. (2000) State Failure Task Force Report: Phase III Findings. McLean, VA: Science Applications International Corporation, http://globalpolicy.gmu.edu/pitf/SFTF%20Phase%20III%20Report%20Final.pdf, accessed 30 September 2000.

    Google Scholar 

  • Di John, J. (2010) The concept, causes and consequences of failed states: A critical review of the literature and agenda for research with specific reference to Sub-Saharan Africa. European Journal of Development Research 22 (1): 10–30.

    Article  Google Scholar 

  • Dorff, R. (2000) Addressing the challenges of failed states. Paper presented at a conference on failed state; 7–10 April, Florence, Italy.

  • Ehrgott, M. and Gandibleux, X. (2002) Multiple Criteria Optimization, State of the art annotated bibliographic survey. Massachusetts: Kluwer Academic Publishers.

    Google Scholar 

  • Goldstone, J. et al (2010) A global model for forecasting political instability. American Journal of Political Science 54 (1): 190–208.

    Article  Google Scholar 

  • Goldstone, J.A., et al in consultation with Christenson, M. Dabelko, G.D. Esty, D.C. and Parris, T.M. (2000) State Failure Task Force Report: Phase III Findings. McLean, VA: Science Applications International Corporation.

    Google Scholar 

  • Gros, J.G. (1996) Toward a taxonomy of failed state in the new world order: Decaying Somalia, Liberia, Rwanda, and Haiti. Third World Quarterly 17 (3): 455–471.

    Article  Google Scholar 

  • Gutierrez, F. and González, A. (2009) Force and ambiguity: Evaluating sources for cross-natioanl research – The case of military intervention. Crisis States Working Papers (Crisis State Research Center) 50: 1–35.

    Google Scholar 

  • Gutierrez, F. and Argoty, A. (2010) Order preserving functions from finite posets to R: How many are there? In: F. Gutierrez, D. Buitrago, A. González and C. Lozano (eds.) Measuring Poor State Performance. Problems, Perspectives, and Paths Ahead. London: LSE UKAID, pp. 136–146.

    Google Scholar 

  • Gutiérrez, F., González, A., Buitrago, D. and Lozano, C. (2010) Measuring poor state performance: Problems, perspectives and paths ahead. Paper presented to the workshop on ‘Measuring State Performance’; 20–21 May, London School of Economics and Political Science, London, United Kingdom.

  • Helman, G. and Ratner, S. (1993) Saving failed statess. Foreign Policy 89: 3–20.

    Google Scholar 

  • Hiroshi, S. (2009) Rough Sets, Fuzzy Sets, Data Mining and Granular Computing. India: Springer.

    Google Scholar 

  • Inkeles, A. (1993) On Measuring Democracy: Its Consequences and Concomitants. New Jersey: The state university.

    Google Scholar 

  • Izenman, A.J. (2008) Modern Multivariate Statistical Techniques. Philadelphia, PA: Springer.

    Book  Google Scholar 

  • Kahraman, C. (2008) Fuzzy Multy-critera Decision Making. Theory and Applications with Recent Developments. Istanbul: Springer.

    Book  Google Scholar 

  • King, G. and Langche, Z. (2000) Improving quantitative studies of international conflict: A conjecture. American Political Science Review 94 (1): 21–36.

    Article  Google Scholar 

  • King, G. and Langche, Z. (2001) Improving forecasts of state failure. World Politics 53: 623–658.

    Article  Google Scholar 

  • Lebart, L. (1995) Multivariate Descriptive Statistical Analysis. Correspondence Analysis and Related Techniques for Large Matrices. New York: John Wiley & Sons.

    Google Scholar 

  • Lootsma, F. (1997) Fuzzy Logic for Planning and Decision Making. London: Kluwer Academic Publishers.

    Book  Google Scholar 

  • Marples, D. (2004) The Collapse of the Soviet Union, 1985–1991. London: Longman.

    Google Scholar 

  • Munck, G. and Verkuilen, J. (2002) Conceptualizing and measuring democracy evaluating alternative indice. Comparative Political Studies 35 (1): 5–34.

    Article  Google Scholar 

  • North, D., Wallis, J. and Weingast, B. (2009) Violence and Social Orders. A Conceptual Framework for Interpreting Recorded Human History. New York: Cambridge University Press.

    Book  Google Scholar 

  • OECD/DAC. (2007) Fragile States: Policy Commitment and Principles for Good International Engagement in Fragile States and Situations: DAC High. Paris: OECD/DAC (Organisation for Economic Co-operation and Development/Development Assistance Committee).

  • Przeworski, A. (2004) States and Markets. A Primer in Political Economy. New York: Cambridge University Press.

    Google Scholar 

  • Rice, S. and Stewar, P. (2009) Index on State Weakness in the Developing World. Washington: The Brookings Institution.

    Google Scholar 

  • Rosenau, J. (2006) The Study of World Politics Globalization and Governance. New York: Routledge.

    Google Scholar 

  • Sartori, G. (1970) Concept misinformation in comparative politics. American Political Science Review 64 (4): 1033–1053.

    Article  Google Scholar 

  • Skocpol, T. (1984) Los estados y las revoluciones sociales. Un análisis comparativo de Francia, Rusia y China. Mexico: Fondo de Cultura Económico.

    Google Scholar 

  • Sørensen, G. (1993) Democracy, authoritarianism and state strength. The European Journal of Development Research 5 (1): 6–34.

    Article  Google Scholar 

  • Taylor, D. (2008) Security expert says new US administration must develop fragile states. Voices of America, http://www.voanews.com/english/archive/2008-11/Security-Expert-says-New-US-Administration-must-Develop-Fragile-States-PART-2-of-5.cfm?CFID=304592121&CFTOKEN=22882106&jsessionid=8430d822b05914a49a8a686e356711325247, accessed August 2009.

  • Tilly, C. (1992) Coerción, capital y los Estados europeos (1990–1990). Madrid, Spain: Alianza.

    Google Scholar 

  • USAID. (2005) Fragile states strategy. Washington, DC: USAID. Document no. PD-ACA-999.

  • Vreeland, J.R. (2008) Research note: A problem with polity – Unpacking anocracy. Journal of Conflict Resolution 52 (3): 401–425.

    Article  Google Scholar 

  • World Bank. (2006) Engaging with Fragile States. An IEG Review of World Bank Support to Low Income Countries under Stress. Washington: World Bank.

  • Zanaky, S., Solomon, A., Wishart, N. and Dublish, S. (1998) Multi-attribute decision making. A simulation comparison of select methods. European Journal of Operational Research 107 (3): 507–529.

    Article  Google Scholar 

  • Zartman, W. (1995) Collapsed States: The Disintegration and Restoration of Legitimate Authority. London: Lynne Rienner Publishers.

    Google Scholar 

Download references

Acknowledgements

I present here results of the Crisis States Research Programme, sponsored by DFID. I wish to thank James Putzel, the director of the program, and my colleague Andrea González, for their support and contributions to the reflection developed here. Thanks also to the anonymous reviewers and the editor, whose comments helped improve substantially a previous version of this article.

Author information

Authors and Affiliations

  1. Instituto de Estudios Políticos y Relaciones Internacionales, Universidad Nacional de Colombia, Carrera 7 no. 83-36 Apartamento 202, Bogotá, Colombia

    Francisco Gutiérrez Sanín

Authors
  1. Francisco Gutiérrez Sanín
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

†Researcher at the Instituto de Estudios Políticos y Relaciones Internacionales – Universidad Nacional de Colombia. He presents here the results of research conducted under the auspices of the LSE Crisis States Research Centre, funded by UK aid from the Department for International Development.

Appendices

Appendix A

See Table A1.

Table a1 Synoptic panorama of selected PSP definitions
Full size table

Appendix B

Examples of Aggregation Functions

LICUS

Fund For Peace

After the sum has been done, then the following If-Then rules are used:

If then Alert

If then Danger

If then Moderate danger

If then Sustainable state

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Sanín, F. Evaluating State Performance: A Critical View of State Failure and Fragility Indexes. Eur J Dev Res 23, 20–42 (2011). https://doi.org/10.1057/ejdr.2010.53

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1057/ejdr.2010.53

Keywords

  • state performance
  • statehood
  • classification
  • order
  • ambiguity
  • aggregation