Skip to main content

Advertisement

SpringerLink
Log in

How, When and Why Do Governance, Justice and Rule of Law Indicators Fail Public Policy Decision Making in Practice?

  • Article
  • Published:
Hague Journal on the Rule of Law Aims and scope Submit manuscript

Abstract

Indicators are widely used tools for evidence-based decision-making today. Yet, even very sophisticated users of indicators often forget or deliberately disregard their limits. Stretching good indicators beyond their inherent limitations may lead to inadequate policy-making. This paper examines how several well-known indicators in the field of governance, justice and the rule of law are used for policy-making in different contexts. It builds from years of field observation, debate participation and meetings with politicians and experts from around the world. Drawing from time-tested social science research principles and methods, the paper concludes with suggestions on the adequate use of indicators for public policy decision making in the broad policy area covered by the newly adopted United Nations Sustainable Development Goal 16.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. De Vaus 2002, p. 360.

  2. Bryman 2004, p. 540.

  3. United Nations Sustainable Development Goals, Goal 16, http://www.un.org/sustainabledevelopment/sustainable-development-goals/. Accessed 30 November 2015. Since the negotiators of the Post-2015 Development Agenda decided very early in the process that specific indicators were not going to be part of the negotiation leading to the adoption of the Sustainable Development Goals, the debate on how to measure the twelve targets of Goal 16 is just starting.

  4. Patton 2002.

  5. The New York Times (1899), Police and Population. News article, The New York Times. http://query.nytimes.com/gst/abstract.html?res=FA0914FF3E5811738DDDAC0894DB405B8985F0D3. Accessed 5 July 2015.

  6. Parsons 2011; Botero et al. 2011.

  7. Human Rights Watch 2015, On Their Watch. Evidence of Senior Army Officers’ Responsibility for False Positive Killings in Colombia.

  8. Pastrana 2005.

  9. DeShazo et al 2007.

  10. Uribe Velez 2012.

  11. DeShazo et al. 2007, above n. 9.

  12. ‘Under pressure from superiors to show ‘positive’ results and boost body counts in their war against guerrillas, soldiers and officers abducted victims or lured them to remote locations under false pretenses — such as with promises of work — killed them, placed weapons on their lifeless bodies, and then reported them as enemy combatants killed in action’ Human Rights Watch 2015, above n. 7.

  13. Romero 2008.

  14. In 2007 the independent think tank CSIS assessed the overall situation as follows: ‘Colombia has pulled back from the brink of a general unraveling. Eight years ago, it was beset by illegal armed groups that threatened the exercise of legitimate government authority. The country was locked in a downward cycle of violence, with guerrilla and paramilitary armies fueled by the profits from the drug trade tilting the balance of power in their favor. In the process, the Colombian people suffered from widespread abuse of basic human rights, as rates of murder, kidnapping, forced displacement, and other abuses soared. The economy plummeted into recession, driving up unemployment and exacerbating already high levels of poverty. Democratic institutions in Colombia were at risk. That Colombia avoided what many feared would be a continued downward slide into greater instability in and of itself constitutes an accomplishment. But the government and people of Colombia in the past eight years have done more than avoid disaster: they have rolled back the influence of the paramilitaries and insurgents, established a state presence in every municipality of the country for the first time in history, sharply reversed levels of violence and criminality, improved the observance of human rights, enhanced the capacity of the state to govern more democratically, and set the economy moving in a very positive direction.’ (DeShazo et al. 2007, above n. 9). More recent accounts tend to confirm this broadly positive assessment, although there are diverging views on the human rights implications of these policies (Human Rights Watch 2015, above n. 7).

  15. DeShazo et al. 2007, above n. 9, Uribe Velez 2012.

  16. Botero et al. 2012.

  17. See, e.g., Foglesong and Stone 2012.

  18. Patton 2002.

  19. Botero 2010.

  20. One of the authors of this paper testified as expert witness before the Colombian Congress, advocating for a public health approach to addressing the high prevalence of child abuse in the country.

  21. Botero 2010, above n. 19.

  22. Parsons 2011.

  23. De Vaus 2006.

  24. Saisana and Saltelli 2011.

  25. OECD (2008) Handbook on Constructing Composite Indicators: Methodology and User Guide, Resource document. Organisation for Economic Co-operation and Development.

  26. Neuman 2006.

  27. See Morse 2011.

  28. The World Bank 2007, p. xvii.

  29. The International Bank for Reconstruction and Development/The World Bank, A Decade of Measuring the Quality of Governance: Governance Matters 2007 —Worldwide Governance Indicators 1996—2006, The International Bank for Reconstruction and Development and The World Bank, http://info.worldbank.org/governance/wgi/pdf/booklet_decade_of_measuring_governance.pdf, p. 1. Accessed 25 November 2015.

  30. Kaufmann et al. (2007)

  31. Kaufmann et al. 2007, above n. 29.

  32. See The World Bank and the International Finance Corporation 2011, Doing Business 2011: Making a Difference for Entrepreneurs, at p. 20.

  33. See footnote [32].

  34. See footnote [32].

  35. The World Bank is ‘not a bank in the common sense… we provide low-interest loans, interest-free credits and grants to developing countries for a wide array of purposes that include investments in education, health, public administration, infrastructure, financial and private sector development, agriculture and environmental and natural resource management’. http://web.worldbank.org/WBSITE/EXTERNAL/EXTABOUTUS/0,,pagePK:50004410~piPK:36602~theSitePK:29708,00.html. Accessed 25 September 2011.

  36. See The World Bank and the International Finance Corporation 2011, Doing Business 2011: Making a Difference for Entrepreneurs, at p. 8.

  37. A binding constraint is the main factor to unlock the solution of a problem; in this case, the key determinant of economic development. Since most problems have multiple determinants, it is generally very difficult to identify one single determining factor.

  38. Some of the WBDB indicators have very limited applicability to the Liberian context, as they are based on standardized assumptions that may not fully apply to this country, such as the standardized size of the company or the amount of the procedure. There are few companies and procedures of this size in Liberia, where the size of the formal economy is extremely small. .

  39. Smith 1893, pp. 386-387.

  40. One of the authors was heavily involved in the original design of the World Bank’s Doing Business indicators. .

  41. Botero and Ponce (2010)‘Measuring the Rule of Law’.

  42. Using a thermometer also implies risks (Botero et al. 2011, above n. 39). Many sicknesses, including fatal ones, do not produce fever. A thermometer in these cases may deliver an accurate, and also harmfully misleading, picture of reality. An accurate thermometer reading may in some cases become a contributing factor for a delayed medical response and ultimately to death. The measurement is accurate because the thermometer measures fever, and the patient does not have fever. But what is the use of this accurate measurement if the patient dies? In our opinion, the same problem applies to the World Justice Project’s Rule of Law Index, and to all other existing cross-country indicators. Like poor Nasrudin, if we look for the lost keys in the wrong place, we will never find them.

  43. Agrast et al (2011).

  44. Results hold if the sample is divided above and below the median, instead of using top and bottom quintiles. Results are also highly consistent across years, using different samples.

  45. The White House and Legal Services Corporation’s Forum on the State of Civil Legal Assistance, April 17, 2012. Video of Mr. Koh’s remarks is available at: http://archive.lsc.gov/media/in-the-spotlight/white-house-and-lsc-co-host-forum. Accessed 25 November 2015.

  46. ‘An international finding that Canada ranks poorly when it comes to access to the courts should serve as a wakeup call to the entire justice system, says Chief Justice Beverley McLachlin. Speaking to the annual Canadian Bar Association conference today, Chief Justice McLachlin said that Canada placed ninth in a recent ranking of 12 European and North America countries. The finding—by the World Justice Institute—underlines the fact that justice is increasingly available only to the wealthy or small minority who are so poor that they qualify for legal aid programs, she said. ‘This is not terrible, but it shows that we are not doing as well as we should,’ Chief Justice McLachlin said. ‘I think the Canadian Bar Association and other groups concerned about justice have to recognize that this is an area in our justice system that needs attention.’ (K. Makin, ‘Top Judge sounds alarm over Canada’s poor access-to-courts ranking’, The Globe and Mail, August 13, 2011, www.theglobeandmail.com/news/politics/top-judge-sounds-alarm-over-canadas-poor-access-to-courts-ranking/article601068/. Accessed 25 November 2015). See also, Toronto Star, Wake-up Call from Top Judge. August 15, 2011.

  47. Using the ‘number of guerrilla casualties in combat’ as the operational tool to assign rewards for soldiers is equivalent to using a construction hammer to remove a tooth at a dentist’s office. See Sect. 1.2 above.

  48. United Nations Sustainable Development Goals, Goal 16.

  49. For instance, the World Bank’s Doing Business report is based on the assumption that business-friendly regulation is conducive to economic development. Policymakers relying on the Doing Business indicators should at least be aware of this underlying assumption. Similarly, the WJP Rule of Law Index is based on two assumptions that are copied in every presentation: ‘First, the rule of law is the foundation for communities of opportunity and equity; and Second, multidisciplinary collaboration is the most effective way to advance the rule of law.’ These assumptions may be inaccurate. Moreover, they may not even be relevant, as the direction of causality between rule of law and social development outcomes (A. Pinzon-Rondon, A. Attaran, J. Botero, A. Ruiz-Sternbert, ‘Association of Rule of Law and Health Outcomes: An Ecological Study’, BMJ Open Vol. 5, No. 10, http://bmjopen.bmj.com/content/5/10/e007004.full. Accessed 30 November 2015; Botero et al. 2013), and among various dimensions of the rule of law, are matters of academic inquiry (Botero et al. 2011, above n. 39, p. 5).

  50. Neuman (2006), p. 539.

  51. Random error may significantly affect indicator-based policy-making when there are not enough observations, or in the case of law probability events.

  52. According to Davis, Kingsbury and Merry, ‘the production of indicators is itself a political process, shaped by the power to categorize, count, analyze and promote a system of knowledge that has effects beyond the producers.’ (Davis et al. 2015).

  53. For instance, in the case of the WJP Rule of Law Index, according to Urueña, ‘one could argue that the perception of a phenomenon is not the actual phenomenon. Following [WJP Rule of Law Index]’s approach, a health indicator would not measure infant mortality, but rather the perception mothers have of the attention their babies received at the hospital. However, that point is moot with regard to the rule of law. The Index’s premise is that the ‘reality’ of the rule of law is the experience of those living (or not living) under it. In that sense, it would be wrong to accuse WJP of mistaking perception with reality, when it is arguable that there is no reality beyond such perception. In the case of the rule of law, the indicator allows the WJP to see reality – in this very restricted sense, reality is being constructed by the indicator.’ (Urueña 2015, p. 87).

  54. This is especially important for policy making, as ‘weighting implies a “subjective” evaluation, which is particularly sensitive in case of complex, interrelated, and multidimensional phenomena’ (Saisana and Saltelli 2011).

  55. Botero et al. (2011).

  56. OECD (2008), above n. 25.

  57. Social science research methods and principles vary across disciplines, from economics to political science, and from anthropology to public health (see, e.g., de Vaus 2002; Neuman 2006; Bryman 2004; Stock and Watson 2008; Hennekens and Buring 1987; Ruane 2005; Patton 2002). Yet, the basic notions of validity, reliability and precision of measures, as they apply to indicators, are equally important across all social sciences.

  58. De Vaus (2006).

  59. Botero et al. (2011).

  60. Botero et al. (2012).

  61. Foglesong and Stone (2012).

  62. ‘A scale is a composite measure of a concept, a measure composed of information derived from several questions or indicators. To create a scale we simply convert the information contained in several relatively specific indicators into one new and more abstract variable.’ (De Vaus 2002, p. 180).

  63. De Vaus (2002).

  64. Constraints on government powers; absence of corruption; open government; fundamental rights; order and security; regulatory enforcement; civil justice; criminal justice; and informal justice.

  65. Saisana and Saltelli (2011).

  66. The notion of the rule of law has been extensively discussed at the HJROL. While there are diverging views on the precise components, all significant definitions largely agree on a handful of key elements of this multi-faceted and complex concept.

  67. 7.1 People can access and afford civil justice. 7.2 Civil justice is free of discrimination. 7.3 Civil justice is free of corruption. 7.4 Civil justice is free of improper government influence. 7.5 Civil justice is not subject to unreasonable delay. 7.6 Civil justice is effectively enforced. 7.7 ADR is accessible, impartial, and effective.

  68. Sub-factor 7.1 comprises five sub-sub-factors: 7.1.1 People are aware of available remedies; 7.1.2 People can access and afford legal advice and representation; 7.1.3 Procedures are accessible; 7.1.4 Accessibility of courts; 7.1.5 Costs (Courts, lawyers and procedures). Each one of these sub-sub-factors is in itself a fourth-layer basket of several variables (Botero et al. 2011, above n. 39).

  69. Botero et al. 2011, above n. 39.

  70. Above n. 1.

  71. Indicators related to Goal 16 were essentially ignored during the second meeting of the Inter-agency and Expert Group on Sustainable Development Goal Indicators (IAEG-SDGs), held on 26–28 October 2015 at the United Nations Conference Center in Bangkok, Thailand.

References

Download references

Author information

Authors and Affiliations

  1. The World Justice Project (WJP), 1025 Vermont Avenue NW Suite 1200, Washington DC, 20005, USA

    Juan Carlos Botero & Christine S. Pratt

  2. Universidad del Rosario School of Medicine, Bogota, Colombia

    Angela Maria Pinzon-Rondon

Authors
  1. Juan Carlos Botero
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Angela Maria Pinzon-Rondon
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christine S. Pratt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Juan Carlos Botero.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Botero, J.C., Pinzon-Rondon, A.M. & Pratt, C.S. How, When and Why Do Governance, Justice and Rule of Law Indicators Fail Public Policy Decision Making in Practice?. Hague J Rule Law 8, 51–74 (2016). https://doi.org/10.1007/s40803-015-0020-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40803-015-0020-8

Keywords

Search

Navigation