Synthesizing and standardizing criteria for the evaluation of sustainability indicators in the water sector

Abstract

Indicators are one of the tools available in planning and management projects that aid in the decision-making process and the monitoring of those decisions on the path to sustainable use and management of water and natural resources. However, the quality and trustworthiness of the indicators depend on the constant improvement in the means to assess and design criteria sets. The identification of criteria to evaluate indicators and its subsequence selection are not an ordinary task. The research identified a proliferation of unconsolidated criteria in use in the sustainability and water resource management domains. In response, a process of synthesis and consolidation was undertaken in order to reduce the level of redundancies and to identify possible candidates for “core criteria” that are identified as being a relevant part of most evaluation frameworks. A representative collection of sources from the specialized literature was screened for evaluation criteria. Altogether, 74 sources were assessed, comprising 346 mentions of criteria applied for indicator assessment. A detailed synthesis was performed to organize the criteria and identify possible redundancies. The analysis allowed a reduction from the 346 initial criteria to 60 unique criteria. The study offers a standard title and description for each criterion, contributing to improve clarity and avoid ambiguity. The criteria were also ranked to identify which criteria were in more systemic use. Of the 60 criteria found, the 12 most cited were identified as possible core criteria for framework development. Also, in order to facilitate the design of indicator sets, all 60 criteria were divided into two approaches (scientific/top-down or end-use/bottom-up). This study identified significant redundancies and a lack of standardization in the use of criteria, and it also ranked criteria to facilitate multi-method framework development. Thus, it is essential that indicator designers not only consider criteria that have some level of standardization to be able to compare and communicate with other agencies and communities but also consider how to utilize core criteria in the design of indicator sets.

This is a preview of subscription content, access via your institution.

Fig. 1

References

  1. Aveline, A., Rousseau, M. L., Guichard, L., Laurent, M., & Bockstaller, C. (2009). Evaluating an environmental indicator: Case study of MERLIN, a method for assessing the risk of nitrate leaching. Agricultural Systems, 100(1–3), 22–30. https://doi.org/10.1016/j.agsy.2008.12.001.

    Article  Google Scholar 

  2. Baker, D., Bridges, D., Hunter, R., Johnson, G., Krupa, J., & Murphy, J. (2002). Guidebook to decision-making methods. USA: Department of Energy.

    Google Scholar 

  3. Becker, J. (2010). Use of backcasting to integrate indicators with principles of sustainability. International Journal of Sustainable Development and World Ecology, 17(3), 189–197.

    Article  Google Scholar 

  4. Bélanger, V., Vanasse, A., Parent, D., Allard, G., & Pellerin, D. (2012). Development of agri-environmental indicators to assess dairy farm sustainability in Quebec, Eastern Canada. Ecological Indicators, 23, 421–430. https://doi.org/10.1016/j.ecolind.2012.04.027.

    Article  Google Scholar 

  5. Bell, S., & Morse, S. (2003). Learning from experience in sustainability. In Proceedings international sustainable development research conference 2003, 2425 Mar 2003, Nottingham, UK.

  6. BNIA - Baltimore Neighborhood Indicators Alliance. (2006). Vital signs IV: Measuring Baltimore’s progress toward strong neighborhoods and a thriving city (p. 106). Retrieved from http://cdm16352.contentdm.oclc.org/cdm/ref/collection/p15224coll6/id/1744.

  7. Bockstaller, C., & Girardin, P. (2003). How to validate environmental indicators. Agricultural Systems, 76(2), 639–653. https://doi.org/10.1016/S0308-521X(02)00053-7.

    Article  Google Scholar 

  8. Bockstaller, C., Guichard, L., & Makowski, D. (2008). Agri-environmental indicators to assess cropping and farming systems: a review. Agronomy for Sustainable Development, 28(1), 139–149. https://doi.org/10.1007/978-90-481-2666-8_44.

    Article  Google Scholar 

  9. Breslow, S. J., Allen, M., Holstein, D., Sojka, B., Barnea, R., Basurto, X., et al. (2017). Evaluating indicators of human well-being for ecosystem-based management. Ecosystem Health and Sustainability, 3(12), 1–18.

    Article  Google Scholar 

  10. Bringhenti, J. R., Zandonade, E., & Günther, W. M. R. (2011). Selection and validation of indicators for programs selective collection evaluation with social inclusion. Resources, Conservation and Recycling, 55(11), 876–884. https://doi.org/10.1016/j.resconrec.2011.04.010.

    Article  Google Scholar 

  11. Buchholz, T., Luzadis, V. A., & Volk, T. A. (2009). Sustainability criteria for bioenergy systems: Results from an expert survey. Journal of Cleaner Production, 17, S86–S98.

    Article  Google Scholar 

  12. Butler, D., Jowitt, P., Ashley, R., Blackwood, D., Davies, J., & Oltean-Dumbrava, C. (2003). SWARD: Decision support processes for the UK. Management of Environmental Quality, 14, 444–459.

    Article  Google Scholar 

  13. Calliera, M., Marchis, A., Bollmohr, S., Sacchettini, G., Lamastra, L., & Capri, E. (2013). A process to provide harmonised criteria for the selection of indicators for pesticide risk reduction within the framework of the sustainable use directive. Pest Management Science, 69(4), 451–456.

    CAS  Article  Google Scholar 

  14. Castillo, H., & Pitfield, D. E. (2010). ELASTIC—A methodological framework for identifying and selecting sustainable transport indicators. Transportation Research Part D: Transport and Environment, 15(4), 179–188.

    Article  Google Scholar 

  15. CBD - Convention on Biological Diversity. (1999). Development of indicators of biological diversity. Nairobi: Convention on Biological Diversity, subsidiary Body on Scientific, Technical and Technological Advice. Report No. UNEP/CBD/SBSTTA/5/12, 14 pp.

  16. Clark, W., & Dickson, N. (1999). The global environmental assessment project: Learning from efforts to link science and policy in an interdependent world. Acclimations, 8:6–7. Retrieved from http://www.hks.harvard.edu/gea/pubs/99art_wc_geaacc.pdf.

  17. Cloquell-Ballester, V. A., Monterde-Díaz, R., & Santamarina-Siurana, M. C. (2006). Indicators validation for the improvement of environmental and social impact quantitative assessment. Environmental Impact Assessment Review, 26(1), 79–105. https://doi.org/10.1016/j.eiar.2005.06.002.

    Article  Google Scholar 

  18. Dahl, A. L. (2012). Achievements and gaps in indicators for sustainability. Ecological Indicators, 17(2012), 14–19.

    Article  Google Scholar 

  19. Deming, M. E., & Swaffield, S. (2011). Landscape architecture research: Inquiry, strategy, design. Hoboken: Wiley.

    Google Scholar 

  20. Doukas, H. C., Andreas, B. M., & Psarras, J. E. (2007). Multi-criteria decision aid for the formulation of sustainable technological energy priorities using linguistic variables. European Journal of Operational Research, 182, 844–855.

    Article  Google Scholar 

  21. EEA - European Environment Agency. (2005). EEA Core set of indicatorsGuide. European Environment Agency, Copenhagen. Report No. 1/2005, pp. 37.

  22. FAO - Food and Agriculture organization of the United Nations. (1999). Indicators for sustainable development of marine capture fisheries. Rome: Food and Agriculture Organigation (FAO). Retrieved from http://www.fao.org/docrep/fao/004/x3307e/x3307e00.pdf.

  23. Fraser, E. D. G., Dougill, A. J., Mabee, W. E., Reed, M., & McAlpine, P. (2006). Bottom up and top down: Analysis of participatory processes for sustainability indicator identification as a pathway to community empowerment and sustainable environmental management. Journal of Environmental Management, 78, 114–127.

    Article  Google Scholar 

  24. Graymore, M. L. M., Wallis, A. M., & Richards, A. J. (2009). An index of regional sustainability: A GIS-based multiple criteria analysis decision support system for progressing sustainability. Ecological Complexity, 6, 453–462.

    Article  Google Scholar 

  25. GRI - Global Reporting Initiative. (2002). Sustainability reporting guidelines 2002. Retrieved from http://www.globalreporting.org/guidelines/2002/contents.asp. 7 April 2004.

  26. Gudmundsson, H. (2010). Criteria and methods for indicator assessment and validationA review of general and sustainable transport related indicator criteria and how to apply them. Retrieved from http://cost356.inrets.fr/pub/reference/reports/C356_2.2_report_criteria_HG_220410.pdf.

  27. Hák, T., Janoušková, S., & Moldan, B. (2016). Sustainable development goals: A need for relevant indicators. Ecological Indicators, 60, 565–573.

    Article  Google Scholar 

  28. Hák, T., Moldan, B., & Dahl, A. L. (Eds.). (2012). Sustainability indicators: A scientific assessment (Vol. 7). Washington: Island Press.

    Google Scholar 

  29. Hardi, P., & Terrence Z. (Eds.) (1997). Assessing sustainable development: Principles in practice. Winnipeg: International Institute for Sustainable Development, Winnipeg, Saskatchewan. Retrieved from https://www.iisd.org/pdf/bellagio.pdf.

  30. Heink, U., & Kowarik, I. (2010). What are indicators? On the definition of indicators in ecology and environmental planning. Ecological Indicators, 10(3), 584–593. https://doi.org/10.1016/j.ecolind.2009.09.009.

    Article  Google Scholar 

  31. IISD - International Institute for Sustainable Developed. (2008). Bellagio STAMP: Sustainability assessment and measurement principles. Winnipeg: IISD. Retrieved from http://www.iisd.org/pdf/2009/brochure_bellagiostamp.pdf.

  32. ITFM - Intergovernmental Task Force on Monitoring Water Quality. (1995) Water-quality monitoring in the United States. U.S. Geological Survey, Water Information Coordination Program, Washington. http://acwi.gov/appendixes/index.html.

  33. James, C. A., Kershner, J., O’Neill, S., & Levin, P. S. (2012). A methodology for evaluating and ranking water quantity indicators in support of ecosystem-based management. Environmental Management, 49(3), 703–719. https://doi.org/10.1007/s00267-012-9808-7.

    Article  Google Scholar 

  34. Juwana, I., Muttil, N., & Perera, B. J. C. (2012). Indicator-based water sustainability assessment—A review. The Science of the Total Environment, 438, 357–371. https://doi.org/10.1016/j.scitotenv.2012.08.093.

    CAS  Article  Google Scholar 

  35. Khadka, C., & Vacik, H. (2012). Comparing a top-down and bottom-up approach in the identification of criteria and indicators for sustainable community forest management in Nepal. Forestry: An International Journal of Forest Research, 85(1), 145–158.

    Article  Google Scholar 

  36. Klug, H., & Kmoch, A. (2014). Operationalizing environmental indicators for real time multi-purpose decision making and action support. Ecological Modelling, 295, 66–74. https://doi.org/10.1016/j.ecolmodel.2014.04.009.

    Article  Google Scholar 

  37. Kurka, T., & Blackwood, D. (2013). Participatory selection of sustainability criteria and indicators for bioenergy developments. Renewable and Sustainable Energy Reviews, 24, 92–102. https://doi.org/10.1016/j.rser.2013.03.062.

    Article  Google Scholar 

  38. Kurtz, J. C., Jackson, L. E., & Fisher, W. S. (2001). Strategies for evaluating indicators based on guidelines from the environmental protection agency’s office of research and development. Ecological Indicators, 1(1), 49–60.

    CAS  Article  Google Scholar 

  39. Lattimore, B., Smith, C. T., Titus, B. D., Stupak, I., & Egnell, G. (2009). Environmental factors in woodfuel production: opportunities, risks, and criteria and indicators for sustainable practices. Biomass and Bioenergy, 33, 1321–1342.

    Article  Google Scholar 

  40. Lutz, B., & Hans-Dieter, D. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64(1), 45–80.

    Article  Google Scholar 

  41. Lyytimäki, J., Tapio, P., Varho, V., & Söderman, T. (2013). The use, non-use and misuse of indicators in sustainability assessment and communication. International Journal of Sustainable Development and World Ecology, 20(5), 385–393.

    Article  Google Scholar 

  42. Mascarenhas, A., Nunes, L. M., & Ramos, T. B. (2015). Selection of sustainability indicators for planning combining stakeholders’ participation and data reduction techniques. Journal of Cleaner Production, 92, 295–307.

    Article  Google Scholar 

  43. McCool, S. F., & Stankey, G. H. (2004). Indicators of sustainability: Challenges and opportunities at the interface of science and policy. Environmental Management, 33(3), 294–305. https://doi.org/10.1007/s00267-003-0084-4.

    Article  Google Scholar 

  44. Mendoza, G. A., & Prabhu, R. (2000). Development of a methodology for selecting criteria and indicators of sustainable forest management: A case study on participatory assessment. Environmental Management, 26(6), 659–673.

    CAS  Article  Google Scholar 

  45. Meul, M., Nevens, F., & Reheul, D. (2009). Validating sustainability indicators: Focus on ecological aspects of Flemish dairy farms. Ecological Indicators, 9(2), 284–295. https://doi.org/10.1016/j.ecolind.2008.05.007.

    Article  Google Scholar 

  46. Mitchell, G., May, A., & McDonald, A. (1995). PICABUE: A methodological framework for the development of indicators of sustainable development. The International Journal of Sustainable Development and World Ecology, 2(2), 104–123.

    Article  Google Scholar 

  47. NCHOD. (2005). Compendium of clinical and health indicators user guide. National Centre for Health Outcomes Development (NCHOD), London site—London School of Hygiene and Tropical Medicine.

  48. Nicholson, E., Collen, B., Barausse, A., Blanchard, J. L., Costelloe, B. T., Sullivan, K. M. E., et al. (2012). Making robust policy decisions using global biodiversity indicators. PLoS ONE. https://doi.org/10.1371/journal.pone.0041128.

    Article  Google Scholar 

  49. Niemeijer, D., & de Groot, R. S. (2008). A conceptual framework for selecting environmental indicator sets. Ecological Indicators, 8(1), 14–25. https://doi.org/10.1016/j.ecolind.2006.11.012.

    Article  Google Scholar 

  50. OECD - Organization for Economic Co-operation and Development. (2003). OECD environmental indicators: Development, measurement and use, Reference paper. Paris: OECD. Retrieved form http://www.oecd.org/dataoecd/7/47/24993546.pdf.

  51. Olsthoorn, X., Tyteca, D., Wehrmeyer, W., & Wagner, M. (2001). Environmental indicators for business: A review of the literature and standardisation methods. Journal of Cleaner Production, 9, 453–463.

    Article  Google Scholar 

  52. Parris, T. M., & Kates, R. W. (2003). Characterizing and measuring sustainable development. Annual Review of Environment and Resources, 28(1), 559–586. https://doi.org/10.1146/annurev.energy.28.050302.105551.

    Article  Google Scholar 

  53. Prescott-Allen, R. (2001). The wellbeing of nations. Washington DC: Island Press.

    Google Scholar 

  54. Radicchi, F., & Castellano, C. (2012). Testing the fairness of citation indicators for comparison across scientific domains: The case of fractional citation counts. Journal of Informetrics, 6(1), 121–130. https://doi.org/10.1016/j.joi.2011.09.002.

    Article  Google Scholar 

  55. Reed, M., Fraser, E. D., Morse, S., & Dougill, A. J. (2005). Integrating methods for developing sustainability indicators to facilitate learning and action. Ecology and society, 10(1). http://www.ecologyandsociety.org/vol10/iss1/resp3/.

  56. Rovere, E. L. L., Soares, J. B., Oliveira, L. B., & Lauria, T. (2010). Sustainable expansion of electricity sector: Sustainability indicators as an instrument to support decision making. Renewable and Sustainable Energy Reviews, 14, 422–429.

    Article  Google Scholar 

  57. Segnestam, L. (2002). Indicators of environment and sustainable development: Theories and practical experience. Washington: World Bank. Retrieved from http://siteresources.worldbank.org/INTEEI/936217-1115801208804/20486265/IndicatorsofEnvironmentandSustainableDevelopment2003.

  58. Shmelev, S. E., & Rodríguez-Labajos, B. (2009). Dynamic multidimensional assessment of sustainability at the macro level: The case of Austria. Ecological Economics, 68, 2560–2573.

    Article  Google Scholar 

  59. Singh, R. K., Murty, H. R., Gupta, S. K., & Dikshit, A. K. (2009). An overview of sustainability assessment methodologies. Ecological Indicators, 9(2), 189–212. https://doi.org/10.1016/j.ecolind.2008.05.011.

    Article  Google Scholar 

  60. SNZ - Statistics New Zealand. (2002) Socio-economic indicators for the environment. Environmental Statistics Team, Christchurch. http://www2.stats.govt.nz/domino/external/web/prod_serv.nsf/092edeb76ed5aa6bcc256afe0081d84e/94b7f3198c9d9111cc256c1500171ea6?OpenDocument.

  61. Spangenberg, J. H. (2008). Second order governance: Learning processes to identify indicators. Corporate Social Responsibility and Environmental Management, 15(3), 125–139. https://doi.org/10.1002/csr.137.

    Article  Google Scholar 

  62. Spreng, D., & Wils, A. (2000). Indicators of sustainability: Indicators in various scientific disciplines. AGS Report: Alliance for Global Sustainability.

    Google Scholar 

  63. Tanguay, G. A., Rajaonson, J., & Therrien, M. C. (2013). Sustainable tourism indicators: Selection criteria for policy implementation and scientific recognition. Journal of Sustainable Tourism, 21(6), 862–879.

    Article  Google Scholar 

  64. UN - United Nations. (2007). Indicators of sustainable development : Guidelines and methodologies. New York: United Nations. Retrieved from http://www.un.org/esa/sustdev/natlinfo/indicators/guidelines.pdf.

  65. UNEP - United Nations Environment Programme. (2006). Environmental indicators for North America. Nairobi: UNEP. Retrieved from http://www.unep.org/pdf/NA_Indicators_FullVersion.pdf.

  66. US EPA - United States Environmental Protection Agency. (2000). Evaluation guidelines for ecological indicators. Washington. Retrieved from http://www.epa.gov/emap/html/pubs/docs/resdocs/ecol_ind.pdf.

  67. US GAO - United States Government Accountability Office. (2004). Environmental indicators: Better coordination is needed to develop environmental indicators sets that inform decisions. Report to Congressional requesters. US Government Accountability Office

  68. Vera, I., & Langlois, L. (2007). Energy indicators for sustainable development. Energy, 32, 875–882.

    Article  Google Scholar 

  69. Wang, J., Jing, Y., Zhang, C., & Zhao, J. (2009). Review on multi-criteria decision analysis aid in sustainable energy decision-making. Renewable and Sustainable Energy Reviews, 13, 2263–2278.

    Article  Google Scholar 

  70. WHO - World Health Organization. (2002). Health in sustainable development planning: The role of indicators (p. 42). Geneva.

  71. World Bank. (2000). Developing indicators: Lessons learned from Central America (p. 22). Washington. Retrieved from http://siteresources.worldbank.org/INTEEI/811099-1115809852605/20486445/DevelopingIndicatorsLessonsLearnedFromCentralAmerica2001part1.pdf.

  72. WWAP - World Water Assessment Programme. (2003). The United Nations world water development report: Water for people water for life. Paris: UNESCO, and London: Earthscan.

  73. WWAP - World Water Assessment Programme. (2006). The United Nations World Water Development Report 2: Water - A Shared Responsibility. Paris: UNESCO, and London: Earthscan.

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to A. Pires.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1: Scientific criteria to assess indicators

Citations* Name and definition of the criteria Sources
19 Scientific foundation—“The extent to which an indicator is based on currently sound and internationally accepted theoretical, conceptual, technical, and scientific standards and principles (adapted from UNEP 2006) Niemeijer and de Groot (2008) (4x)**, WWAP (2006), Bockstaller and Girardin (2003), Bockstaller et al. (2008), Aveline et al. (2009), Cloquell-Ballester et al. (2006), Parris and Kates (2003), Clark and Dickson (1999), UNEP (2006), SNZ (2002), UN (2007), BNIA (2006), WHO (2002), OECD (2003), FAO (1999), World Bank (2000)
18 Reliability—“The extent to which an experiment, test, or measuring procedure yields the same results on repeated trials” (Webster’s Dictionary) Niemeijer and de Groot (2008) (2x)**, Bringhenti et al. (2011), Cloquell-Ballester et al. (2006), UNEP (2006), SNZ (2002), ITFM (1995), BNIA (2006), Kurka and Blackwood (2013), Rovere et al. (2010), Graymore et al. (2009), Buchholz et al. (2009), Singh et al. (2009), Wang et al. (2009), Bélanger et al. (2012), Segnestam (2002), OECD (2003), World Bank (2000), Gudmundsson (2010)
15 Measurability—“The extent to which the proposed measurement procedures to obtain the indicator adopts standardized methods” (adapted from Cloquell-Ballester et al. 2006) Niemeijer and de Groot (2008) (4x)**, WWAP (2006), IISD (2008), Cloquell-Ballester et al. (2006), Prescott-Allen (2001), USEPA (2000), UNEP (2006), SNZ (2002), ITFM (1995), BNIA (2006), WHO (2002), World Bank (2000)
13 Sensitivity—“The extent to which a small change in the factor measured should result in a measurable change in the indicator” (adapted from WWAP 2006) Niemeijer and de Groot (2008), WWAP (2006), Aveline et al. (2009), Bockstaller et al.(2008), IISD (2008), Cloquell-Ballester et al. (2006), SNZ (2002), ITFM (1995), Bélanger et al. (2012), WHO (2002), OECD (2003), World Bank. (2000), Gudmundsson (2010)
10 Accuracy—“The extend to which the result of a measurement, or of an indicator conforms to the correct value” (adapted Oxford Dictionary of English 2014) Bockstaller and Girardin (2003), Bockstaller et al. (2008), Aveline et al. (2009), ITFM (1995), Bringhenti et al. (2011), Cloquell-Ballester et al. (2006), Meul et al. (2009), OECD (2003), FAO (1999), Gudmundsson (2010)
9 Specificity—“Clearly and unambiguously defined” (Niemeijer and de Groot 2008) Niemeijer and de Groot (2008) (4x)**, WWAP (2006), UNEP (2006), SNZ (2002), Segnestam (2002), World Bank. (2000)
8 Time-bound—“Measure changes on an appropriate temporal scale” (SNZ 2002) Niemeijer and de Groot (2008) (4x)**, IISD (2008), US EPA (2000), SNZ (2002), ITFM (1995)
8 Representativeness—”Related to a specific question or issue of concern and representative of the conditions in question” (WHO (2002)) Niemeijer and de Groot (2008), WWAP (2006), Bringhenti et al. (2011), ITFM (1995), WHO (2002), Prescott-Allen 2001, SNZ (2002), World Bank (2000)
6 Data quality—“The data used to establish the indicator are adequately documented and of known quality” (adapted from OECD 2003) US EPA (2000), Segnestam (2002), OECD (2003), FAO (1999), World Bank (2000), Cloquell-Ballester et al. (2006)
5 Space-bound—“Adopt an appropriate geographical scope” (IISD 2008) Niemeijer and de Groot (2008), IISD (2008), US EPA (2000), SNZ (2002), ITFM (1995)
5 Anticipatory—“Provides an early warning of changes” (ITFM 1995) Niemeijer and de Groot (2008), WWAP (2006), ITFM (1995), WHO (2002), World Bank (2000)
5 Spatial and temporal scales of applicability—“Provide information at the appropriate spatial and temporal scales” (Niemeijer and de Groot 2008) Niemeijer and de Groot (2008) (2x)**, Segnestam (2002), ITFM (1995), World Bank (2000)
4 Robustness—“Be relatively insensitive to expected sources of interference” (Niemeijer and de Groot 2008) Niemeijer and de Groot (2008), WWAP (2006), WHO (2002), FAO (1999)
2 Predictability—“Respond in a predictable manner to changes and stresses” (Niemeijer and de Groot 2008) Niemeijer and de Groot (2008), SNZ (2002)
2 Universality—“Applicable to many areas situations and scales” (Niemeijer and de Groot 2008) Niemeijer and de Groot (2008), WWAP (2006)
2 Discriminatory—“Ability to discriminate differences separating extraneous variability (US EPA, 2000) US EPA (2000), ITFM (1995)
2 Uncertainty—“Detailed with regards to uncertainties and limitations” (WWAP (2006)) Niemeijer and de Groot (2008), WWAP (2006)
1 Portability—“Be repeatable and reproducible in different contexts” Niemeijer and de Groot (2008)
1 Specific for a certain stress or effect” WWAP (2006)
1 General importance—Beer on a fundaments process or widespread change” Niemeijer and de Groot (2008)
1 Formulation—The mathematical formulation of the indicator is suitable with regard to the concept which is to be quantified” Cloquell-Ballester et al. (2006)
1 Transformable—intelligent” WWAP (2006)
1 Estimation of measurement error—must be estimated and reported” US EPA (2000)
1 Integrates effects/exposure—Integrates effects or exposure over time and space” ITFM (1995)
1 Focus on causes not symptoms” BNIA (2006)
  1. Total: 141 mentions/24 scientific criteria
  2. *Number of mentions of the criterion under analyses
  3. ** Number of citations on the meta-review performed by Niemeijer and de Groot (2008)

Appendix 2: End-use criteria to assess indicators

Citations* Name and definition of the criteria Sources
31 Data availability—“The extent which the data required for the indicator is easy or possible to get at a reasonable cost” (adapted from Merriam-Webster Dictionary of English 2014 and OECD 2003) Niemeijer and de Groot (2008) (3x)**, WWAP (2006), Bringhenti et al. (2011), BNIA (2006), Cloquell-Ballester et al. (2006), Prescott-Allen 2001, UNEP (2006), SNZ (2002), Segnestam (2002), ITFM (1995), UN (2007), Kurka and Blackwood (2013), Lattimore et al. (2009), Vera and Langlois (2007), Fraser et al. (2006), Olsthoorn et al. 2001, Rovere et al. (2010), Shmelev and Rodríguez-Labajos (2009), Graymore et al. (2009), Buchholz et al. (2009), Singh et al. (2009), Wang et al. (2009), Butler et al. (2003), Bélanger et al. (2012), WHO (2002), OECD (2003), FAO (1999), World Bank (2000), Gudmundsson (2010)
25 Relevance—“The extent which an indicator is related or connected to the matter in hand” (adapted from Merriam-Webster Dictionary of English 2014) Niemeijer and de Groot (2008) (4x)**, Cloquell-Ballester et al. (2006), Parris and Kates (2003), Clark and Dickson (1999), US EPA (2000), UNEP (2006), SNZ (2002), ITFM (1995), UN (2007), BNIA (2006), Kurka and Blackwood (2013), Lattimore et al. (2009), Rovere et al. (2010): Graymore et al. (2009), Gilmour et al. (2007), Buchholz et al. (2009), Wang et al. (2009), Doukas et al. (2007), Baker et al. (2002), Butler et al. (2003), WHO (2002), World Bank (2000)
21 Comprehensibility—“The extent which the indicator is able to be understood by the target audience” (adapted from Oxford Dictionary of English 2014) Niemeijer and de Groot (2008) (2x)**, WWAP (2006), Aveline et al. (2009), Bockstaller et al.(2008), IISD (2008), Cloquell-Ballester et al. (2006), UNEP (2006), SNZ (2002), ITFM (1995), UN (2007), BNIA (2006), Kurka and Blackwood (2013), Fraser et al. (2006), Singh et al. (2009), Butler et al. (2003), Bélanger et al. (2012), WHO (2002), OECD (2003), FAO (1999), Gudmundsson (2010)
13 Usefulness—“User-driven to be relevant to target audience” (Niemeijer and de Groot 2008) Niemeijer and de Groot (2008), WWAP (2006), Bockstaller and Girardin (2003), Bockstaller et al. (2008), Aveline et al. (2009), IISD (2008), Meul et al. (2009), Segnestam (2002), BNIA (2006), Bélanger et al. 2012, WHO (2002), FAO (1999), World Bank (2000)
10 Target-oriented—“Have a threshold and/or target against which to compare the indicator” (adapted from OECD 2003) Niemeijer and de Groot (2008), WWAP (2006), IISD (2008), US EPA (2000), UNEP (2006), SNZ (2002), ITFM (1995), BNIA (2006), OECD (2003), Gudmundsson (2010)
10 Operational simplicity “Simple to measure, manage and analyze” (Niemeijer and de Groot 2008) Niemeijer and de Groot (2008) (2x)**, Aveline et al. (2009), Bockstaller et al. (2008), SNZ (2002), ITFM (1995), UN (2007), FAO (1999), World Bank (2000), Gudmundsson (2010)
10 Compatibility—“Be compatible with indicators developed and used in other regions” (Niemeijer and de Groot 2008) Niemeijer and de Groot (2008) (2x)**, WWAP (2006), UNEP (2006), UN (2007), Kurka and Blackwood (2013), Rovere et al. (2010), Wang et al. (2009), Doukas et al. (2007), OECD (2003)
9 Linkage to management action—“Provide information to support a management decision or to quantify the success of past decisions” (US EPA, 2000) Niemeijer and de Groot (2008) (3x)**, WWAP (2006), US EPA (2000), UNEP (2006), SNZ (2002), FAO (1999), Gudmundsson (2010)
9 Retrospectivity—“Able to show trends over time” (OECD 2003) Niemeijer and de Groot (2008) (2x)**, WWAP (2006), SNZ (2002), IISD (2008), ITFM (1995), OECD (2003), FAO (1999), FAO (2000)
9 Resource demand—“Logistical requirements (personnel, equipment, training) are reasonable” (US EPA 2000) Niemeijer and de Groot (2008) (5x)**, IISD (2008), US EPA (2000), UNEP (2006), UN (2007)
7 Sustainability—“Consider the underlying social, economic and environmental system as a whole, including issues related to governance and the interactions among its components” (IISD 2008) Niemeijer and de Groot (2008), IISD (2008), SNZ (2002), UN (2007), BNIA (2006), Bélanger et al. 2012, WHO (2002)
7 Cost-effectiveness—“Benefits of the information provided by the indicators should outweigh the cost of usage” (Niemeijer and de Groot 2008) Niemeijer and de Groot (2008), Cloquell-Ballester et al. (2006), US EPA (2000), Segnestam (2002), ITFM (1995), FAO (1999), World Bank (2000)
5 Participatory—“Developed with the palpation of a broad range of stakeholders to ensure the indicators: encompass community visions and values, and promote ownership” (UNEP 2006) IISD (2008), Parris and Kates (2003), Clark and Dickson (1999), WWAP (2006), UNEP (2006)
5 Causal links—“Cause–effect chain has to be known to enable tackling of the problem” (WWAP 2006) WWAP (2006), IISD (2008), SNZ (2002), OECD (2003), World Bank (2000)
5 Individuality—“Are the indicators independent enough or do they duplicate other C and I?’ (Kurka and Blackwood 2013) Kurka and Blackwood (2013), Rovere et al. (2010), Graymore et al. (2009), Wang et al. (2009), Doukas et al. (2007)
3 Transparency—“ensure it is accessible to the public; explain the underlined choices assumptions and uncertainties; disclose data sources and methods and disclose all sources of funding and potential conflicts of interest.” (IISD 2008) IISD (2008), Gudmundsson (2010), UNEP (2006)
3 Flexibility—“Are flexible, so new information can lead to adjustments in the indicator” (UNEP 2006) Aveline et al. (2009), Bockstaller et al. (2008), UNEP (2006)
2 Linked to models, forecasting and information systems WWAP (2006), OECD (2003)
2 Conceptual framework—“Be developed within an agreed-upon conceptual and operational framework” (WWAP 2006) IISD (2008), WWAP (2006)
2 National in scope—“Be either national in scope or applicable to regional environmental issues of national significance” (OECD 2003) UN (2007), OECD (2003)
2 Ecological function—“Conceptually linked to ecological function of concern” (USEPA 2000) US EPA (2000), Segnestam (2002)
2 Pedagogy—Educational am “helps to make the factors understandable to the pubic” (Aveline et al. 2009) Aveline et al. (2009), Bockstaller et al. (2008)
1 Quantified—“Information should be quantified in such a way that it is significant apparent” Niemeijer and de Groot (2008)
1 Policy Changes—“Recording ether changes in the means recommended by policy or changes in the development impact attributable to policy “ WWAP (2006)
1 Guiding vision—be guided by the goal of delivering well being within the capacity of the biosphere to sustain it for future generations IISD (2008)
1 Boundaries—Takes into consideration risks uncertainties and activities that can have an impact across boundaries IISD (2008)
1 Continuous learning and improvement IISD (2008)
1 Ethical concerns—An indicator must comply with fundamental human rights and must require only data that are consistent with moral beliefs or values of the population Gudmundsson (2010)
1 Information management—requirements for data analysis storage, processing documentation and retrieval are feasible US EPA (2000)
1 Quality assurance—degree of validity of the steps in collation and computation of data aiming to assure the quality of the indicator US EPA (2000)
1 Program coverage—Program uses a suite of indicators that encompass major components of the ecosystem over the range of environmental conditions that can be expected ITFM (1995)
1 Relate to the whole community BNIA (2006)
1 Focus on resources and assets (framed in a positive way/focus on problems or assets) BNIA (2006)
1 Adapted to the objectives—Does the indicator meet the objectives? Bélanger et al. (2012)
1 Formal (legal) foundation FAO (1999)
  1. Total: 205 mentions/25 end-use criteria
  2. *Number of mentions of the criterion under analyses
  3. **Number of citations on the meta-review performed by Niemeijer and de Groot (2008)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Pires, A., Morato, J., Peixoto, H. et al. Synthesizing and standardizing criteria for the evaluation of sustainability indicators in the water sector. Environ Dev Sustain 22, 6671–6689 (2020). https://doi.org/10.1007/s10668-019-00508-z

Download citation

Keywords

  • Indicator
  • Criteria
  • Selection
  • Top-down and bottom-up approach
  • Synthesis
  • Standardization