Synthesizing and standardizing criteria for the evaluation of sustainability indicators in the water sector

  • A. PiresEmail author
  • J. Morato
  • H. Peixoto
  • S. Bradley
  • A. Muller


Indicators are one of the tools available in planning and management projects that aid in the decision-making process and the monitoring of those decisions on the path to sustainable use and management of water and natural resources. However, the quality and trustworthiness of the indicators depend on the constant improvement in the means to assess and design criteria sets. The identification of criteria to evaluate indicators and its subsequence selection are not an ordinary task. The research identified a proliferation of unconsolidated criteria in use in the sustainability and water resource management domains. In response, a process of synthesis and consolidation was undertaken in order to reduce the level of redundancies and to identify possible candidates for “core criteria” that are identified as being a relevant part of most evaluation frameworks. A representative collection of sources from the specialized literature was screened for evaluation criteria. Altogether, 74 sources were assessed, comprising 346 mentions of criteria applied for indicator assessment. A detailed synthesis was performed to organize the criteria and identify possible redundancies. The analysis allowed a reduction from the 346 initial criteria to 60 unique criteria. The study offers a standard title and description for each criterion, contributing to improve clarity and avoid ambiguity. The criteria were also ranked to identify which criteria were in more systemic use. Of the 60 criteria found, the 12 most cited were identified as possible core criteria for framework development. Also, in order to facilitate the design of indicator sets, all 60 criteria were divided into two approaches (scientific/top-down or end-use/bottom-up). This study identified significant redundancies and a lack of standardization in the use of criteria, and it also ranked criteria to facilitate multi-method framework development. Thus, it is essential that indicator designers not only consider criteria that have some level of standardization to be able to compare and communicate with other agencies and communities but also consider how to utilize core criteria in the design of indicator sets.


Indicator Criteria Selection Top-down and bottom-up approach Synthesis Standardization 



  1. Aveline, A., Rousseau, M. L., Guichard, L., Laurent, M., & Bockstaller, C. (2009). Evaluating an environmental indicator: Case study of MERLIN, a method for assessing the risk of nitrate leaching. Agricultural Systems, 100(1–3), 22–30. Scholar
  2. Baker, D., Bridges, D., Hunter, R., Johnson, G., Krupa, J., & Murphy, J. (2002). Guidebook to decision-making methods. USA: Department of Energy.Google Scholar
  3. Becker, J. (2010). Use of backcasting to integrate indicators with principles of sustainability. International Journal of Sustainable Development and World Ecology, 17(3), 189–197.CrossRefGoogle Scholar
  4. Bélanger, V., Vanasse, A., Parent, D., Allard, G., & Pellerin, D. (2012). Development of agri-environmental indicators to assess dairy farm sustainability in Quebec, Eastern Canada. Ecological Indicators, 23, 421–430. Scholar
  5. Bell, S., & Morse, S. (2003). Learning from experience in sustainability. In Proceedings international sustainable development research conference 2003, 2425 Mar 2003, Nottingham, UK.Google Scholar
  6. BNIA - Baltimore Neighborhood Indicators Alliance. (2006). Vital signs IV: Measuring Baltimore’s progress toward strong neighborhoods and a thriving city (p. 106). Retrieved from
  7. Bockstaller, C., & Girardin, P. (2003). How to validate environmental indicators. Agricultural Systems, 76(2), 639–653. Scholar
  8. Bockstaller, C., Guichard, L., & Makowski, D. (2008). Agri-environmental indicators to assess cropping and farming systems: a review. Agronomy for Sustainable Development, 28(1), 139–149. Scholar
  9. Breslow, S. J., Allen, M., Holstein, D., Sojka, B., Barnea, R., Basurto, X., et al. (2017). Evaluating indicators of human well-being for ecosystem-based management. Ecosystem Health and Sustainability, 3(12), 1–18.CrossRefGoogle Scholar
  10. Bringhenti, J. R., Zandonade, E., & Günther, W. M. R. (2011). Selection and validation of indicators for programs selective collection evaluation with social inclusion. Resources, Conservation and Recycling, 55(11), 876–884. Scholar
  11. Buchholz, T., Luzadis, V. A., & Volk, T. A. (2009). Sustainability criteria for bioenergy systems: Results from an expert survey. Journal of Cleaner Production, 17, S86–S98.CrossRefGoogle Scholar
  12. Butler, D., Jowitt, P., Ashley, R., Blackwood, D., Davies, J., & Oltean-Dumbrava, C. (2003). SWARD: Decision support processes for the UK. Management of Environmental Quality, 14, 444–459.CrossRefGoogle Scholar
  13. Calliera, M., Marchis, A., Bollmohr, S., Sacchettini, G., Lamastra, L., & Capri, E. (2013). A process to provide harmonised criteria for the selection of indicators for pesticide risk reduction within the framework of the sustainable use directive. Pest Management Science, 69(4), 451–456.CrossRefGoogle Scholar
  14. Castillo, H., & Pitfield, D. E. (2010). ELASTIC—A methodological framework for identifying and selecting sustainable transport indicators. Transportation Research Part D: Transport and Environment, 15(4), 179–188.CrossRefGoogle Scholar
  15. CBD - Convention on Biological Diversity. (1999). Development of indicators of biological diversity. Nairobi: Convention on Biological Diversity, subsidiary Body on Scientific, Technical and Technological Advice. Report No. UNEP/CBD/SBSTTA/5/12, 14 pp.Google Scholar
  16. Clark, W., & Dickson, N. (1999). The global environmental assessment project: Learning from efforts to link science and policy in an interdependent world. Acclimations, 8:6–7. Retrieved from
  17. Cloquell-Ballester, V. A., Monterde-Díaz, R., & Santamarina-Siurana, M. C. (2006). Indicators validation for the improvement of environmental and social impact quantitative assessment. Environmental Impact Assessment Review, 26(1), 79–105. Scholar
  18. Dahl, A. L. (2012). Achievements and gaps in indicators for sustainability. Ecological Indicators, 17(2012), 14–19.CrossRefGoogle Scholar
  19. Deming, M. E., & Swaffield, S. (2011). Landscape architecture research: Inquiry, strategy, design. Hoboken: Wiley.Google Scholar
  20. Doukas, H. C., Andreas, B. M., & Psarras, J. E. (2007). Multi-criteria decision aid for the formulation of sustainable technological energy priorities using linguistic variables. European Journal of Operational Research, 182, 844–855.CrossRefGoogle Scholar
  21. EEA - European Environment Agency. (2005). EEA Core set of indicatorsGuide. European Environment Agency, Copenhagen. Report No. 1/2005, pp. 37.Google Scholar
  22. FAO - Food and Agriculture organization of the United Nations. (1999). Indicators for sustainable development of marine capture fisheries. Rome: Food and Agriculture Organigation (FAO). Retrieved from
  23. Fraser, E. D. G., Dougill, A. J., Mabee, W. E., Reed, M., & McAlpine, P. (2006). Bottom up and top down: Analysis of participatory processes for sustainability indicator identification as a pathway to community empowerment and sustainable environmental management. Journal of Environmental Management, 78, 114–127.CrossRefGoogle Scholar
  24. Graymore, M. L. M., Wallis, A. M., & Richards, A. J. (2009). An index of regional sustainability: A GIS-based multiple criteria analysis decision support system for progressing sustainability. Ecological Complexity, 6, 453–462.CrossRefGoogle Scholar
  25. GRI - Global Reporting Initiative. (2002). Sustainability reporting guidelines 2002. Retrieved from 7 April 2004.
  26. Gudmundsson, H. (2010). Criteria and methods for indicator assessment and validationA review of general and sustainable transport related indicator criteria and how to apply them. Retrieved from
  27. Hák, T., Janoušková, S., & Moldan, B. (2016). Sustainable development goals: A need for relevant indicators. Ecological Indicators, 60, 565–573.CrossRefGoogle Scholar
  28. Hák, T., Moldan, B., & Dahl, A. L. (Eds.). (2012). Sustainability indicators: A scientific assessment (Vol. 7). Washington: Island Press.Google Scholar
  29. Hardi, P., & Terrence Z. (Eds.) (1997). Assessing sustainable development: Principles in practice. Winnipeg: International Institute for Sustainable Development, Winnipeg, Saskatchewan. Retrieved from
  30. Heink, U., & Kowarik, I. (2010). What are indicators? On the definition of indicators in ecology and environmental planning. Ecological Indicators, 10(3), 584–593. Scholar
  31. IISD - International Institute for Sustainable Developed. (2008). Bellagio STAMP: Sustainability assessment and measurement principles. Winnipeg: IISD. Retrieved from
  32. ITFM - Intergovernmental Task Force on Monitoring Water Quality. (1995) Water-quality monitoring in the United States. U.S. Geological Survey, Water Information Coordination Program, Washington.
  33. James, C. A., Kershner, J., O’Neill, S., & Levin, P. S. (2012). A methodology for evaluating and ranking water quantity indicators in support of ecosystem-based management. Environmental Management, 49(3), 703–719. Scholar
  34. Juwana, I., Muttil, N., & Perera, B. J. C. (2012). Indicator-based water sustainability assessment—A review. The Science of the Total Environment, 438, 357–371. Scholar
  35. Khadka, C., & Vacik, H. (2012). Comparing a top-down and bottom-up approach in the identification of criteria and indicators for sustainable community forest management in Nepal. Forestry: An International Journal of Forest Research, 85(1), 145–158.CrossRefGoogle Scholar
  36. Klug, H., & Kmoch, A. (2014). Operationalizing environmental indicators for real time multi-purpose decision making and action support. Ecological Modelling, 295, 66–74. Scholar
  37. Kurka, T., & Blackwood, D. (2013). Participatory selection of sustainability criteria and indicators for bioenergy developments. Renewable and Sustainable Energy Reviews, 24, 92–102. Scholar
  38. Kurtz, J. C., Jackson, L. E., & Fisher, W. S. (2001). Strategies for evaluating indicators based on guidelines from the environmental protection agency’s office of research and development. Ecological Indicators, 1(1), 49–60.CrossRefGoogle Scholar
  39. Lattimore, B., Smith, C. T., Titus, B. D., Stupak, I., & Egnell, G. (2009). Environmental factors in woodfuel production: opportunities, risks, and criteria and indicators for sustainable practices. Biomass and Bioenergy, 33, 1321–1342.CrossRefGoogle Scholar
  40. Lutz, B., & Hans-Dieter, D. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64(1), 45–80.CrossRefGoogle Scholar
  41. Lyytimäki, J., Tapio, P., Varho, V., & Söderman, T. (2013). The use, non-use and misuse of indicators in sustainability assessment and communication. International Journal of Sustainable Development and World Ecology, 20(5), 385–393.CrossRefGoogle Scholar
  42. Mascarenhas, A., Nunes, L. M., & Ramos, T. B. (2015). Selection of sustainability indicators for planning combining stakeholders’ participation and data reduction techniques. Journal of Cleaner Production, 92, 295–307.CrossRefGoogle Scholar
  43. McCool, S. F., & Stankey, G. H. (2004). Indicators of sustainability: Challenges and opportunities at the interface of science and policy. Environmental Management, 33(3), 294–305. Scholar
  44. Mendoza, G. A., & Prabhu, R. (2000). Development of a methodology for selecting criteria and indicators of sustainable forest management: A case study on participatory assessment. Environmental Management, 26(6), 659–673.CrossRefGoogle Scholar
  45. Meul, M., Nevens, F., & Reheul, D. (2009). Validating sustainability indicators: Focus on ecological aspects of Flemish dairy farms. Ecological Indicators, 9(2), 284–295. Scholar
  46. Mitchell, G., May, A., & McDonald, A. (1995). PICABUE: A methodological framework for the development of indicators of sustainable development. The International Journal of Sustainable Development and World Ecology, 2(2), 104–123.CrossRefGoogle Scholar
  47. NCHOD. (2005). Compendium of clinical and health indicators user guide. National Centre for Health Outcomes Development (NCHOD), London site—London School of Hygiene and Tropical Medicine.Google Scholar
  48. Nicholson, E., Collen, B., Barausse, A., Blanchard, J. L., Costelloe, B. T., Sullivan, K. M. E., et al. (2012). Making robust policy decisions using global biodiversity indicators. PLoS ONE. Scholar
  49. Niemeijer, D., & de Groot, R. S. (2008). A conceptual framework for selecting environmental indicator sets. Ecological Indicators, 8(1), 14–25. Scholar
  50. OECD - Organization for Economic Co-operation and Development. (2003). OECD environmental indicators: Development, measurement and use, Reference paper. Paris: OECD. Retrieved form
  51. Olsthoorn, X., Tyteca, D., Wehrmeyer, W., & Wagner, M. (2001). Environmental indicators for business: A review of the literature and standardisation methods. Journal of Cleaner Production, 9, 453–463.CrossRefGoogle Scholar
  52. Parris, T. M., & Kates, R. W. (2003). Characterizing and measuring sustainable development. Annual Review of Environment and Resources, 28(1), 559–586. Scholar
  53. Prescott-Allen, R. (2001). The wellbeing of nations. Washington DC: Island Press.Google Scholar
  54. Radicchi, F., & Castellano, C. (2012). Testing the fairness of citation indicators for comparison across scientific domains: The case of fractional citation counts. Journal of Informetrics, 6(1), 121–130. Scholar
  55. Reed, M., Fraser, E. D., Morse, S., & Dougill, A. J. (2005). Integrating methods for developing sustainability indicators to facilitate learning and action. Ecology and society, 10(1).
  56. Rovere, E. L. L., Soares, J. B., Oliveira, L. B., & Lauria, T. (2010). Sustainable expansion of electricity sector: Sustainability indicators as an instrument to support decision making. Renewable and Sustainable Energy Reviews, 14, 422–429.CrossRefGoogle Scholar
  57. Segnestam, L. (2002). Indicators of environment and sustainable development: Theories and practical experience. Washington: World Bank. Retrieved from
  58. Shmelev, S. E., & Rodríguez-Labajos, B. (2009). Dynamic multidimensional assessment of sustainability at the macro level: The case of Austria. Ecological Economics, 68, 2560–2573.CrossRefGoogle Scholar
  59. Singh, R. K., Murty, H. R., Gupta, S. K., & Dikshit, A. K. (2009). An overview of sustainability assessment methodologies. Ecological Indicators, 9(2), 189–212. Scholar
  60. SNZ - Statistics New Zealand. (2002) Socio-economic indicators for the environment. Environmental Statistics Team, Christchurch.
  61. Spangenberg, J. H. (2008). Second order governance: Learning processes to identify indicators. Corporate Social Responsibility and Environmental Management, 15(3), 125–139. Scholar
  62. Spreng, D., & Wils, A. (2000). Indicators of sustainability: Indicators in various scientific disciplines. AGS Report: Alliance for Global Sustainability.Google Scholar
  63. Tanguay, G. A., Rajaonson, J., & Therrien, M. C. (2013). Sustainable tourism indicators: Selection criteria for policy implementation and scientific recognition. Journal of Sustainable Tourism, 21(6), 862–879.CrossRefGoogle Scholar
  64. UN - United Nations. (2007). Indicators of sustainable development : Guidelines and methodologies. New York: United Nations. Retrieved from
  65. UNEP - United Nations Environment Programme. (2006). Environmental indicators for North America. Nairobi: UNEP. Retrieved from
  66. US EPA - United States Environmental Protection Agency. (2000). Evaluation guidelines for ecological indicators. Washington. Retrieved from
  67. US GAO - United States Government Accountability Office. (2004). Environmental indicators: Better coordination is needed to develop environmental indicators sets that inform decisions. Report to Congressional requesters. US Government Accountability OfficeGoogle Scholar
  68. Vera, I., & Langlois, L. (2007). Energy indicators for sustainable development. Energy, 32, 875–882.CrossRefGoogle Scholar
  69. Wang, J., Jing, Y., Zhang, C., & Zhao, J. (2009). Review on multi-criteria decision analysis aid in sustainable energy decision-making. Renewable and Sustainable Energy Reviews, 13, 2263–2278.CrossRefGoogle Scholar
  70. WHO - World Health Organization. (2002). Health in sustainable development planning: The role of indicators (p. 42). Geneva.Google Scholar
  71. World Bank. (2000). Developing indicators: Lessons learned from Central America (p. 22). Washington. Retrieved from
  72. WWAP - World Water Assessment Programme. (2003). The United Nations world water development report: Water for people water for life. Paris: UNESCO, and London: Earthscan.Google Scholar
  73. WWAP - World Water Assessment Programme. (2006). The United Nations World Water Development Report 2: Water - A Shared Responsibility. Paris: UNESCO, and London: Earthscan.Google Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.Universidade Federal da BahiaSalvadorBrazil
  2. 2.Universidad Politecnica de CatalunyaBarcelonaSpain
  3. 3.International Living Future InstituteSeattleUSA

Personalised recommendations