Sustainability indicators are well recognized for their potential to assess and monitor sustainable development of agricultural systems. A large number of indicators are proposed in various sustainability assessment frameworks, which raises concerns regarding the validity of approaches, usefulness and trust in such frameworks. Selecting indicators requires transparent and well-defined procedures to ensure the relevance and validity of sustainability assessments. The objective of this study, therefore, was to determine whether experts agree on which criteria are most important in the selection of indicators and indicator sets for robust sustainability assessments. Two groups of experts (Temperate Agriculture Research Network and New Zealand Sustainability Dashboard) were asked to rank the relative importance of eleven criteria for selecting individual indicators and of nine criteria for balancing a collective set of indicators. Both ranking surveys reveal a startling lack of consensus amongst experts about how best to measure agricultural sustainability and call for a radical rethink about how complementary approaches to sustainability assessments are used alongside each other to ensure a plurality of views and maximum collaboration and trust amongst stakeholders. To improve the transparency, relevance and robustness of sustainable assessments, the context of the sustainability assessment, including prioritizations of selection criteria for indicator selection, must be accounted for. A collaborative design process will enhance the acceptance of diverse values and prioritizations embedded in sustainability assessments. The process by which indicators and sustainability frameworks are established may be a much more important determinant of their success than the final shape of the assessment tools. Such an emphasis on process would make assessments more transparent, transformative and enduring.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Agrawal, A. (2005). Environmentality. Technologies of government and the making of subjects. Durham and London: Duke University Press.
Alrøe, H. F., & Kristensen, E. S. (2002). Towards a systemic research methodology in agriculture: Rethinking the role of values in science. Agriculture and Human Values, 19(1), 3–23.
Alrøe, H. F., Moller, H., Læssøe, J., & Noe, E. (2016). Opportunities and challenges for multicriteria assessment of food system sustainability. Ecology and Society. doi:10.5751/ES-08394-210138.
Ansell, C., & Gash, A. (2008). Collaborative governance in theory and practice. Journal of Public Administration Research and Theory, 18(4), 543–571. doi:10.1093/jopart/mum032.
Bäckstrand, K. (2004). Scientisation vs. civic expertise in environmental governance: Eco-feminist, eco-modern and post-modern responses. Environmental Politics, 13(4), 695–714. doi:10.1080/0964401042000274322.
Barnaud, C., & Van Paassen, A. (2013). Equity, power games, and legitimacy: Dilemmas of participatory natural resource management. Ecology and Society. doi:10.5751/ES-05459-180221.
Bell, S., & Morse, S. (2008). Sustainability indicators: Measuring the immeasurable?. London: Earthscan.
Belt, M., & Blake, D. (2015). Mediated modeling in water resource dialogues connecting multiple scales. JAWRA Journal of the American Water Resources Association, 51(6), 1581–1599.
Binder, C. R., Feola, G., & Steinberger, J. K. (2010). Considering the normative, systemic and procedural dimensions in indicator-based sustainability assessments in agriculture. Environmental Impact Assessment Review, 30(2), 71–81.
Bockstaller, C., Feschet, P., & Angevin, F. (2015). Issues in evaluating sustainability of farming systems with indicators. OCL Oilseeds and Fats, Crops and Lipids. doi:10.1051/ocl/2014052.
Bockstaller, C., Guichard, L., Keichinger, O., Girardin, P., Galan, M. B., & Gaillard, G. (2009). Comparison of methods to assess the sustainability of agricultural systems. A review. Agronomy for Sustainable Development, 29(1), 223–235.
Cloquell-Ballester, V.-A., Cloquell-Ballester, V.-A., Monterde-Díaz, R., & Santamarina-Siurana, M.-C. (2006). Indicators validation for the improvement of environmental and social impact quantitative assessment. Environmental Impact Assessment Review, 26(1), 79–105. doi:10.1016/j.eiar.2005.06.002.
Dale, V. H., & Beyeler, S. C. (2001). Challenges in the development and use of ecological indicators. Ecological Indicators, 1(1), 3–10.
De Mey, K., D’Haene, K., Marchand, F., Meul, M., & Lauwers, L. (2011). Learning through stakeholder involvement in the implementation of MOTIFS: An integrated assessment model for sustainable farming in Flanders. International Journal of Agricultural Sustainability, 9(2), 350–363.
de Olde, E. M., Oudshoorn, F. W., Sørensen, C. A. G., Bokkers, E. A. M., & de Boer, I. J. M. (2016). Assessing sustainability at farm-level: Lessons learned from a comparison of tools in practice. Ecological Indicators, 66, 391–404. doi:10.1016/j.ecolind.2016.01.047.
Dovers, S. (2005). Clarifying the imperative of integration research for sustainable environmental management. Journal of Research Practice, 1(2), 1–19.
Elsaesser, M., Jilg, T., Herrmann, K., Boonen, J., Debruyne, L., Laidlaw, A. S., et al. (2015). Quantifying sustainability of dairy farms with the DAIRYMAN sustainability-index. In: Paper presented at the European Grassland Federation, Wageningen, The Netherlands.
FAO. (2013). Sustainability assessment of food and agriculture systems (SAFA): Guidelines, version 3.0. Rome: Food and Agricultural Organization of the United Nations.
Ferraro, P. J., & Pattanayak, S. K. (2006). Money for nothing? A call for empirical evaluation of biodiversity conservation investments. PLoS Biology, 4(4), e105. doi:10.1371/journal.pbio.0040105.
Gasparatos, A. (2010). Embedded value systems in sustainability assessment tools and their implications. Journal of Environmental Management, 91(8), 1613–1622. doi:10.1016/j.jenvman.2010.03.014.
Gasso, V., Oudshoorn, F. W., de Olde, E., & Sørensen, C. A. G. (2015). Generic sustainability assessment themes and the role of context: The case of Danish maize for German biogas. Ecological Indicators, 49, 143–153. doi:10.1016/j.ecolind.2014.10.008.
Gibbons, J. D., & Chakraborti, S. (2011). Nonparametric statistical inference. Berlin: Springer.
Hansen, J. (1996). Is agricultural sustainability a useful concept? Agricultural Systems, 50(2), 117–143.
Herzog, F., Balázs, K., Dennis, P., Friedel, J., Geijzendorffer, I., Jeanneret, P., et al. (2012). Biodiversity indicators for European farming systems: A guidebook: Forschungsanstalt Agroscope Reckenholz-Tänikon ART.
IAASTD. (2009). Agriculture at a crossroads: Synthesis report. International Assessment of Agricultural Knowledge, Science and Technology for Development (IAASTD) Science and Technology for Development. Island Press.
Jones, C., Cowan, P., & Allen, W. (2012). Setting outcomes, and measuring and reporting performance of regional council pest and weed management programmes. Guidelines and resource materials. Landcare Research Contract Report LC144: Landcare Research New Zealand Ltd.
Kates, R. W., Clark, W. C., Corell, R., Hall, J. M., Jaeger, C. C., Lowe, I., et al. (2001). Environment and development: Sustainability science. Science, 292(5517), 641–642. doi:10.1126/science.1059386.
Kendall, M. G., & Smith, B. B. (1939). The problem of m rankings. The Annals of Mathematical Statistics, 10(3), 275–287.
Keulen, H. V., van Ittersum, M., & Leffelaar, P. (2005). Multiscale methodological framework to derive criteria and indicators for sustainability evaluation of peasant natural resource management systems. Environment, Development and Sustainability, 7(1), 51–69.
Komiyama, H., & Takeuchi, K. (2006). Sustainability science: Building a new discipline. Sustainability Science, 1(1), 1–6. doi:10.1007/s11625-006-0007-4.
Lebacq, T., Baret, P. V., & Stilmant, D. (2013). Sustainability indicators for livestock farming. A review. Agronomy for Sustainable Development, 33(2), 311–327.
Lee, W., McGlone, M., & Wright, E. (2005). Biodiversity inventory and monitoring: A review of national and international systems and a proposed framework for future biodiversity monitoring by the Department of Conservation. Landcare Research Contract Report LC0405/122.
Lupia, A. (2013). Communicating science in politicized environments. Proceedings of the National Academy of Sciences, 110(Supplement 3), 14048–14054. doi:10.1073/pnas.1212726110.
Marchand, F., Debruyne, L., Triste, L., Gerrard, C., Padel, S., & Lauwers, L. (2014). Key characteristics for tool choice in indicator-based sustainability assessment at farm level. Ecology and Society. doi:10.5751/ES-06876-190346.
Merfield, C., Moller, H., Manhire, J., Rosin, C., Norton, S., Carey, P., et al. (2015). Are organic standards sufficient to ensure sustainable agriculture? Lessons from New Zealand’s ARGOS and Sustainability Dashboard projects. Sustainable Agriculture Research, 4(3), p158.
Moller, H., & MacLeod, C. J. (2013). Design criteria for effective assessment of sustainability in New Zealand’s production landscapes. (Vol. 13/07, pp. 73): NZ Sustainability Dashboard Research Report.
Moller, H., O’Blyver, P., Bragg, C., Newman, J., Clucas, R., Fletcher, D., et al. (2009). Guidelines for cross-cultural participatory action research partnerships: A case study of a customary seabird harvest in New Zealand. New Zealand Journal of Zoology, 36(3), 211–241. doi:10.1080/03014220909510152.
Niemeijer, D., & de Groot, R. S. (2008). A conceptual framework for selecting environmental indicator sets. Ecological Indicators, 8(1), 14–25. doi:10.1016/j.ecolind.2006.11.012.
OECD. (2001). Environmental indicators for agriculture. Methods and results (Vol. 3). Paris: Organisation for Economic Co-operation and Development.
Ostrom, E. (2009). A general framework for analyzing sustainability of social-ecological systems. Science, 325(5939), 419–422. doi:10.1126/science.1172133.
Owens, S. (2003). Is there a meaningful definition of sustainability? Plant Genetic Resources: Characterization and Utilization, 1(01), 5–9.
Parris, T. M., & Kates, R. W. (2003). Characterizing and measuring sustainable development. Annual Review of Environment and Resources, 28, 559–586.
Popa, F., Guillermin, M., & Dedeurwaerdere, T. (2015). A pragmatist approach to transdisciplinarity in sustainability research: From complex systems theory to reflexive science. Futures. doi:10.1016/j.futures.2014.02.002.
Pretty, J. (2008). Agricultural sustainability: Concepts, principles and evidence. Philosophical Transactions of the Royal Society B: Biological Sciences, 363(1491), 447–465.
Pretty, J., Sutherland, W. J., Ashby, J., Auburn, J., Baulcombe, D., Bell, M., et al. (2010). The top 100 questions of importance to the future of global agriculture. International Journal of Agricultural Sustainability, 8(4), 219–236.
Reed, M. S., Fraser, E. D. G., & Dougill, A. J. (2006). An adaptive learning process for developing and applying sustainability indicators with local communities. Ecological Economics, 59(4), 406–418.
Sadok, W., Angevin, F., Bergez, J.-E., Bockstaller, C., Colomb, B., Guichard, L., et al. (2009). MASC, a qualitative multi-attribute decision model for ex ante assessment of the sustainability of cropping systems. Agronomy for Sustainable Development, 29(3), 447–461. doi:10.1051/agro/2009006.
Schader, C., Grenz, J., Meier, M. S., & Stolze, M. (2014). Scope and precision of sustainability assessment approaches to food systems. Ecology and Society. doi:10.5751/ES-06866-190342.
Schiere, J. B., Lyklema, J., Schakel, J., & Rickert, K. G. (1999). Evolution of farming systems and system philosophy. Systems Research and Behavioral Science, 16(4), 375–390.
Seimon, A., Plumptre, A. J., & Watson, J. E. M. (2012). Building consensus on Albertine Rift climate change adaptation for conservation: A report on 2011–2012 workshops in Uganda and Rwanda. WCS Workshop Report. New York, USA: Wildlife Conservation Society (WCS).
Seimon, A., Yager, K., Seimon, T., Schmidt, S., Grau, A., Beck, S., et al. (2009). Changes in biodiversity patterns in the high andes—Understanding the consequences and seeking adaptation to global change. Mountain Forum Bulletin, 9, 25–27.
Siegel, S. (1956). Nonparametric statistics for the behavioral sciences. New York, NY: McGraw Hill.
Sommerville, M. M., Milner-Gulland, E., & Jones, J. P. (2011). The challenge of monitoring biodiversity in payment for environmental service interventions. Biological Conservation, 144(12), 2832–2841.
Steinfeld, H., Gerber, P., Wassenaar, T., Castel, V., Rosales, M., & Haan, C. D. (2006). Livestock’s long shadow: Environmental issues and options. Rome: Food and Agriculture Organization of the United Nations (FAO).
Te Velde, H., Aarts, N., & Van Woerkum, C. (2002). Dealing with ambivalence: Farmers’ and consumers’ perceptions of animal welfare in livestock breeding. Journal of Agricultural and Environmental Ethics, 15(2), 203–219. doi:10.1023/A:1015012403331.
Triste, L., Marchand, F., Debruyne, L., Meul, M., & Lauwers, L. (2014). Reflection on the development process of a sustainability assessment tool: Learning from a Flemish case. Ecology and Society. doi:10.5751/ES-06789-190347.
Yager, K., Ulloa, D., & Halloy, S. (2009). Chapter 16. Conducting an interdisciplinary workshop on climate change: Facilitating awareness and adaptation in Sajama National Park, Bolivia. (Interdisciplinary Aspects of Climate Change). Hamburg: Hamburg University of Applied Sciences.
We would like to thank all the participants from TempAg and NZSD for their participation. We were also grateful for guidance and permission from Paul Hansen and Franz Ombler for deploying the 1000Minds software. We would like to acknowledge Peter Groffman and the anonymous reviewers for their constructive suggestions on an earlier version of this paper. This is the first paper of the Resilient Agricultural Production Systems team of the Temperate Agriculture Research Network and international collaboration initiated by OECD’s Global Science Forum. NZSD’s participation is funded by New Zealand’s Ministry for Business, Innovation and Employment (contract AGRB1201).
About this article
Cite this article
de Olde, E.M., Moller, H., Marchand, F. et al. When experts disagree: the need to rethink indicator selection for assessing sustainability of agriculture. Environ Dev Sustain 19, 1327–1342 (2017). https://doi.org/10.1007/s10668-016-9803-x
- Indicator selection
- Multi-criteria assessment
- Sustainability assessment
- Temperate agriculture