When experts disagree: the need to rethink indicator selection for assessing sustainability of agriculture

Abstract

Sustainability indicators are well recognized for their potential to assess and monitor sustainable development of agricultural systems. A large number of indicators are proposed in various sustainability assessment frameworks, which raises concerns regarding the validity of approaches, usefulness and trust in such frameworks. Selecting indicators requires transparent and well-defined procedures to ensure the relevance and validity of sustainability assessments. The objective of this study, therefore, was to determine whether experts agree on which criteria are most important in the selection of indicators and indicator sets for robust sustainability assessments. Two groups of experts (Temperate Agriculture Research Network and New Zealand Sustainability Dashboard) were asked to rank the relative importance of eleven criteria for selecting individual indicators and of nine criteria for balancing a collective set of indicators. Both ranking surveys reveal a startling lack of consensus amongst experts about how best to measure agricultural sustainability and call for a radical rethink about how complementary approaches to sustainability assessments are used alongside each other to ensure a plurality of views and maximum collaboration and trust amongst stakeholders. To improve the transparency, relevance and robustness of sustainable assessments, the context of the sustainability assessment, including prioritizations of selection criteria for indicator selection, must be accounted for. A collaborative design process will enhance the acceptance of diverse values and prioritizations embedded in sustainability assessments. The process by which indicators and sustainability frameworks are established may be a much more important determinant of their success than the final shape of the assessment tools. Such an emphasis on process would make assessments more transparent, transformative and enduring.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2

References

  1. Agrawal, A. (2005). Environmentality. Technologies of government and the making of subjects. Durham and London: Duke University Press.

    Google Scholar 

  2. Alrøe, H. F., & Kristensen, E. S. (2002). Towards a systemic research methodology in agriculture: Rethinking the role of values in science. Agriculture and Human Values, 19(1), 3–23.

    Article  Google Scholar 

  3. Alrøe, H. F., Moller, H., Læssøe, J., & Noe, E. (2016). Opportunities and challenges for multicriteria assessment of food system sustainability. Ecology and Society. doi:10.5751/ES-08394-210138.

    Google Scholar 

  4. Ansell, C., & Gash, A. (2008). Collaborative governance in theory and practice. Journal of Public Administration Research and Theory, 18(4), 543–571. doi:10.1093/jopart/mum032.

    Article  Google Scholar 

  5. Bäckstrand, K. (2004). Scientisation vs. civic expertise in environmental governance: Eco-feminist, eco-modern and post-modern responses. Environmental Politics, 13(4), 695–714. doi:10.1080/0964401042000274322.

    Article  Google Scholar 

  6. Barnaud, C., & Van Paassen, A. (2013). Equity, power games, and legitimacy: Dilemmas of participatory natural resource management. Ecology and Society. doi:10.5751/ES-05459-180221.

    Google Scholar 

  7. Bell, S., & Morse, S. (2008). Sustainability indicators: Measuring the immeasurable?. London: Earthscan.

    Google Scholar 

  8. Belt, M., & Blake, D. (2015). Mediated modeling in water resource dialogues connecting multiple scales. JAWRA Journal of the American Water Resources Association, 51(6), 1581–1599.

    Article  Google Scholar 

  9. Binder, C. R., Feola, G., & Steinberger, J. K. (2010). Considering the normative, systemic and procedural dimensions in indicator-based sustainability assessments in agriculture. Environmental Impact Assessment Review, 30(2), 71–81.

    Article  Google Scholar 

  10. Bockstaller, C., Feschet, P., & Angevin, F. (2015). Issues in evaluating sustainability of farming systems with indicators. OCL Oilseeds and Fats, Crops and Lipids. doi:10.1051/ocl/2014052.

    Google Scholar 

  11. Bockstaller, C., Guichard, L., Keichinger, O., Girardin, P., Galan, M. B., & Gaillard, G. (2009). Comparison of methods to assess the sustainability of agricultural systems. A review. Agronomy for Sustainable Development, 29(1), 223–235.

    Article  Google Scholar 

  12. Cloquell-Ballester, V.-A., Cloquell-Ballester, V.-A., Monterde-Díaz, R., & Santamarina-Siurana, M.-C. (2006). Indicators validation for the improvement of environmental and social impact quantitative assessment. Environmental Impact Assessment Review, 26(1), 79–105. doi:10.1016/j.eiar.2005.06.002.

    Article  Google Scholar 

  13. Dale, V. H., & Beyeler, S. C. (2001). Challenges in the development and use of ecological indicators. Ecological Indicators, 1(1), 3–10.

    Article  Google Scholar 

  14. De Mey, K., D’Haene, K., Marchand, F., Meul, M., & Lauwers, L. (2011). Learning through stakeholder involvement in the implementation of MOTIFS: An integrated assessment model for sustainable farming in Flanders. International Journal of Agricultural Sustainability, 9(2), 350–363.

    Google Scholar 

  15. de Olde, E. M., Oudshoorn, F. W., Sørensen, C. A. G., Bokkers, E. A. M., & de Boer, I. J. M. (2016). Assessing sustainability at farm-level: Lessons learned from a comparison of tools in practice. Ecological Indicators, 66, 391–404. doi:10.1016/j.ecolind.2016.01.047.

    Article  Google Scholar 

  16. Dovers, S. (2005). Clarifying the imperative of integration research for sustainable environmental management. Journal of Research Practice, 1(2), 1–19.

    Google Scholar 

  17. Elsaesser, M., Jilg, T., Herrmann, K., Boonen, J., Debruyne, L., Laidlaw, A. S., et al. (2015). Quantifying sustainability of dairy farms with the DAIRYMAN sustainability-index. In: Paper presented at the European Grassland Federation, Wageningen, The Netherlands.

  18. FAO. (2013). Sustainability assessment of food and agriculture systems (SAFA): Guidelines, version 3.0. Rome: Food and Agricultural Organization of the United Nations.

    Google Scholar 

  19. Ferraro, P. J., & Pattanayak, S. K. (2006). Money for nothing? A call for empirical evaluation of biodiversity conservation investments. PLoS Biology, 4(4), e105. doi:10.1371/journal.pbio.0040105.

    Article  Google Scholar 

  20. Gasparatos, A. (2010). Embedded value systems in sustainability assessment tools and their implications. Journal of Environmental Management, 91(8), 1613–1622. doi:10.1016/j.jenvman.2010.03.014.

    Article  Google Scholar 

  21. Gasso, V., Oudshoorn, F. W., de Olde, E., & Sørensen, C. A. G. (2015). Generic sustainability assessment themes and the role of context: The case of Danish maize for German biogas. Ecological Indicators, 49, 143–153. doi:10.1016/j.ecolind.2014.10.008.

    Article  Google Scholar 

  22. Gibbons, J. D., & Chakraborti, S. (2011). Nonparametric statistical inference. Berlin: Springer.

    Book  Google Scholar 

  23. Hansen, J. (1996). Is agricultural sustainability a useful concept? Agricultural Systems, 50(2), 117–143.

    Article  Google Scholar 

  24. Herzog, F., Balázs, K., Dennis, P., Friedel, J., Geijzendorffer, I., Jeanneret, P., et al. (2012). Biodiversity indicators for European farming systems: A guidebook: Forschungsanstalt Agroscope Reckenholz-Tänikon ART.

  25. IAASTD. (2009). Agriculture at a crossroads: Synthesis report. International Assessment of Agricultural Knowledge, Science and Technology for Development (IAASTD) Science and Technology for Development. Island Press.

  26. Jones, C., Cowan, P., & Allen, W. (2012). Setting outcomes, and measuring and reporting performance of regional council pest and weed management programmes. Guidelines and resource materials. Landcare Research Contract Report LC144: Landcare Research New Zealand Ltd.

  27. Kates, R. W., Clark, W. C., Corell, R., Hall, J. M., Jaeger, C. C., Lowe, I., et al. (2001). Environment and development: Sustainability science. Science, 292(5517), 641–642. doi:10.1126/science.1059386.

    CAS  Article  Google Scholar 

  28. Kendall, M. G., & Smith, B. B. (1939). The problem of m rankings. The Annals of Mathematical Statistics, 10(3), 275–287.

    Article  Google Scholar 

  29. Keulen, H. V., van Ittersum, M., & Leffelaar, P. (2005). Multiscale methodological framework to derive criteria and indicators for sustainability evaluation of peasant natural resource management systems. Environment, Development and Sustainability, 7(1), 51–69.

    Article  Google Scholar 

  30. Komiyama, H., & Takeuchi, K. (2006). Sustainability science: Building a new discipline. Sustainability Science, 1(1), 1–6. doi:10.1007/s11625-006-0007-4.

    Article  Google Scholar 

  31. Lebacq, T., Baret, P. V., & Stilmant, D. (2013). Sustainability indicators for livestock farming. A review. Agronomy for Sustainable Development, 33(2), 311–327.

    Article  Google Scholar 

  32. Lee, W., McGlone, M., & Wright, E. (2005). Biodiversity inventory and monitoring: A review of national and international systems and a proposed framework for future biodiversity monitoring by the Department of Conservation. Landcare Research Contract Report LC0405/122.

  33. Lupia, A. (2013). Communicating science in politicized environments. Proceedings of the National Academy of Sciences, 110(Supplement 3), 14048–14054. doi:10.1073/pnas.1212726110.

    CAS  Article  Google Scholar 

  34. Marchand, F., Debruyne, L., Triste, L., Gerrard, C., Padel, S., & Lauwers, L. (2014). Key characteristics for tool choice in indicator-based sustainability assessment at farm level. Ecology and Society. doi:10.5751/ES-06876-190346.

    Google Scholar 

  35. Merfield, C., Moller, H., Manhire, J., Rosin, C., Norton, S., Carey, P., et al. (2015). Are organic standards sufficient to ensure sustainable agriculture? Lessons from New Zealand’s ARGOS and Sustainability Dashboard projects. Sustainable Agriculture Research, 4(3), p158.

    Article  Google Scholar 

  36. Moller, H., & MacLeod, C. J. (2013). Design criteria for effective assessment of sustainability in New Zealand’s production landscapes. (Vol. 13/07, pp. 73): NZ Sustainability Dashboard Research Report.

  37. Moller, H., O’Blyver, P., Bragg, C., Newman, J., Clucas, R., Fletcher, D., et al. (2009). Guidelines for cross-cultural participatory action research partnerships: A case study of a customary seabird harvest in New Zealand. New Zealand Journal of Zoology, 36(3), 211–241. doi:10.1080/03014220909510152.

    Article  Google Scholar 

  38. Niemeijer, D., & de Groot, R. S. (2008). A conceptual framework for selecting environmental indicator sets. Ecological Indicators, 8(1), 14–25. doi:10.1016/j.ecolind.2006.11.012.

    Article  Google Scholar 

  39. OECD. (2001). Environmental indicators for agriculture. Methods and results (Vol. 3). Paris: Organisation for Economic Co-operation and Development.

    Google Scholar 

  40. Ostrom, E. (2009). A general framework for analyzing sustainability of social-ecological systems. Science, 325(5939), 419–422. doi:10.1126/science.1172133.

    CAS  Article  Google Scholar 

  41. Owens, S. (2003). Is there a meaningful definition of sustainability? Plant Genetic Resources: Characterization and Utilization, 1(01), 5–9.

    Article  Google Scholar 

  42. Parris, T. M., & Kates, R. W. (2003). Characterizing and measuring sustainable development. Annual Review of Environment and Resources, 28, 559–586.

    Article  Google Scholar 

  43. Popa, F., Guillermin, M., & Dedeurwaerdere, T. (2015). A pragmatist approach to transdisciplinarity in sustainability research: From complex systems theory to reflexive science. Futures. doi:10.1016/j.futures.2014.02.002.

    Google Scholar 

  44. Pretty, J. (2008). Agricultural sustainability: Concepts, principles and evidence. Philosophical Transactions of the Royal Society B: Biological Sciences, 363(1491), 447–465.

    Article  Google Scholar 

  45. Pretty, J., Sutherland, W. J., Ashby, J., Auburn, J., Baulcombe, D., Bell, M., et al. (2010). The top 100 questions of importance to the future of global agriculture. International Journal of Agricultural Sustainability, 8(4), 219–236.

    Article  Google Scholar 

  46. Reed, M. S., Fraser, E. D. G., & Dougill, A. J. (2006). An adaptive learning process for developing and applying sustainability indicators with local communities. Ecological Economics, 59(4), 406–418.

    Article  Google Scholar 

  47. Sadok, W., Angevin, F., Bergez, J.-E., Bockstaller, C., Colomb, B., Guichard, L., et al. (2009). MASC, a qualitative multi-attribute decision model for ex ante assessment of the sustainability of cropping systems. Agronomy for Sustainable Development, 29(3), 447–461. doi:10.1051/agro/2009006.

    Article  Google Scholar 

  48. Schader, C., Grenz, J., Meier, M. S., & Stolze, M. (2014). Scope and precision of sustainability assessment approaches to food systems. Ecology and Society. doi:10.5751/ES-06866-190342.

    Google Scholar 

  49. Schiere, J. B., Lyklema, J., Schakel, J., & Rickert, K. G. (1999). Evolution of farming systems and system philosophy. Systems Research and Behavioral Science, 16(4), 375–390.

    Article  Google Scholar 

  50. Seimon, A., Plumptre, A. J., & Watson, J. E. M. (2012). Building consensus on Albertine Rift climate change adaptation for conservation: A report on 2011–2012 workshops in Uganda and Rwanda. WCS Workshop Report. New York, USA: Wildlife Conservation Society (WCS).

  51. Seimon, A., Yager, K., Seimon, T., Schmidt, S., Grau, A., Beck, S., et al. (2009). Changes in biodiversity patterns in the high andes—Understanding the consequences and seeking adaptation to global change. Mountain Forum Bulletin, 9, 25–27.

    Google Scholar 

  52. Siegel, S. (1956). Nonparametric statistics for the behavioral sciences. New York, NY: McGraw Hill.

    Google Scholar 

  53. Sommerville, M. M., Milner-Gulland, E., & Jones, J. P. (2011). The challenge of monitoring biodiversity in payment for environmental service interventions. Biological Conservation, 144(12), 2832–2841.

    Article  Google Scholar 

  54. Steinfeld, H., Gerber, P., Wassenaar, T., Castel, V., Rosales, M., & Haan, C. D. (2006). Livestock’s long shadow: Environmental issues and options. Rome: Food and Agriculture Organization of the United Nations (FAO).

    Google Scholar 

  55. Te Velde, H., Aarts, N., & Van Woerkum, C. (2002). Dealing with ambivalence: Farmers’ and consumers’ perceptions of animal welfare in livestock breeding. Journal of Agricultural and Environmental Ethics, 15(2), 203–219. doi:10.1023/A:1015012403331.

    Article  Google Scholar 

  56. Triste, L., Marchand, F., Debruyne, L., Meul, M., & Lauwers, L. (2014). Reflection on the development process of a sustainability assessment tool: Learning from a Flemish case. Ecology and Society. doi:10.5751/ES-06789-190347.

    Google Scholar 

  57. Yager, K., Ulloa, D., & Halloy, S. (2009). Chapter 16. Conducting an interdisciplinary workshop on climate change: Facilitating awareness and adaptation in Sajama National Park, Bolivia. (Interdisciplinary Aspects of Climate Change). Hamburg: Hamburg University of Applied Sciences.

Download references

Acknowledgments

We would like to thank all the participants from TempAg and NZSD for their participation. We were also grateful for guidance and permission from Paul Hansen and Franz Ombler for deploying the 1000Minds software. We would like to acknowledge Peter Groffman and the anonymous reviewers for their constructive suggestions on an earlier version of this paper. This is the first paper of the Resilient Agricultural Production Systems team of the Temperate Agriculture Research Network and international collaboration initiated by OECD’s Global Science Forum. NZSD’s participation is funded by New Zealand’s Ministry for Business, Innovation and Employment (contract AGRB1201).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Evelien M. de Olde.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

de Olde, E.M., Moller, H., Marchand, F. et al. When experts disagree: the need to rethink indicator selection for assessing sustainability of agriculture. Environ Dev Sustain 19, 1327–1342 (2017). https://doi.org/10.1007/s10668-016-9803-x

Download citation

Keywords

  • Indicator selection
  • Multi-criteria assessment
  • Ranking
  • Sustainability assessment
  • Temperate agriculture