Environment, Development and Sustainability

, Volume 19, Issue 4, pp 1327–1342 | Cite as

When experts disagree: the need to rethink indicator selection for assessing sustainability of agriculture

  • Evelien M. de Olde
  • Henrik Moller
  • Fleur Marchand
  • Richard W. McDowell
  • Catriona J. MacLeod
  • Marion Sautier
  • Stephan Halloy
  • Andrew Barber
  • Jayson Benge
  • Christian Bockstaller
  • Eddie A. M. Bokkers
  • Imke J. M. de Boer
  • Katharine A. Legun
  • Isabelle Le Quellec
  • Charles Merfield
  • Frank W. Oudshoorn
  • John Reid
  • Christian Schader
  • Erika Szymanski
  • Claus A. G. Sørensen
  • Jay Whitehead
  • Jon Manhire
Article

Abstract

Sustainability indicators are well recognized for their potential to assess and monitor sustainable development of agricultural systems. A large number of indicators are proposed in various sustainability assessment frameworks, which raises concerns regarding the validity of approaches, usefulness and trust in such frameworks. Selecting indicators requires transparent and well-defined procedures to ensure the relevance and validity of sustainability assessments. The objective of this study, therefore, was to determine whether experts agree on which criteria are most important in the selection of indicators and indicator sets for robust sustainability assessments. Two groups of experts (Temperate Agriculture Research Network and New Zealand Sustainability Dashboard) were asked to rank the relative importance of eleven criteria for selecting individual indicators and of nine criteria for balancing a collective set of indicators. Both ranking surveys reveal a startling lack of consensus amongst experts about how best to measure agricultural sustainability and call for a radical rethink about how complementary approaches to sustainability assessments are used alongside each other to ensure a plurality of views and maximum collaboration and trust amongst stakeholders. To improve the transparency, relevance and robustness of sustainable assessments, the context of the sustainability assessment, including prioritizations of selection criteria for indicator selection, must be accounted for. A collaborative design process will enhance the acceptance of diverse values and prioritizations embedded in sustainability assessments. The process by which indicators and sustainability frameworks are established may be a much more important determinant of their success than the final shape of the assessment tools. Such an emphasis on process would make assessments more transparent, transformative and enduring.

Keywords

Indicator selection Multi-criteria assessment Ranking Sustainability assessment Temperate agriculture 

Notes

Acknowledgments

We would like to thank all the participants from TempAg and NZSD for their participation. We were also grateful for guidance and permission from Paul Hansen and Franz Ombler for deploying the 1000Minds software. We would like to acknowledge Peter Groffman and the anonymous reviewers for their constructive suggestions on an earlier version of this paper. This is the first paper of the Resilient Agricultural Production Systems team of the Temperate Agriculture Research Network and international collaboration initiated by OECD’s Global Science Forum. NZSD’s participation is funded by New Zealand’s Ministry for Business, Innovation and Employment (contract AGRB1201).

References

  1. Agrawal, A. (2005). Environmentality. Technologies of government and the making of subjects. Durham and London: Duke University Press.Google Scholar
  2. Alrøe, H. F., & Kristensen, E. S. (2002). Towards a systemic research methodology in agriculture: Rethinking the role of values in science. Agriculture and Human Values, 19(1), 3–23.CrossRefGoogle Scholar
  3. Alrøe, H. F., Moller, H., Læssøe, J., & Noe, E. (2016). Opportunities and challenges for multicriteria assessment of food system sustainability. Ecology and Society. doi:10.5751/ES-08394-210138.Google Scholar
  4. Ansell, C., & Gash, A. (2008). Collaborative governance in theory and practice. Journal of Public Administration Research and Theory, 18(4), 543–571. doi:10.1093/jopart/mum032.CrossRefGoogle Scholar
  5. Bäckstrand, K. (2004). Scientisation vs. civic expertise in environmental governance: Eco-feminist, eco-modern and post-modern responses. Environmental Politics, 13(4), 695–714. doi:10.1080/0964401042000274322.CrossRefGoogle Scholar
  6. Barnaud, C., & Van Paassen, A. (2013). Equity, power games, and legitimacy: Dilemmas of participatory natural resource management. Ecology and Society. doi:10.5751/ES-05459-180221.Google Scholar
  7. Bell, S., & Morse, S. (2008). Sustainability indicators: Measuring the immeasurable?. London: Earthscan.Google Scholar
  8. Belt, M., & Blake, D. (2015). Mediated modeling in water resource dialogues connecting multiple scales. JAWRA Journal of the American Water Resources Association, 51(6), 1581–1599.CrossRefGoogle Scholar
  9. Binder, C. R., Feola, G., & Steinberger, J. K. (2010). Considering the normative, systemic and procedural dimensions in indicator-based sustainability assessments in agriculture. Environmental Impact Assessment Review, 30(2), 71–81.CrossRefGoogle Scholar
  10. Bockstaller, C., Feschet, P., & Angevin, F. (2015). Issues in evaluating sustainability of farming systems with indicators. OCL Oilseeds and Fats, Crops and Lipids. doi:10.1051/ocl/2014052.Google Scholar
  11. Bockstaller, C., Guichard, L., Keichinger, O., Girardin, P., Galan, M. B., & Gaillard, G. (2009). Comparison of methods to assess the sustainability of agricultural systems. A review. Agronomy for Sustainable Development, 29(1), 223–235.CrossRefGoogle Scholar
  12. Cloquell-Ballester, V.-A., Cloquell-Ballester, V.-A., Monterde-Díaz, R., & Santamarina-Siurana, M.-C. (2006). Indicators validation for the improvement of environmental and social impact quantitative assessment. Environmental Impact Assessment Review, 26(1), 79–105. doi:10.1016/j.eiar.2005.06.002.CrossRefGoogle Scholar
  13. Dale, V. H., & Beyeler, S. C. (2001). Challenges in the development and use of ecological indicators. Ecological Indicators, 1(1), 3–10.CrossRefGoogle Scholar
  14. De Mey, K., D’Haene, K., Marchand, F., Meul, M., & Lauwers, L. (2011). Learning through stakeholder involvement in the implementation of MOTIFS: An integrated assessment model for sustainable farming in Flanders. International Journal of Agricultural Sustainability, 9(2), 350–363.Google Scholar
  15. de Olde, E. M., Oudshoorn, F. W., Sørensen, C. A. G., Bokkers, E. A. M., & de Boer, I. J. M. (2016). Assessing sustainability at farm-level: Lessons learned from a comparison of tools in practice. Ecological Indicators, 66, 391–404. doi:10.1016/j.ecolind.2016.01.047.CrossRefGoogle Scholar
  16. Dovers, S. (2005). Clarifying the imperative of integration research for sustainable environmental management. Journal of Research Practice, 1(2), 1–19.Google Scholar
  17. Elsaesser, M., Jilg, T., Herrmann, K., Boonen, J., Debruyne, L., Laidlaw, A. S., et al. (2015). Quantifying sustainability of dairy farms with the DAIRYMAN sustainability-index. In: Paper presented at the European Grassland Federation, Wageningen, The Netherlands.Google Scholar
  18. FAO. (2013). Sustainability assessment of food and agriculture systems (SAFA): Guidelines, version 3.0. Rome: Food and Agricultural Organization of the United Nations.Google Scholar
  19. Ferraro, P. J., & Pattanayak, S. K. (2006). Money for nothing? A call for empirical evaluation of biodiversity conservation investments. PLoS Biology, 4(4), e105. doi:10.1371/journal.pbio.0040105.CrossRefGoogle Scholar
  20. Gasparatos, A. (2010). Embedded value systems in sustainability assessment tools and their implications. Journal of Environmental Management, 91(8), 1613–1622. doi:10.1016/j.jenvman.2010.03.014.CrossRefGoogle Scholar
  21. Gasso, V., Oudshoorn, F. W., de Olde, E., & Sørensen, C. A. G. (2015). Generic sustainability assessment themes and the role of context: The case of Danish maize for German biogas. Ecological Indicators, 49, 143–153. doi:10.1016/j.ecolind.2014.10.008.CrossRefGoogle Scholar
  22. Gibbons, J. D., & Chakraborti, S. (2011). Nonparametric statistical inference. Berlin: Springer.CrossRefGoogle Scholar
  23. Hansen, J. (1996). Is agricultural sustainability a useful concept? Agricultural Systems, 50(2), 117–143.CrossRefGoogle Scholar
  24. Herzog, F., Balázs, K., Dennis, P., Friedel, J., Geijzendorffer, I., Jeanneret, P., et al. (2012). Biodiversity indicators for European farming systems: A guidebook: Forschungsanstalt Agroscope Reckenholz-Tänikon ART.Google Scholar
  25. IAASTD. (2009). Agriculture at a crossroads: Synthesis report. International Assessment of Agricultural Knowledge, Science and Technology for Development (IAASTD) Science and Technology for Development. Island Press.Google Scholar
  26. Jones, C., Cowan, P., & Allen, W. (2012). Setting outcomes, and measuring and reporting performance of regional council pest and weed management programmes. Guidelines and resource materials. Landcare Research Contract Report LC144: Landcare Research New Zealand Ltd.Google Scholar
  27. Kates, R. W., Clark, W. C., Corell, R., Hall, J. M., Jaeger, C. C., Lowe, I., et al. (2001). Environment and development: Sustainability science. Science, 292(5517), 641–642. doi:10.1126/science.1059386.CrossRefGoogle Scholar
  28. Kendall, M. G., & Smith, B. B. (1939). The problem of m rankings. The Annals of Mathematical Statistics, 10(3), 275–287.CrossRefGoogle Scholar
  29. Keulen, H. V., van Ittersum, M., & Leffelaar, P. (2005). Multiscale methodological framework to derive criteria and indicators for sustainability evaluation of peasant natural resource management systems. Environment, Development and Sustainability, 7(1), 51–69.CrossRefGoogle Scholar
  30. Komiyama, H., & Takeuchi, K. (2006). Sustainability science: Building a new discipline. Sustainability Science, 1(1), 1–6. doi:10.1007/s11625-006-0007-4.CrossRefGoogle Scholar
  31. Lebacq, T., Baret, P. V., & Stilmant, D. (2013). Sustainability indicators for livestock farming. A review. Agronomy for Sustainable Development, 33(2), 311–327.CrossRefGoogle Scholar
  32. Lee, W., McGlone, M., & Wright, E. (2005). Biodiversity inventory and monitoring: A review of national and international systems and a proposed framework for future biodiversity monitoring by the Department of Conservation. Landcare Research Contract Report LC0405/122.Google Scholar
  33. Lupia, A. (2013). Communicating science in politicized environments. Proceedings of the National Academy of Sciences, 110(Supplement 3), 14048–14054. doi:10.1073/pnas.1212726110.CrossRefGoogle Scholar
  34. Marchand, F., Debruyne, L., Triste, L., Gerrard, C., Padel, S., & Lauwers, L. (2014). Key characteristics for tool choice in indicator-based sustainability assessment at farm level. Ecology and Society. doi:10.5751/ES-06876-190346.Google Scholar
  35. Merfield, C., Moller, H., Manhire, J., Rosin, C., Norton, S., Carey, P., et al. (2015). Are organic standards sufficient to ensure sustainable agriculture? Lessons from New Zealand’s ARGOS and Sustainability Dashboard projects. Sustainable Agriculture Research, 4(3), p158.CrossRefGoogle Scholar
  36. Moller, H., & MacLeod, C. J. (2013). Design criteria for effective assessment of sustainability in New Zealand’s production landscapes. (Vol. 13/07, pp. 73): NZ Sustainability Dashboard Research Report.Google Scholar
  37. Moller, H., O’Blyver, P., Bragg, C., Newman, J., Clucas, R., Fletcher, D., et al. (2009). Guidelines for cross-cultural participatory action research partnerships: A case study of a customary seabird harvest in New Zealand. New Zealand Journal of Zoology, 36(3), 211–241. doi:10.1080/03014220909510152.CrossRefGoogle Scholar
  38. Niemeijer, D., & de Groot, R. S. (2008). A conceptual framework for selecting environmental indicator sets. Ecological Indicators, 8(1), 14–25. doi:10.1016/j.ecolind.2006.11.012.CrossRefGoogle Scholar
  39. OECD. (2001). Environmental indicators for agriculture. Methods and results (Vol. 3). Paris: Organisation for Economic Co-operation and Development.Google Scholar
  40. Ostrom, E. (2009). A general framework for analyzing sustainability of social-ecological systems. Science, 325(5939), 419–422. doi:10.1126/science.1172133.CrossRefGoogle Scholar
  41. Owens, S. (2003). Is there a meaningful definition of sustainability? Plant Genetic Resources: Characterization and Utilization, 1(01), 5–9.CrossRefGoogle Scholar
  42. Parris, T. M., & Kates, R. W. (2003). Characterizing and measuring sustainable development. Annual Review of Environment and Resources, 28, 559–586.CrossRefGoogle Scholar
  43. Popa, F., Guillermin, M., & Dedeurwaerdere, T. (2015). A pragmatist approach to transdisciplinarity in sustainability research: From complex systems theory to reflexive science. Futures. doi:10.1016/j.futures.2014.02.002.Google Scholar
  44. Pretty, J. (2008). Agricultural sustainability: Concepts, principles and evidence. Philosophical Transactions of the Royal Society B: Biological Sciences, 363(1491), 447–465.CrossRefGoogle Scholar
  45. Pretty, J., Sutherland, W. J., Ashby, J., Auburn, J., Baulcombe, D., Bell, M., et al. (2010). The top 100 questions of importance to the future of global agriculture. International Journal of Agricultural Sustainability, 8(4), 219–236.CrossRefGoogle Scholar
  46. Reed, M. S., Fraser, E. D. G., & Dougill, A. J. (2006). An adaptive learning process for developing and applying sustainability indicators with local communities. Ecological Economics, 59(4), 406–418.CrossRefGoogle Scholar
  47. Sadok, W., Angevin, F., Bergez, J.-E., Bockstaller, C., Colomb, B., Guichard, L., et al. (2009). MASC, a qualitative multi-attribute decision model for ex ante assessment of the sustainability of cropping systems. Agronomy for Sustainable Development, 29(3), 447–461. doi:10.1051/agro/2009006.CrossRefGoogle Scholar
  48. Schader, C., Grenz, J., Meier, M. S., & Stolze, M. (2014). Scope and precision of sustainability assessment approaches to food systems. Ecology and Society. doi:10.5751/ES-06866-190342.Google Scholar
  49. Schiere, J. B., Lyklema, J., Schakel, J., & Rickert, K. G. (1999). Evolution of farming systems and system philosophy. Systems Research and Behavioral Science, 16(4), 375–390.CrossRefGoogle Scholar
  50. Seimon, A., Plumptre, A. J., & Watson, J. E. M. (2012). Building consensus on Albertine Rift climate change adaptation for conservation: A report on 2011–2012 workshops in Uganda and Rwanda. WCS Workshop Report. New York, USA: Wildlife Conservation Society (WCS).Google Scholar
  51. Seimon, A., Yager, K., Seimon, T., Schmidt, S., Grau, A., Beck, S., et al. (2009). Changes in biodiversity patterns in the high andes—Understanding the consequences and seeking adaptation to global change. Mountain Forum Bulletin, 9, 25–27.Google Scholar
  52. Siegel, S. (1956). Nonparametric statistics for the behavioral sciences. New York, NY: McGraw Hill.Google Scholar
  53. Sommerville, M. M., Milner-Gulland, E., & Jones, J. P. (2011). The challenge of monitoring biodiversity in payment for environmental service interventions. Biological Conservation, 144(12), 2832–2841.CrossRefGoogle Scholar
  54. Steinfeld, H., Gerber, P., Wassenaar, T., Castel, V., Rosales, M., & Haan, C. D. (2006). Livestock’s long shadow: Environmental issues and options. Rome: Food and Agriculture Organization of the United Nations (FAO).Google Scholar
  55. Te Velde, H., Aarts, N., & Van Woerkum, C. (2002). Dealing with ambivalence: Farmers’ and consumers’ perceptions of animal welfare in livestock breeding. Journal of Agricultural and Environmental Ethics, 15(2), 203–219. doi:10.1023/A:1015012403331.CrossRefGoogle Scholar
  56. Triste, L., Marchand, F., Debruyne, L., Meul, M., & Lauwers, L. (2014). Reflection on the development process of a sustainability assessment tool: Learning from a Flemish case. Ecology and Society. doi:10.5751/ES-06789-190347.Google Scholar
  57. Yager, K., Ulloa, D., & Halloy, S. (2009). Chapter 16. Conducting an interdisciplinary workshop on climate change: Facilitating awareness and adaptation in Sajama National Park, Bolivia. (Interdisciplinary Aspects of Climate Change). Hamburg: Hamburg University of Applied Sciences.Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2016

Authors and Affiliations

  • Evelien M. de Olde
    • 1
    • 2
  • Henrik Moller
    • 3
  • Fleur Marchand
    • 4
    • 5
  • Richard W. McDowell
    • 6
    • 7
  • Catriona J. MacLeod
    • 8
  • Marion Sautier
    • 3
    • 9
  • Stephan Halloy
    • 10
    • 11
  • Andrew Barber
    • 12
  • Jayson Benge
    • 12
  • Christian Bockstaller
    • 13
    • 14
  • Eddie A. M. Bokkers
    • 2
  • Imke J. M. de Boer
    • 2
  • Katharine A. Legun
    • 15
  • Isabelle Le Quellec
    • 12
  • Charles Merfield
    • 16
  • Frank W. Oudshoorn
    • 1
    • 17
  • John Reid
    • 18
  • Christian Schader
    • 19
  • Erika Szymanski
    • 20
  • Claus A. G. Sørensen
    • 1
  • Jay Whitehead
    • 21
  • Jon Manhire
    • 12
  1. 1.Department of EngineeringAarhus UniversityAarhusDenmark
  2. 2.Animal Production Systems GroupWageningen UniversityWageningenThe Netherlands
  3. 3.Centre for Sustainability: Agriculture, Food, Energy, EnvironmentUniversity of OtagoDunedinNew Zealand
  4. 4.Social Sciences UnitInstitute for Agricultural and Fisheries Research (ILVO)MerelbekeBelgium
  5. 5.Ecosystem Management Research Group and IMDOUniversity of AntwerpWilrijkBelgium
  6. 6.Invermay Agricultural CentreAgResearchMosgielNew Zealand
  7. 7.Agriculture and Life SciencesLincoln UniversityLincolnNew Zealand
  8. 8.Landcare ResearchDunedinNew Zealand
  9. 9.INRA, UMR 1248 AGIRCastanet-TolosanFrance
  10. 10.Universidad Nacional de ChilecitoLa RiojaArgentina
  11. 11.Ministry for Primary IndustriesWellingtonNew Zealand
  12. 12.The Agribusiness GroupLincoln UniversityLincolnNew Zealand
  13. 13.INRA, UMR 1121 Agronomie et EnvironnementINRA-Université de LorraineColmar CedexFrance
  14. 14.UMR 1121, Agronomie et EnvironnementUniversité de LorraineColmar CedexFrance
  15. 15.Department of Sociology, Gender and Social WorkUniversity of OtagoDunedinNew Zealand
  16. 16.The BHU Future Farming CentreLincolnNew Zealand
  17. 17.SEGESAarhus NDenmark
  18. 18.Ngai Tahu Research CentreUniversity of CanterburyChristchurchNew Zealand
  19. 19.Research Institute of Organic Agriculture (FiBL)FrickSwitzerland
  20. 20.Centre for Science CommunicationUniversity of OtagoDunedinNew Zealand
  21. 21.Agribusiness and Economics Research UnitLincoln UniversityLincolnNew Zealand

Personalised recommendations