Skip to main content

Advertisement

SpringerLink
Log in

When experts disagree: the need to rethink indicator selection for assessing sustainability of agriculture

  • Published:
Environment, Development and Sustainability Aims and scope Submit manuscript

Abstract

Sustainability indicators are well recognized for their potential to assess and monitor sustainable development of agricultural systems. A large number of indicators are proposed in various sustainability assessment frameworks, which raises concerns regarding the validity of approaches, usefulness and trust in such frameworks. Selecting indicators requires transparent and well-defined procedures to ensure the relevance and validity of sustainability assessments. The objective of this study, therefore, was to determine whether experts agree on which criteria are most important in the selection of indicators and indicator sets for robust sustainability assessments. Two groups of experts (Temperate Agriculture Research Network and New Zealand Sustainability Dashboard) were asked to rank the relative importance of eleven criteria for selecting individual indicators and of nine criteria for balancing a collective set of indicators. Both ranking surveys reveal a startling lack of consensus amongst experts about how best to measure agricultural sustainability and call for a radical rethink about how complementary approaches to sustainability assessments are used alongside each other to ensure a plurality of views and maximum collaboration and trust amongst stakeholders. To improve the transparency, relevance and robustness of sustainable assessments, the context of the sustainability assessment, including prioritizations of selection criteria for indicator selection, must be accounted for. A collaborative design process will enhance the acceptance of diverse values and prioritizations embedded in sustainability assessments. The process by which indicators and sustainability frameworks are established may be a much more important determinant of their success than the final shape of the assessment tools. Such an emphasis on process would make assessments more transparent, transformative and enduring.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Agrawal, A. (2005). Environmentality. Technologies of government and the making of subjects. Durham and London: Duke University Press.

    Google Scholar 

  • Alrøe, H. F., & Kristensen, E. S. (2002). Towards a systemic research methodology in agriculture: Rethinking the role of values in science. Agriculture and Human Values, 19(1), 3–23.

    Article  Google Scholar 

  • Alrøe, H. F., Moller, H., Læssøe, J., & Noe, E. (2016). Opportunities and challenges for multicriteria assessment of food system sustainability. Ecology and Society. doi:10.5751/ES-08394-210138.

    Google Scholar 

  • Ansell, C., & Gash, A. (2008). Collaborative governance in theory and practice. Journal of Public Administration Research and Theory, 18(4), 543–571. doi:10.1093/jopart/mum032.

    Article  Google Scholar 

  • Bäckstrand, K. (2004). Scientisation vs. civic expertise in environmental governance: Eco-feminist, eco-modern and post-modern responses. Environmental Politics, 13(4), 695–714. doi:10.1080/0964401042000274322.

    Article  Google Scholar 

  • Barnaud, C., & Van Paassen, A. (2013). Equity, power games, and legitimacy: Dilemmas of participatory natural resource management. Ecology and Society. doi:10.5751/ES-05459-180221.

    Google Scholar 

  • Bell, S., & Morse, S. (2008). Sustainability indicators: Measuring the immeasurable?. London: Earthscan.

    Google Scholar 

  • Belt, M., & Blake, D. (2015). Mediated modeling in water resource dialogues connecting multiple scales. JAWRA Journal of the American Water Resources Association, 51(6), 1581–1599.

    Article  Google Scholar 

  • Binder, C. R., Feola, G., & Steinberger, J. K. (2010). Considering the normative, systemic and procedural dimensions in indicator-based sustainability assessments in agriculture. Environmental Impact Assessment Review, 30(2), 71–81.

    Article  Google Scholar 

  • Bockstaller, C., Feschet, P., & Angevin, F. (2015). Issues in evaluating sustainability of farming systems with indicators. OCL Oilseeds and Fats, Crops and Lipids. doi:10.1051/ocl/2014052.

    Google Scholar 

  • Bockstaller, C., Guichard, L., Keichinger, O., Girardin, P., Galan, M. B., & Gaillard, G. (2009). Comparison of methods to assess the sustainability of agricultural systems. A review. Agronomy for Sustainable Development, 29(1), 223–235.

    Article  Google Scholar 

  • Cloquell-Ballester, V.-A., Cloquell-Ballester, V.-A., Monterde-Díaz, R., & Santamarina-Siurana, M.-C. (2006). Indicators validation for the improvement of environmental and social impact quantitative assessment. Environmental Impact Assessment Review, 26(1), 79–105. doi:10.1016/j.eiar.2005.06.002.

    Article  Google Scholar 

  • Dale, V. H., & Beyeler, S. C. (2001). Challenges in the development and use of ecological indicators. Ecological Indicators, 1(1), 3–10.

    Article  Google Scholar 

  • De Mey, K., D’Haene, K., Marchand, F., Meul, M., & Lauwers, L. (2011). Learning through stakeholder involvement in the implementation of MOTIFS: An integrated assessment model for sustainable farming in Flanders. International Journal of Agricultural Sustainability, 9(2), 350–363.

    Google Scholar 

  • de Olde, E. M., Oudshoorn, F. W., Sørensen, C. A. G., Bokkers, E. A. M., & de Boer, I. J. M. (2016). Assessing sustainability at farm-level: Lessons learned from a comparison of tools in practice. Ecological Indicators, 66, 391–404. doi:10.1016/j.ecolind.2016.01.047.

    Article  Google Scholar 

  • Dovers, S. (2005). Clarifying the imperative of integration research for sustainable environmental management. Journal of Research Practice, 1(2), 1–19.

    Google Scholar 

  • Elsaesser, M., Jilg, T., Herrmann, K., Boonen, J., Debruyne, L., Laidlaw, A. S., et al. (2015). Quantifying sustainability of dairy farms with the DAIRYMAN sustainability-index. In: Paper presented at the European Grassland Federation, Wageningen, The Netherlands.

  • FAO. (2013). Sustainability assessment of food and agriculture systems (SAFA): Guidelines, version 3.0. Rome: Food and Agricultural Organization of the United Nations.

    Google Scholar 

  • Ferraro, P. J., & Pattanayak, S. K. (2006). Money for nothing? A call for empirical evaluation of biodiversity conservation investments. PLoS Biology, 4(4), e105. doi:10.1371/journal.pbio.0040105.

    Article  Google Scholar 

  • Gasparatos, A. (2010). Embedded value systems in sustainability assessment tools and their implications. Journal of Environmental Management, 91(8), 1613–1622. doi:10.1016/j.jenvman.2010.03.014.

    Article  Google Scholar 

  • Gasso, V., Oudshoorn, F. W., de Olde, E., & Sørensen, C. A. G. (2015). Generic sustainability assessment themes and the role of context: The case of Danish maize for German biogas. Ecological Indicators, 49, 143–153. doi:10.1016/j.ecolind.2014.10.008.

    Article  Google Scholar 

  • Gibbons, J. D., & Chakraborti, S. (2011). Nonparametric statistical inference. Berlin: Springer.

    Book  Google Scholar 

  • Hansen, J. (1996). Is agricultural sustainability a useful concept? Agricultural Systems, 50(2), 117–143.

    Article  Google Scholar 

  • Herzog, F., Balázs, K., Dennis, P., Friedel, J., Geijzendorffer, I., Jeanneret, P., et al. (2012). Biodiversity indicators for European farming systems: A guidebook: Forschungsanstalt Agroscope Reckenholz-Tänikon ART.

  • IAASTD. (2009). Agriculture at a crossroads: Synthesis report. International Assessment of Agricultural Knowledge, Science and Technology for Development (IAASTD) Science and Technology for Development. Island Press.

  • Jones, C., Cowan, P., & Allen, W. (2012). Setting outcomes, and measuring and reporting performance of regional council pest and weed management programmes. Guidelines and resource materials. Landcare Research Contract Report LC144: Landcare Research New Zealand Ltd.

  • Kates, R. W., Clark, W. C., Corell, R., Hall, J. M., Jaeger, C. C., Lowe, I., et al. (2001). Environment and development: Sustainability science. Science, 292(5517), 641–642. doi:10.1126/science.1059386.

    Article  CAS  Google Scholar 

  • Kendall, M. G., & Smith, B. B. (1939). The problem of m rankings. The Annals of Mathematical Statistics, 10(3), 275–287.

    Article  Google Scholar 

  • Keulen, H. V., van Ittersum, M., & Leffelaar, P. (2005). Multiscale methodological framework to derive criteria and indicators for sustainability evaluation of peasant natural resource management systems. Environment, Development and Sustainability, 7(1), 51–69.

    Article  Google Scholar 

  • Komiyama, H., & Takeuchi, K. (2006). Sustainability science: Building a new discipline. Sustainability Science, 1(1), 1–6. doi:10.1007/s11625-006-0007-4.

    Article  Google Scholar 

  • Lebacq, T., Baret, P. V., & Stilmant, D. (2013). Sustainability indicators for livestock farming. A review. Agronomy for Sustainable Development, 33(2), 311–327.

    Article  Google Scholar 

  • Lee, W., McGlone, M., & Wright, E. (2005). Biodiversity inventory and monitoring: A review of national and international systems and a proposed framework for future biodiversity monitoring by the Department of Conservation. Landcare Research Contract Report LC0405/122.

  • Lupia, A. (2013). Communicating science in politicized environments. Proceedings of the National Academy of Sciences, 110(Supplement 3), 14048–14054. doi:10.1073/pnas.1212726110.

    Article  CAS  Google Scholar 

  • Marchand, F., Debruyne, L., Triste, L., Gerrard, C., Padel, S., & Lauwers, L. (2014). Key characteristics for tool choice in indicator-based sustainability assessment at farm level. Ecology and Society. doi:10.5751/ES-06876-190346.

    Google Scholar 

  • Merfield, C., Moller, H., Manhire, J., Rosin, C., Norton, S., Carey, P., et al. (2015). Are organic standards sufficient to ensure sustainable agriculture? Lessons from New Zealand’s ARGOS and Sustainability Dashboard projects. Sustainable Agriculture Research, 4(3), p158.

    Article  Google Scholar 

  • Moller, H., & MacLeod, C. J. (2013). Design criteria for effective assessment of sustainability in New Zealand’s production landscapes. (Vol. 13/07, pp. 73): NZ Sustainability Dashboard Research Report.

  • Moller, H., O’Blyver, P., Bragg, C., Newman, J., Clucas, R., Fletcher, D., et al. (2009). Guidelines for cross-cultural participatory action research partnerships: A case study of a customary seabird harvest in New Zealand. New Zealand Journal of Zoology, 36(3), 211–241. doi:10.1080/03014220909510152.

    Article  Google Scholar 

  • Niemeijer, D., & de Groot, R. S. (2008). A conceptual framework for selecting environmental indicator sets. Ecological Indicators, 8(1), 14–25. doi:10.1016/j.ecolind.2006.11.012.

    Article  Google Scholar 

  • OECD. (2001). Environmental indicators for agriculture. Methods and results (Vol. 3). Paris: Organisation for Economic Co-operation and Development.

    Google Scholar 

  • Ostrom, E. (2009). A general framework for analyzing sustainability of social-ecological systems. Science, 325(5939), 419–422. doi:10.1126/science.1172133.

    Article  CAS  Google Scholar 

  • Owens, S. (2003). Is there a meaningful definition of sustainability? Plant Genetic Resources: Characterization and Utilization, 1(01), 5–9.

    Article  Google Scholar 

  • Parris, T. M., & Kates, R. W. (2003). Characterizing and measuring sustainable development. Annual Review of Environment and Resources, 28, 559–586.

    Article  Google Scholar 

  • Popa, F., Guillermin, M., & Dedeurwaerdere, T. (2015). A pragmatist approach to transdisciplinarity in sustainability research: From complex systems theory to reflexive science. Futures. doi:10.1016/j.futures.2014.02.002.

    Google Scholar 

  • Pretty, J. (2008). Agricultural sustainability: Concepts, principles and evidence. Philosophical Transactions of the Royal Society B: Biological Sciences, 363(1491), 447–465.

    Article  Google Scholar 

  • Pretty, J., Sutherland, W. J., Ashby, J., Auburn, J., Baulcombe, D., Bell, M., et al. (2010). The top 100 questions of importance to the future of global agriculture. International Journal of Agricultural Sustainability, 8(4), 219–236.

    Article  Google Scholar 

  • Reed, M. S., Fraser, E. D. G., & Dougill, A. J. (2006). An adaptive learning process for developing and applying sustainability indicators with local communities. Ecological Economics, 59(4), 406–418.

    Article  Google Scholar 

  • Sadok, W., Angevin, F., Bergez, J.-E., Bockstaller, C., Colomb, B., Guichard, L., et al. (2009). MASC, a qualitative multi-attribute decision model for ex ante assessment of the sustainability of cropping systems. Agronomy for Sustainable Development, 29(3), 447–461. doi:10.1051/agro/2009006.

    Article  Google Scholar 

  • Schader, C., Grenz, J., Meier, M. S., & Stolze, M. (2014). Scope and precision of sustainability assessment approaches to food systems. Ecology and Society. doi:10.5751/ES-06866-190342.

    Google Scholar 

  • Schiere, J. B., Lyklema, J., Schakel, J., & Rickert, K. G. (1999). Evolution of farming systems and system philosophy. Systems Research and Behavioral Science, 16(4), 375–390.

    Article  Google Scholar 

  • Seimon, A., Plumptre, A. J., & Watson, J. E. M. (2012). Building consensus on Albertine Rift climate change adaptation for conservation: A report on 2011–2012 workshops in Uganda and Rwanda. WCS Workshop Report. New York, USA: Wildlife Conservation Society (WCS).

  • Seimon, A., Yager, K., Seimon, T., Schmidt, S., Grau, A., Beck, S., et al. (2009). Changes in biodiversity patterns in the high andes—Understanding the consequences and seeking adaptation to global change. Mountain Forum Bulletin, 9, 25–27.

    Google Scholar 

  • Siegel, S. (1956). Nonparametric statistics for the behavioral sciences. New York, NY: McGraw Hill.

    Google Scholar 

  • Sommerville, M. M., Milner-Gulland, E., & Jones, J. P. (2011). The challenge of monitoring biodiversity in payment for environmental service interventions. Biological Conservation, 144(12), 2832–2841.

    Article  Google Scholar 

  • Steinfeld, H., Gerber, P., Wassenaar, T., Castel, V., Rosales, M., & Haan, C. D. (2006). Livestock’s long shadow: Environmental issues and options. Rome: Food and Agriculture Organization of the United Nations (FAO).

    Google Scholar 

  • Te Velde, H., Aarts, N., & Van Woerkum, C. (2002). Dealing with ambivalence: Farmers’ and consumers’ perceptions of animal welfare in livestock breeding. Journal of Agricultural and Environmental Ethics, 15(2), 203–219. doi:10.1023/A:1015012403331.

    Article  Google Scholar 

  • Triste, L., Marchand, F., Debruyne, L., Meul, M., & Lauwers, L. (2014). Reflection on the development process of a sustainability assessment tool: Learning from a Flemish case. Ecology and Society. doi:10.5751/ES-06789-190347.

    Google Scholar 

  • Yager, K., Ulloa, D., & Halloy, S. (2009). Chapter 16. Conducting an interdisciplinary workshop on climate change: Facilitating awareness and adaptation in Sajama National Park, Bolivia. (Interdisciplinary Aspects of Climate Change). Hamburg: Hamburg University of Applied Sciences.

Download references

Acknowledgments

We would like to thank all the participants from TempAg and NZSD for their participation. We were also grateful for guidance and permission from Paul Hansen and Franz Ombler for deploying the 1000Minds software. We would like to acknowledge Peter Groffman and the anonymous reviewers for their constructive suggestions on an earlier version of this paper. This is the first paper of the Resilient Agricultural Production Systems team of the Temperate Agriculture Research Network and international collaboration initiated by OECD’s Global Science Forum. NZSD’s participation is funded by New Zealand’s Ministry for Business, Innovation and Employment (contract AGRB1201).

Author information

Authors and Affiliations

  1. Department of Engineering, Aarhus University, Inge Lehmanns Gade 10, 8000, Aarhus, Denmark

    Evelien M. de Olde, Frank W. Oudshoorn & Claus A. G. Sørensen

  2. Animal Production Systems Group, Wageningen University, Wageningen, The Netherlands

    Evelien M. de Olde, Eddie A. M. Bokkers & Imke J. M. de Boer

  3. Centre for Sustainability: Agriculture, Food, Energy, Environment, University of Otago, Dunedin, New Zealand

    Henrik Moller & Marion Sautier

  4. Social Sciences Unit, Institute for Agricultural and Fisheries Research (ILVO), Merelbeke, Belgium

    Fleur Marchand

  5. Ecosystem Management Research Group and IMDO, University of Antwerp, Wilrijk, Belgium

    Fleur Marchand

  6. Invermay Agricultural Centre, AgResearch, Mosgiel, New Zealand

    Richard W. McDowell

  7. Agriculture and Life Sciences, Lincoln University, Lincoln, New Zealand

    Richard W. McDowell

  8. Landcare Research, Dunedin, New Zealand

    Catriona J. MacLeod

  9. INRA, UMR 1248 AGIR, Castanet-Tolosan, France

    Marion Sautier

  10. Universidad Nacional de Chilecito, La Rioja, Argentina

    Stephan Halloy

  11. Ministry for Primary Industries, Wellington, New Zealand

    Stephan Halloy

  12. The Agribusiness Group, Lincoln University, Lincoln, New Zealand

    Andrew Barber, Jayson Benge, Isabelle Le Quellec & Jon Manhire

  13. INRA, UMR 1121 Agronomie et Environnement, INRA-Université de Lorraine, BP20507, Colmar Cedex, France

    Christian Bockstaller

  14. UMR 1121, Agronomie et Environnement, Université de Lorraine, BP20507, Colmar Cedex, France

    Christian Bockstaller

  15. Department of Sociology, Gender and Social Work, University of Otago, Dunedin, New Zealand

    Katharine A. Legun

  16. The BHU Future Farming Centre, Lincoln, New Zealand

    Charles Merfield

  17. SEGES, Aarhus N, Denmark

    Frank W. Oudshoorn

  18. Ngai Tahu Research Centre, University of Canterbury, Christchurch, New Zealand

    John Reid

  19. Research Institute of Organic Agriculture (FiBL), Frick, Switzerland

    Christian Schader

  20. Centre for Science Communication, University of Otago, Dunedin, New Zealand

    Erika Szymanski

  21. Agribusiness and Economics Research Unit, Lincoln University, Lincoln, New Zealand

    Jay Whitehead

Authors
  1. Evelien M. de Olde
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Henrik Moller
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fleur Marchand
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Richard W. McDowell
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Catriona J. MacLeod
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Marion Sautier
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Stephan Halloy
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Andrew Barber
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Jayson Benge
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Christian Bockstaller
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Eddie A. M. Bokkers
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Imke J. M. de Boer
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Katharine A. Legun
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Isabelle Le Quellec
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Charles Merfield
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Frank W. Oudshoorn
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. John Reid
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. Christian Schader
    View author publications

    You can also search for this author in PubMed Google Scholar

  19. Erika Szymanski
    View author publications

    You can also search for this author in PubMed Google Scholar

  20. Claus A. G. Sørensen
    View author publications

    You can also search for this author in PubMed Google Scholar

  21. Jay Whitehead
    View author publications

    You can also search for this author in PubMed Google Scholar

  22. Jon Manhire
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Evelien M. de Olde.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

de Olde, E.M., Moller, H., Marchand, F. et al. When experts disagree: the need to rethink indicator selection for assessing sustainability of agriculture. Environ Dev Sustain 19, 1327–1342 (2017). https://doi.org/10.1007/s10668-016-9803-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10668-016-9803-x

Keywords

Search

Navigation