Environmental Management

, 44:1180 | Cite as

Evaluating Participatory Modeling: Developing a Framework for Cross-Case Analysis

  • Natalie A. JonesEmail author
  • Pascal Perez
  • Thomas G. Measham
  • Gail J. Kelly
  • Patrick d’Aquino
  • Katherine A. Daniell
  • Anne Dray
  • Nils Ferrand


Participatory modeling is increasingly recognized as an effective way to assist collective decision-making processes in the domain of natural resource management. This article introduces a framework for evaluating projects that have adopted a participatory modeling approach. This evaluation framework—known as the “Protocol of Canberra”—was developed through a collaboration between French and Australian researchers engaged in participatory modeling and evaluation research. The framework seeks to assess the extent to which different participatory modeling initiatives not only modify perceptions among and interactions between participants, but also contribute to collective decision-making. The article discusses the development of the framework and it’s application to three case-studies, two from Australia and one from the Pacific Island of the Republic of Kiribati. The article concludes with some comments for future use of the framework in a range of participatory modeling contexts.


Participation Modeling Evaluation Complex systems science 



This research was supported by CIRAD (France) and CSIRO-CSS (Australia). The evaluation framework and case studies are part of an international project called ADD-ComMod, led by INRA (France) and funded by the Agence Nationale de Recherche (France). Thanks to Russell Goddard for facilitating the Catalyst study evaluation, Pieter Bots for assistance with Fig. 1, and Rachel Williams and Kostas Alexandridis for helpful comments on the draft manuscript. Thank you especially to the participants and project teams of the case studies, who gave their time to complete the evaluation procedures.


  1. ANR (2005) Programme Federateur “Agriculture et Developpement Durable”:13-14. Online [URL]:
  2. Argyris C (1999) On organizational learning. Wiley-Blackwell, Oxford, UK, 464 ppGoogle Scholar
  3. Barreteau O, Le Page C, Perez P (2007) Contribution of simulation and gaming to natural resource management issues: an introduction. Simulation and Gaming 38(2):185CrossRefGoogle Scholar
  4. Beuret JE (2006) La Conduite de la Concertation pour la Gestion de l’Environnement et le Partage des Ressources. Editions L’Harmattan, Paris, 342 ppGoogle Scholar
  5. Blackstock KL, Kelly GJ, Horsey BL (2007) Developing and applying a framework to evaluate participatory research for sustainability. Ecological Economics 60(4):726–742CrossRefGoogle Scholar
  6. Bousquet F, Barreteau O, D’Aquino P, Etienne M, Boissau S, Aubert S, Le page C, Babin D, Castella J-C (2002) Multi-agent systems and role games: collective learning processes for ecosystem management. In: Janssen MA (ed) Complexity and ecosystem management: the theory and practice of mulit-agent systems. Edward Elgar Publishers, pp 248–285Google Scholar
  7. Brown V, Pitcher J (2005) Linking community and government: islands and beaches. In: Keen M, Brown V, Dyball R (eds) Social learning in environmental management: towards a sustainable future. James & James/Earthscan, London, pp 123–145Google Scholar
  8. Checkland P (1981) Systems thinking, systems practice. Wiley, Chichester, UKGoogle Scholar
  9. Colella V (2000) Participatory simulations: building collaborative understanding through immersive dynamic modeling. Journal of the Learning Sciences 9(4):471–500CrossRefGoogle Scholar
  10. ComMod Group (2003) Our companion modelling. Journal of Artificial Societies & Social Simulation 6(1). http://jasss,
  11. Curnan S, LaCava L, Sharpstee D, Lelle M, Reece M (1998) W.K Kellogg Foundation evaluation handbook. Online [URL]:
  12. Daniell KA (2007) Summary report: stakeholder workshops 2 & 3 for the lower hawkesbury estuary management plan. LHEMP project report prepared for the Hornsby Shire Council and BMT WBM, Fenner School, Australian National University, p 76Google Scholar
  13. Daniell KA, Ferrand N (2006) Participatory modelling for water resources management and planning, D3.8.2, Aquastress IP, FP6, EuropeGoogle Scholar
  14. Dray A, Perez P, Jones N, Le Page C, D’Aquino P, White I, Auatabu T (2006) The AtollGame experience: from knowledge engineering to a computer-assisted role playing game. Journal of Artificial Societies and Social Simulation 9(1). Online [URL]:
  15. Dray A, Perez P, Le Page C, D’Aquino P, White I (2007) Who wants to terminate the game? The role of vested interests and metaplayers in the ATOLLGAME experience. Simulation and Gaming, May 2007Google Scholar
  16. Goddard R (2005) Central coast regional futures planning project: a research partnership to help the communities and governments of the central coast (NSW) develop a capacity for evidence-based Strategic Planning Final Report. CSIRO Sustainable Ecosystems, CanberraGoogle Scholar
  17. Hare M, Letcher RA, Jakeman AJ (2003) Participatory modelling in natural resource management: a comparison of four case studies, vol 4(2). Taylor & Francis, pp 62–72Google Scholar
  18. Hisschemoller R, Tol RSJ, Vellinga P (2001) The relevance of participatory approaches in integrated environmental assessment. Integrated Assessment 2:57–72CrossRefGoogle Scholar
  19. Hutchins E (1995) Cognition in the Wild (ISBN 0-262-58146-9). MIT PressGoogle Scholar
  20. Kelly G, Walker P (2004) Community-level systems thinking. In: 2nd international conference of the system dynamics society, OxfordGoogle Scholar
  21. Lynam T, de Jong W, Sheil W, Kusumanto T, Evans K (2007) A review of tools for incorporating community knowledge, preferances and values into decision making in natural resource management. Ecology and Society 12(1). Online [URL]:
  22. Maurel P, Flavie C, Ferrand N, Marc C, Valkering P (2004) Some methodological concepts to analyse the role of IC-tools in social learning processes. HarmoniCOP Work Package 3. International Environmental Modeling and Software Society (iEMSs). Online [URL]:
  23. Mermet L, Billé R, Leroy M, Narcy J-B, Poux X (2005) L’analyse stratégique de la gestion environnementale: un cadre théorique pour penser l’efficacité en matière d’environnement. Natures Sciences Sociétés 13:127–137CrossRefGoogle Scholar
  24. Midgley G (2000) Systemic intervention: philosophy, methodology and practice. Kluwer Academic, New York, USA, 468 ppGoogle Scholar
  25. Mostert E, Pahl-Wostl C, Rees Y, Searle B, Tabara D, Tippett J (2007) Social learning in European river-basin management: barriers and fosterin mechanisms from 10 river basin. Ecology and Society 12(1):19Google Scholar
  26. Pahl-Wostl C (2002) Participative and stakeholder-based policy design, evaluation and modeling processes. Integrated Assessment 3(1):3–14CrossRefGoogle Scholar
  27. Patton MQ (1992) Qualitative evaluation and research methods, 3rd edn. Sage Publications, Newbury Park, CaliforniaGoogle Scholar
  28. Reeson AF, Tisdell JG (2008) Institutions, motivations and public goods: An experimental test of motivational crowding. Journal of Economic Behavior & Organization, 68(1):273–281CrossRefGoogle Scholar
  29. Rouwette EAJA, Vennix J, van Mullekom T (2002) Group model building effectiveness: a review of assessment studies. System Dynamics Review 18(1):5–45CrossRefGoogle Scholar
  30. Siebenhuner V, Barth V (2005) The role of computer modeling in participatory integrated assessments. Environmental Impact Assessment Review 25:367–389CrossRefGoogle Scholar
  31. Standards Australia (2006). Environmental risk management—principles and processes. Standards Australia, HB 203:2006, Sydney, AustraliaGoogle Scholar
  32. Van den Belt M (2004) Mediated modeling: a systems dynamics approach to environmental consensus building. Washington, Island PressGoogle Scholar
  33. Webler T (1999) The craft and theory of public participation: a dialectical process. Journal of Risk Research 2(1):55–71CrossRefGoogle Scholar
  34. World Bank Operations Evaluation Department (2004) Monitoring and evaluation: some tools, methods and approaches. The World Bank, Washington, DC. Online [URL]:
  35. Zuber-Skerritt O (1992) Professional development in higher education: a theoretical framework for action research. Kogan Page, LondonGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  • Natalie A. Jones
    • 1
    Email author
  • Pascal Perez
    • 2
    • 3
  • Thomas G. Measham
    • 4
  • Gail J. Kelly
    • 5
  • Patrick d’Aquino
    • 3
  • Katherine A. Daniell
    • 2
    • 6
  • Anne Dray
    • 2
  • Nils Ferrand
    • 6
  1. 1.School of Integrative SystemsUniversity of QueenslandBrisbaneAustralia
  2. 2.RMAP/RSPAS, Australian National UniversityCanberra, Australian Capital TerritoryAustralia
  3. 3.CIRAD-UPR-GREENMontpellierFrance
  4. 4.Resource Futures Program, CSIRO Sustainable EcosystemsCanberra, Australian Capital TerritoryAustralia
  5. 5.Bureau of Rural SciencesCanberra, Australian Capital TerritoryAustralia
  6. 6.CEMAGREF-UMR-GEAUMontpellierFrance

Personalised recommendations