Evaluating Complexity

  • Andrew MitchellEmail author


The findings from evaluations of community-based sustainability and international developmental aid initiatives are generally disappointing. Mitchell reviews the literature and concludes that the findings converge around an instrumental critique, which holds that the monitoring and evaluation frameworks and indicators are inappropriate or inadequate and that project practitioners are ill-equipped to collect, analyse and manage evaluation data, and a design error critique, which suggests that projects deployed to address complex and wicked problems are predicated on a linear design model. Mitchell introduces and discusses the developmental evaluation approach that supports practitioners to draw from their project-based experience to innovate and respond adaptively to complex operating conditions.


Wicked problems Community-based sustainability International developmental aid Monitoring and evaluation frameworks Indicators Developmental evaluation 


  1. Abrahamse, W., Steg, L., Vlek, C., & Rothengatter, T. (2005). A review of intervention studies aimed at household energy conservation. Journal of Environmental Psychology, 25(3), 273–291. Scholar
  2. Agol, D., Latawiec, A. E., & Strassburg, B. B. N. (2014). Evaluating impacts of development and conservation projects using sustainability indicators: Opportunities and challenges. Environmental Impact Assessment Review, 48, 1–9. Scholar
  3. Allen, P. M. (2010). What is the science of complexity? Knowledge of the limits to knowledge. In A. Tait & K. A. Richardson (Eds.), Complexity and knowledge management: Understanding the role of knowledge in the management of social networks (pp. 3–22). Charlotte, NC: Information Age Publishers.Google Scholar
  4. Argyris, C., & Schön, D. (1978). Organizational learning: A theory of action perspective. Cambridge, MA: Addison-Wesley.Google Scholar
  5. Arvesen, A., Bright, R. M., & Hertwich, E. G. (2011). Considering only first-order effects? How simplifications lead to unrealistic technology optimism in climate change mitigation. Energy Policy, 39(11), 7448–7454. Scholar
  6. Bamberg, S., & Möser, G. (2007). Twenty years after Hines, Hungerford, and Tomera: A new meta-analysis of psycho-social determinants of pro-environmental behaviour. Journal of Environmental Psychology, 27(1), 14–25. Scholar
  7. Bateson, G. (1979). Mind and nature: A necessary unity. New York: Bantam.Google Scholar
  8. Bell, S., & Coudert, E. (2005). A practitioner’s guide to ‘Imagine’, the systemic and prospective sustainability analysis. Valbonne. Available at:
  9. Bell, S., & Morse, S. (2008). Sustainability indicators: Measuring the immeasurable? (Revised). London: Earthscan.Google Scholar
  10. Bell, S., & Morse, S. (2010). Triple task method: Systemic, reflective action research. Systemic Practice and Action Research, 23(6), 443–452. Scholar
  11. Bell, S. J., & Wilby, J. M. (2012). Editorial: The pattern that connects and the difference that makes a difference. Emergence: Complexity and Organization, 14(1), vii–x.Google Scholar
  12. Blühdorn, I. (2007). Sustaining the unsustainable: Symbolic politics and the politics of simulation. Environmental Politics, 16(2), 251–275. Scholar
  13. Blunden, J., & Arndt, D. S. (Eds.). (2016). State of the climate in 2015. Bulletin of the American Meteorogical Society, 97(8), S1–S275.Google Scholar
  14. Bolderdijk, J. W., Steg, L., Geller, E. S., Lehman, P. K., & Postmes, T. (2012). Comparing the effectiveness of monetary versus moral motives in environmental campaigning. Nature Climate Change, 3(4), 413–416. Scholar
  15. Bond, A. J., & Morrison-Saunders, A. (2009). Sustainability appraisal: Jack of all trades, master of none? Impact Assessment and Project Appraisal, 27(4), 321–329. Scholar
  16. Bradbury, S., & Middlemiss, L. (2015). The role of learning in sustainable communities of practice. Local Environment, 20(7), 796–810. Scholar
  17. Brulle, R. J. (2013). Institutionalizing delay: Foundation funding and the creation of U.S. climate change counter-movement organizations. Climatic Change, 122(4), 681–694. Scholar
  18. Burns, D., & Worsley, S. (2015). Navigating complexity in international development: Facilitating sustainable change at scale. Rugby: Practical Action Publishing Ltd.CrossRefGoogle Scholar
  19. Centre for the Evaluation of Complexity Across the Nexus [CECAN]. (2018). Policy evaluation for a complex world. Guildford. Available at:
  20. Cinderby, S., Haq, G., Cambridge, H., & Lock, K. (2014). Practical action to build community resilience: The Good Life initiative in New Earswick. Joseph Rowntree Foundation. Available at:
  21. Dahl, A. L. (2012). Achievements and gaps in indicators for sustainability. Ecological Indicators, 17, 14–19. Scholar
  22. Dennard, L. F., Richardson, K. A., & Morçöl, G. (2008). Editorial. In L. F. Dennard, K. A. Richardson, & G. Morçöl (Eds.), Complexity and policy analysis: Tools and methods for designing robust policies in a complex world (pp. 1–18). Goodyear, AZ: ISCE Publishing.Google Scholar
  23. Department of Energy and Climate Change [DECC]. (2014). Community Energy Strategy: Full Report. London: Department of Energy and Climate Change.Google Scholar
  24. Dozois, E., Langlois, M., & Blanchet-Cohen, N. (2010). DE 201: A practitioner’s guide to developmental evaluation. Available at:
  25. Dunkley, R. A., & Franklin, A. (2017). Failing better: The stochastic art of evaluating community-led environmental action programs. Evaluation and Program Planning, 60, 112–122. Scholar
  26. Edwards-Jones, G., Milà i Canals, L., Hounsome, N., Truninger, M., Koerber, G., Hounsome, B., et al. (2008). Testing the assertion that “local food is best”: The challenges of an evidence-based approach. Trends in Food Science & Technology, 19(5), 265–274. Scholar
  27. Einsiedel, E. F., Boyd, A. D., Medlock, J., & Ashworth, P. (2013). Assessing socio-technical mindsets: Public deliberations on carbon capture and storage in the context of energy sources and climate change. Energy Policy, 53, 149–158. Scholar
  28. Faber, N., Jorna, R., & Van Engelen, J. (2005). The sustainability of “sustainability”—A study into the conceptual foundations of the notion of “sustainability”. Journal of Environmental Assessment Policy and Management, 7(1), 1–33. Scholar
  29. Fagen, M. C., Redman, S. D., Stacks, J., Barrett, V., Thullen, B., & Neiger, B. L. (2011). Developmental evaluation: Building innovations in complex environments. Health Promotion Practice, 12(5), 645–650. Scholar
  30. Feola, G., & Nunes, R. (2013). Failure and success of transition initiatives: A study of the international replication of the transition movement, Research note 4. Reading: Walker Institute for Climate System Research. Available at:
  31. Feola, G., & Nunes, R. (2014). Success and failure of grassroots innovations for addressing climate change: The case of the transition movement. Global Environmental Change, 24(1), 232–250. Scholar
  32. Flyvbjerg, B., Landman, T., & Schram, S. (2012). In B. Flyvbjerg, T. Landman, & S. Schram (Eds.), Real social science: Applied phronesis. New York: Cambridge University Press.Google Scholar
  33. Forrest, N., & Wiek, A. (2014). Learning from success—Toward evidence-informed sustainability transitions in communities. Environmental Innovation and Societal Transitions, 12, 66–88. Scholar
  34. Foster, J. (2015). After sustainability: Denial, hope, retrieval. Oxford: Earthscan.Google Scholar
  35. Fraser, E. D. G., Dougill, A. J., Mabee, W. E., Reed, M., & McAlpine, P. (2006). Bottom up and top down: Analysis of participatory processes for sustainability indicator identification as a pathway to community empowerment and sustainable environmental management. Journal of Environmental Management, 78(2), 114–127. Scholar
  36. Fricker, A. (1998). Measuring up to sustainability. Futures, 30(4), 367–375. Scholar
  37. Funtowicz, S., & Ravetz, J. (1993). Post-normal science. Futures, 25(7), 739–755. Scholar
  38. Gamble, J. A. A. (2008). A developmental evaluation primer. Available at:
  39. Garnåsjordet, P. A., Aslaksen, I., Giampietro, M., Funtowicz, S., & Ericson, T. (2012). Sustainable development indicators: From statistics to policy. Environmental Policy and Governance, 22(5), 322–336. Scholar
  40. Gates, E. F. (2016). Making sense of the emerging conversation in evaluation about systems thinking and complexity science. Evaluation and Program Planning, 59, 62–73. Scholar
  41. Geels, F. W. (2005). Processes and patterns in transitions and system innovations: Refining the co-evolutionary multi-level perspective. Technological Forecasting and Social Change, 72, 681–696. Scholar
  42. Geels, F. W., & Kemp, R. (2007). Dynamics in socio-technical systems: Typology of change processes and contrasting case studies. Technology in Society, 29(4), 441–455. Scholar
  43. Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., & Vermeersch, C. M. J. (2011). Impact evaluation in practice. Washington, DC: The International Bank for Reconstruction and Development/The World Bank. Scholar
  44. Giampietro, M., & Saltelli, A. (2014). Footprints to nowhere. Ecological Indicators, 46, 610–621. Scholar
  45. Gibson, R. (2005). Sustainability assessment: Criteria and processes. London: Earthscan.Google Scholar
  46. Gotts, N. M. (2007). Resilience, panarchy, and world-systems analysis. Ecology & Society, 12(1), 24. Available at:
  47. Gunderson, L. H., & Holling, C. S. (Eds.). (2002). Panarchy: Understanding transformations in human and natural systems. Washington, DC: Island Press.Google Scholar
  48. H.M. Treasury. (2011). The Magenta Book: Guidance for evaluation. London.Google Scholar
  49. H.M. Treasury. (2012). Quality in qualitative evaluation: A framework for assessing research evidence (supplemetary Magenta Book guidance). London.Google Scholar
  50. Hamilton, C. (2010). Consumerism, self-creation and prospects for a new ecological consciousness. Journal of Cleaner Production, 18(6), 571–575. Scholar
  51. Hansen, J. (2009). Storms of my grandchildren. London: Bloomsbury Publishing.Google Scholar
  52. Hansen, J., Sato, M., Hearty, P., Ruedy, R., Kelley, M., Masson-Delmotte, V., et al. (2015). Ice melt, sea level rise and superstorms: Evidence from paleoclimate data, climate modeling, and modern observations that 2°C global warming is highly dangerous. Atmospheric Chemistry and Physics Discussion, 15, 20059–20179. Scholar
  53. Hargreaves, T., Hielscher, S., Seyfang, G., & Smith, A. (2013). Grassroots innovations in community energy: The role of intermediaries in niche development. Global Environmental Change, 23(5), 868–880. Scholar
  54. Hayes, H., Witkowski, S., & Smith, L. (2016). Failing forward quickly as a developmental evaluator: Lessons from year one of the LiveWell Kershaw journey. Journal of MultiDisciplinary Evaluation, 12(27), 112–118.Google Scholar
  55. Hertwich, E. G. (2005). Consumption and the rebound effect: An industrial ecology perspective. Journal of Industrial Ecology, 9(1–2), 85–98.Google Scholar
  56. Hertwich, E. G., & Peters, G. P. (2009). Carbon footprint of nations: A global, trade-linked analysis. Environmental Science and Technology, 43(16), 6414–6420. Scholar
  57. Hezri, A. A., & Dovers, S. R. (2006). Sustainability indicators, policy and governance: Issues for ecological economics. Ecological Economics, 60(1), 86–99. Scholar
  58. Hobson, K., Hamilton, J., & Mayne, R. (2014). Monitoring and evaluation in UK low-carbon community groups: Benefits, barriers and the politics of the local. Local Environment, 21(1), 124–136. Scholar
  59. Hobson, K., Mayne, R., & Hamilton, J. (2016). Monitoring and evaluating eco-localisation: Lessons from UK low carbon community groups. Environment and Planning A (Forthcoming). Scholar
  60. Hoffman, S. M., & High-Pippert, A. (2010). From private lives to collective action: Recruitment and participation incentives for a community energy program. Energy Policy, 38(12), 7567–7574. Scholar
  61. Ika, L. A. (2012). Project management for development in Africa: Why projects are failing and what can be done about it. Project Management Journal, 43(4), 27–41. Scholar
  62. Ika, L. A., & Donnelly, J. (2017). Success conditions for international development capacity building projects. International Journal of Project Management, 35(1), 44–63. Scholar
  63. Jacques, P. J., Dunlap, R. E., & Freeman, M. (2008). The organisation of denial: Conservative think tanks and environmental scepticism. Environmental Politics, 17(3), 349–385. Scholar
  64. Kahneman, D. (2011). Thinking, fast and slow. London: Penguin Books.Google Scholar
  65. Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decision under risk. Econometrica: Journal of the Econometric Society, 47(3), 263–291. Scholar
  66. Kern, F. (2012). Using the multi-level perspective on socio-technical transitions to assess innovation policy. Technological Forecasting and Social Change, 79(2), 298–310. Scholar
  67. Knight, A. T., Cowling, R. M., Rouget, M., Balmford, A., Lombard, A. T., & Campbell, B. M. (2008). Knowing but not doing: Selecting priority conservation areas and the research-implementation gap. Conservation Biology, 22(3), 610–617. Scholar
  68. Kollmuss, A., & Agyeman, J. (2002). Mind the gap: Why do people act environmentally and what are the barriers to pro-environmental behavior? Environmental Education Research, 8(3), 239–260. Scholar
  69. Krause, G., & Welp, M. (2012). Systems thinking and social learning for sustainability. In M. Glaser, G. Krause, B. M. W. Ratter, & M. Welp (Eds.), Human-nature interactions in the Anthropocene: Potentials of social-ecological systems analysis (pp. 13–33). Abingdon, Oxon: Routledge.Google Scholar
  70. Lawrence, D. P. (2007). Impact significance determination—Pushing the boundaries. Environmental Impact Assessment Review, 27(8), 770–788. Scholar
  71. Letcher, R., Roberts, S., & Redgrove, Z. (2007). Mobilising individual behavioural change through community initiatives: Lessons for tackling climate change. The Energy Review Study [on-line]. Bristol.
  72. Lyons, M. (2005). Knowledge and the modelling of complex systems. Futures, 37, 711–719. Scholar
  73. Mascarenhas, A., Coelho, P., Subtil, E., & Ramos, T. B. (2010). The role of common local indicators in regional sustainability assessment. Ecological Indicators, 10(3), 646–656. Scholar
  74. McCormick, K., Anderberg, S., Coenen, L., & Neij, L. (2013). Advancing sustainable urban transformation. Journal of Cleaner Production, 50, 1–11. Scholar
  75. McDonald, H. (2016). Developmental evaluation: A tool to support innovation. Evaluation Matters—He Take Tō Te Aromatawai, 2, 79–97. Scholar
  76. Mears, C. A., & Wentz, F. J. (2016). Sensitivity of satellite-derived tropospheric temperature trends to the diurnal cycle adjustment. Journal of Climate. Scholar
  77. Middlemiss, L. (2011). The effects of community-based action for sustainability on participants’ lifestyles. Local Environment, 16(3), 265–280. Scholar
  78. Middlemiss, L., & Parrish, B. D. (2010). Building capacity for low-carbon communities: The role of grassroots initiatives. Energy Policy, 38(12), 7559–7566. Scholar
  79. Mitchell, M. (2009). Complexity: A guided tour. New York: Oxford University Press.Google Scholar
  80. Moore, D. T. (2011). Sensemaking: A structure for an intelligence revolution. Washington, DC: National Defense Intelligence College.Google Scholar
  81. Mori, K., & Christodoulou, A. (2012). Review of sustainability indices and indicators: Towards a new City Sustainability Index (CSI). Environmental Impact Assessment Review, 32(1), 94–106. Scholar
  82. Morton, T. (2010). The ecological thought. Cambridge, MA: Harvard University Press.Google Scholar
  83. O’Brien, W., & Sarkis, J. (2014). The potential of community-based sustainability projects for deep learning initiatives. Journal of Cleaner Production, 62, 48–61. Scholar
  84. Patton, M. Q. (1997). Utilization-focused evaluation (3rd ed.). Thousand Oaks, CA: Sage.Google Scholar
  85. Patton, M. Q. (2011). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York: The Guildford Press.Google Scholar
  86. Petersen, J. E., Shunturov, V., Janda, K., Platt, G., & Weinberger, K. (2007). Dormitory residents reduce electricity consumption when exposed to real-time visual feedback and incentives. International Journal of Sustainability in Higher Education, 8(1), 16–33. Scholar
  87. Plass, G. N. (1956). The carbon dioxide theory of climatic change. Tellus A, 8(1896), 140–154. Scholar
  88. Ramalingam, B. (2013). Aid on the edge of chaos: Rethinking international development in a complex world. Oxford: Oxford University Press.Google Scholar
  89. Ramírez, R., & Brodhead, D. (2013). Utilization focused evaluation: A primer for evaluators. Penang, Malaysia: Southbound Sdn. Bhd.Google Scholar
  90. Reed, M. S., Fraser, E. D. G., & Dougill, A. J. (2006). An adaptive learning process for developing and applying sustainability indicators with local communities. Ecological Economics, 59(4), 406–418. Scholar
  91. Rees, W. (2010). What’s blocking sustainability? Human nature, cognition, and denial. Sustainability: Science, Practice, & Policy, 6(2), 13–25.Google Scholar
  92. Revelle, R., & Suess, H. E. (1957). Carbon dioxide exchange between atmosphere and ocean and the question of an increase of atmospheric CO2 during the past decades. Tellus A, 9(1), 18–27. Scholar
  93. Richardson, K., & Cilliers, P. (2001). What is complexity science? A view from different directions. Emergence, 3(1), 5–23.CrossRefGoogle Scholar
  94. Rittel, H. W. J., & Weber, M. M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4(2), 155–169. Scholar
  95. Rogers, P. (2014). Theory of change, methodological briefs: Impact evaluation 2 (p. 16). Available at:;
  96. Ruiz, A. B. (1996). The contribution of Humberto Maturana to the sciences of complexity and psychology. Journal of Constructivist Psychology, 9, 283–302.CrossRefGoogle Scholar
  97. Seyfang, G. (2005). Shopping for sustainability: Can sustainable consumption promote ecological citizenship? Environmental Politics, 14(2), 290–306. Scholar
  98. Seyfang, G., Hielscher, S., Hargreaves, T., Martiskainen, M., & Smith, A. (2014). A grassroots sustainable energy niche? Reflections on community energy in the UK. Environmental Innovation and Societal Transitions. Scholar
  99. Shea, J., & Taylor, T. (2017). Using developmental evaluation as a system of organizational learning: An example from San Francisco. Evaluation and Program Planning, 65(July), 84–93. Scholar
  100. Singh, R. K., Murty, H. R., Gupta, S. K., & Dikshit, A. K. (2009). An overview of sustainability assessment methodologies. Ecological Indicators, 9(2), 189–212. Scholar
  101. Smith, A. (2007). Translating sustainabilities between green niches and socio-technical regimes. Technology Analysis & Strategic Management, 19(4), 427–450. Scholar
  102. Steg, L., & Vlek, C. (2009). Encouraging pro-environmental behaviour: An integrative review and research agenda. Journal of Environmental Psychology, 29(3), 309–317. Scholar
  103. Thomas, W. H. (2015). The basics of project evaluation and lessons learned (2nd ed.). Boca Raton, FL: CRC Press.Google Scholar
  104. Tognetti, S. S. (1999). Science in a double-bind: Gregory Bateson and the origins of post-normal science. Futures, 31(7), 689–703. Scholar
  105. Tovey, H. (2009). “Local Food” as a contested concept: Networks, knowledges and power in food-based strategies for rural development. International Journal of Sociology of Agriculture & Food, 16(2), 21–35.Google Scholar
  106. Turcu, C. (2013). Re-thinking sustainability indicators: Local perspectives of urban sustainability. Journal of Environmental Planning and Management, 56(5), 695–719. Scholar
  107. Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 211(4481), 453–458.CrossRefGoogle Scholar
  108. UK Parliament. (2008). Climate Change Act. Scholar
  109. Walton, M. (2014). Applying complexity theory: A review to inform evaluation design. Evaluation and Program Planning, 45, 119–126. Scholar
  110. Weiss, C. H. (1995). Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families. In J. Connell, A. Kubisch, L. Schorr, & C. Weiss (Eds.), New approaches to evaluating community initiatives (pp. 65–92). Washington, DC: Aspen Institute.Google Scholar
  111. Wiek, A., Xiong, A., Brundiers, K., & van der Leeuw, S. (2014). Integrating problem—And project-based learning into sustainability programs. International Journal of Sustainability in Higher Education, 15(4), 431–449. Scholar
  112. World Resources Institute (WRI). (2009). The National Adaptive Capacity Framework: Key institutional functions for a changing climate. Washington, DC. Available at:
  113. Xia, B., Chen, Q., Skitmore, M., Zuo, J., & Li, M. (2014). Comparison of sustainable community rating tools in Australia. Journal of Cleaner Production, 109, 84–91. Scholar

Copyright information

© The Author(s) 2019

Authors and Affiliations

  1. 1.ADAPT ManagementLeicesterUK

Personalised recommendations