Advertisement

Journal of the Operational Research Society

, Volume 58, Issue 10, pp 1306–1320 | Cite as

Systemic evaluation: a participative, multi-method approach

  • A Boyd
  • T Geerling
  • W J Gregory
  • C Kagan
  • G MidgleyEmail author
  • P Murray
  • M P Walsh
Case-Oriented Paper

Abstract

This paper presents some new ideas on systemic evaluation developed in the context of a project to support capacity building for the evaluation of community health services. Emphases are placed on the need for stakeholder participation; dialogue on the boundaries of evaluations; considering multiple values; and ensuring that marginalized people and issues are properly accounted for. Further developing the work of previous authors, three different approaches to evaluation are outlined, each of which can be applied participatively: goal-based (where goals are set and their achievement is measured); stakeholder (where there are no pre-set goals, and different people's experiences and stories are surveyed to reveal significant issues); and organizational (where organizational processes are compared with models of good practice). There is a logical relationship between these: a stakeholder evaluation can lead to the setting of community-sensitive goals, the achievement of which can be measured through goal-based evaluation, and pursuit of the goals can be enhanced by organizational evaluation. A participative, flexible and responsive evaluation practice will often need to draw upon aspects of all three approaches. Initial feedback from health practitioners suggests that this way of thinking about systemic evaluation may prove useful in a range of situations faced by people in the statutory, voluntary and community sectors.

Keywords

community health services community operational research critical systems thinking health participation systems methodology 

References

  1. Ackoff RL (1981). Creating the Corporate Future. Wiley: New York.Google Scholar
  2. Baker V, Foote J, Gregor J, Houston D and Midgley G (2004). Boundary critique and community involvement in small watershed management. In: Dew K and Fitzgerald R (Eds). Challenging Science: Science and Society Issues in New Zealand. Dunmore Press: Palmerston North.Google Scholar
  3. Beer S (1985). Diagnosing the System for Organizations. Wiley: Chichester.Google Scholar
  4. Boyd A (2002). Capacity-Building for Evaluation: Follow Up to the HAZE Project. Centre for Systems Studies, Business School, University of Hull: Hull.Google Scholar
  5. Boyd A, Brown M and Midgley G (1999). Home and Away: Developing Services with Young People Missing from Home or Care. Centre for Systems Studies: Hull.Google Scholar
  6. Boyd A, Brown M and Midgley G (2004). Systemic intervention for community OR: developing services with young people (under 16) living on the streets. In: Midgley G and Ochoa-Arias AE (Eds). Community Operational Research: OR and Systems Thinking for Community Development. Kluwer/Plenum: New York.Google Scholar
  7. Boyd A, Geerling T, Gregory W, Kagan C, Midgley G, Murray P and Walsh M (2001). Capacity-Building for Evaluation: A Report on the HAZE Project to the Manchester, Salford and Trafford Health Action Zone. Centre for Systems Studies, Business School, University of Hull: Hull.Google Scholar
  8. Brocklesby J and Cummings S (1996). Foucault plays Habermas: an alternative philosophical underpinning for critical systems thinking (CST). J Opl Res Soc 47: 741–754.CrossRefGoogle Scholar
  9. Brockwell PJ and Davis RA (2002). Introduction to Time Series and Forecasting, 2nd edn. Springer-Verlag: New York.CrossRefGoogle Scholar
  10. Brown M (1996). A framework for assessing participation. In: Flood RL and Romm NRA (Eds). Critical Systems Thinking: Current Research and Practice. Plenum: New York.Google Scholar
  11. Chambers R (1997). Whose Reality Counts? Putting the Last First. Intermediate Technology Publications: London.CrossRefGoogle Scholar
  12. Charnes A, Cooper W, Lewin AY and Seiford LM (eds) (1995). Data Envelopment Analysis: Theory, Methodology and Applications. Kluwer: London.Google Scholar
  13. Checkland P (1981). Systems Thinking, Systems Practice. Wiley: Chichester.Google Scholar
  14. Checkland P and Scholes J (1990). Soft Systems Methodology in Action. Wiley: Chichester.Google Scholar
  15. Churchman CW (1970). Operations research as a profession. Mngt Sci 17: B37–B53.CrossRefGoogle Scholar
  16. Cilliers P (1998). Complexity and Post-Modernism. Routledge: London.Google Scholar
  17. Clemson M and Jackson MC (1988). Evaluating organizations with multiple goals. OR Insight 1(2): 2–5.CrossRefGoogle Scholar
  18. Codling S (1995). Best Practice Benchmarking. Gower: Aldershot.Google Scholar
  19. Cohen C and Midgley G (1994). The North Humberside Diversion from Custody Project for Mentally Disordered Offenders: Research Report. Centre for Systems Studies: Hull.Google Scholar
  20. Cooper WW, Seiford LM and Zhu J (eds) (2004). Handbook on Data Envelopment Analysis.Kluwer:Boston.Google Scholar
  21. Drury C (1985). Management and Cost Accounting. Van Nostrand Reinhold (UK): Wokingham.Google Scholar
  22. Dutt PK (1994). Problem contexts—a consultant's perspective. Systems Pract 7: 539–550.CrossRefGoogle Scholar
  23. Eden C (1989). Using cognitive mapping for strategic options development and analysis. In: Rosenhead J. (Ed). Rational Analysis for a Problematic World: Problem Structuring Methods for Complexity, Uncertainty and Conflict. Wiley: Chichester.Google Scholar
  24. Fishman GS (2001). Discrete-Event Simulation: Modeling, Programming, and Analysis. Springer-Verlag: Berlin.CrossRefGoogle Scholar
  25. Flood RL (1995). Solving Problem Solving. Wiley: Chichester.Google Scholar
  26. Flood RL and Jackson MC (eds) (1991a). Critical Systems Thinking: Directed Readings. Wiley: Chichester.Google Scholar
  27. Flood RL and Jackson MC (1991b). Creative Problem Solving: Total Systems Intervention. Wiley: Chichester.Google Scholar
  28. Foucault M (1976). Two lectures. In: Gordon C. (Ed). Michel Foucault: Power/Knowledge. Harvester Wheatsheaf: London.Google Scholar
  29. Friend JK and Hickling A (1987). Planning under Pressure: The Strategic Choice Approach. Pergamon: Oxford.Google Scholar
  30. Gregory AJ (1995). Organizational evaluation: a complementarist approach. PhD thesis, University of Hull.Google Scholar
  31. Gregory AJ (1996a). The road to integration: reflections on the development of organizational evaluation theory and practice. Omega 24(3): 295–307.CrossRefGoogle Scholar
  32. Gregory AJ (1996b). Long-term evaluation systems. OR Insight 9(1): 29–32.CrossRefGoogle Scholar
  33. Gregory AJ (1997). Evaluation practice and the tricky issue of coercive contexts. Systems Pract 10: 589–609.CrossRefGoogle Scholar
  34. Gregory AJ and Jackson MC (1992a). Evaluation methodologies: a system for use. J Opl Res Soc 43: 19–28.CrossRefGoogle Scholar
  35. Gregory AJ and Jackson MC (1992b). Evaluating organizations: a systems and contingency approach. Systems Pract 5: 37–60.CrossRefGoogle Scholar
  36. Gregory WJ (1992). Critical systems thinking and pluralism: a new constellation. PhD thesis, City University, London.Google Scholar
  37. Gregory WJ (1996). Discordant pluralism: a new strategy for critical systems thinking. Systems Pract 9: 605–625.CrossRefGoogle Scholar
  38. Guba E and Lincoln Y (1989). Fourth Generation Evaluation. Sage: London.Google Scholar
  39. Hearn J and Parkin W (1993). Organizations, multiple oppressions and postmodernism. In: Hassard J and Parker M (Eds). Postmodernism and Organizations. Sage: London.Google Scholar
  40. Hughes A and Halsall DN (2002). Comparison of the 14 deadly diseases and the business excellence model. Total Qual Mngt 13(2): 255–263.CrossRefGoogle Scholar
  41. Jackson MC (1991). The origins and nature of critical systems thinking. Systems Pract 4: 131–149.CrossRefGoogle Scholar
  42. Jackson MC (2000). Systems Approaches to Management. Kluwer/Plenum: New York.Google Scholar
  43. Jackson MC and Keys P (1984). Towards a system of systems methodologies. J Opl Res Soc 35: 473–486.CrossRefGoogle Scholar
  44. Jackson MC and Medjedoub S (1988). Designing evaluation systems: theoretical groundings and a practical intervention. In: Trappl R. (Ed). Cybernetics and Systems. Reidel: Dordrecht.Google Scholar
  45. Jackson N and Carter P (1991). In defence of paradigm incommensurability. Organ Stud 12: 109–127.CrossRefGoogle Scholar
  46. Mason RO and Mitroff II (1981). Challenging Strategic Planning Assumptions. Wiley: New York.Google Scholar
  47. Maturana HR and Varela FJ (1980). Autopoiesis and Cognition: The Realization of the Living. Reidel: Dordrecht.CrossRefGoogle Scholar
  48. Midgley G (1988). A systems analysis and evaluation of microjob: a vocational rehabilitation and information technology training centre for people with disabilities. MPhil thesis, City University, London.Google Scholar
  49. Midgley G (1990). Creative methodology design. Systemist 12: 108–113.Google Scholar
  50. Midgley G (1992). The sacred and profane in critical systems thinking. Systems Pract 5: 5–16.CrossRefGoogle Scholar
  51. Midgley G (1994). Ecology and the poverty of humanism: a critical systems perspective. Systems Res 11: 67–76.CrossRefGoogle Scholar
  52. Midgley G (1996). What is this thing called CST? In: Flood RL and Romm NRA (Eds). Critical Systems Thinking: Current Research and Practice. Plenum: New York.Google Scholar
  53. Midgley G (1997a). Dealing with coercion: critical systems heuristics and beyond. Systems Pract 10: 37–57.CrossRefGoogle Scholar
  54. Midgley G (1997b). Developing the methodology of TSI: from the oblique use of methods to creative design. Systems Pract 10: 305–319.CrossRefGoogle Scholar
  55. Midgley G (1999). Ethical Dilemmas: a reply to Richard Ormerod. J Opl Res Soc 50: 549–553.CrossRefGoogle Scholar
  56. Midgley G (2000). Systemic Intervention: Philosophy, Methodology, and Practice. Kluwer/Plenum: New York.CrossRefGoogle Scholar
  57. Midgley G and Floyd M (1988). Microjob: A Computer Training Service for People with Disabilities. Rehabilitation Resource Centre: London.Google Scholar
  58. Midgley G and Floyd M (1990). Vocational training in the use of new technologies for people with disabilities. Behav Inform Technol 9: 409–424.CrossRefGoogle Scholar
  59. Midgley G, Munlo I and Brown M (1998). The theory and practice of boundary critique: developing housing services for older people. J Opl Res Soc 49: 467–478.CrossRefGoogle Scholar
  60. Midgley G and Ochoa-Arias AE (2004). An introduction to community operational research. In: Midgley G and Ochoa-Arias AE (Eds). Community Operational Research: OR and Systems Thinking for Community Development. Kluwer/Plenum: New York.CrossRefGoogle Scholar
  61. Mingers JC (1992). What are real friends for? A reply to Mike Jackson. J Opl Res Soc 43: 732–735.CrossRefGoogle Scholar
  62. Mingers JC (1997). Towards critical pluralism. In: Mingers J and Gill A (Eds). Multimethodology: The Theory and Practice of Combining Management Science Methodologies. Wiley: Chichester.Google Scholar
  63. Ochoa-Arias AE (2004). An interpretive systemic exploration of community action in Venezuela. In: Midgley G and Ochoa-Arias AE (Eds). Community Operational Research: OR and Systems Thinking for Community Development. Kluwer/Plenum: New York.Google Scholar
  64. Ormerod RJ (1999). Viewpoint—ethical dilemmas. J Opl Res Soc 50: 546–548.CrossRefGoogle Scholar
  65. Owen H (1997). Open Space Technology: A User's Guide, 2nd edn. Berrett-Koehler: San Francisco.Google Scholar
  66. Pratt J, Gordon P and Plamping D (1999). Working Whole Systems. King's Fund: London.Google Scholar
  67. Prigogine I and Stengers I (1984). Order out of Chaos: Man's New Dialogue with Nature. Fontana: London.Google Scholar
  68. Probyn E (1990). Travels in the postmodern: making sense of the local. In: Nicholson L.J. (Ed). Feminism/Postmodernism. Routledge: London.Google Scholar
  69. Puchta C and Potter J (2004). Focus Group Practice. Cassell: London.CrossRefGoogle Scholar
  70. Ritchie C, Taket A and Bryant J (eds) (1994). Community Works: 26 Case Studies showing Community Operational Research in Action. Pavic Press: Sheffield.Google Scholar
  71. Rosenau PM (1992). Post-Modernism and the Social Sciences: Insights, Inroads, and Intrusions. Princeton University Press: Princeton, NJ.Google Scholar
  72. Rosenhead J, (ed) (1989). Rational Analysis for a Problematic World: Problem Structuring Methods for Complexity, Uncertainty and Conflict. Wiley: Chichester.Google Scholar
  73. Rosenhead J and Mingers J (eds) (2001). Rational Analysis for a Problematic World Revisited: Problem Structuring Methods for Complexity, Uncertainty and Conflict. Wiley: Chichester.Google Scholar
  74. Schecter D (1991). Critical systems thinking in the 1980s: a connective summary. In: Flood RL and Jackson MC (Eds). Critical Systems Thinking: Directed Readings. Wiley: Chichester.Google Scholar
  75. Shaddish WR, Cook TD and Leviton LC (1991). Foundations of Program Evaluation: Theories of Practice. Sage: London.Google Scholar
  76. Spaul MWJ (1997). Multimethodology and critical theory: an intersection of interests? In: Mingers J and Gill A (Eds). Multimethodology: The Theory and Practice of Combining Management Science Methodologies. John Wiley: Chichester.Google Scholar
  77. Stacey RD, Griffin D and Shaw P (2000). Complexity and Management: Fad or Radical Challenge to Systems Thinking? Routledge: London.Google Scholar
  78. Stake RE (1980). Program evaluation, particularly responsive evaluation. In: Dockrell WB and Hamilton D (Eds). Rethinking Educational Research. Hodder and Stoughton: London 72–87.Google Scholar
  79. Stewart DW and Shamdasani PN (1990). Focus Groups: Theory and Practice. Sage: Thousand Oaks, CA.Google Scholar
  80. Sudman S and Bradburn NM (1982). Asking Questions: A Practical Guide to Questionnaire Design. Jossey Bass: San Francisco.Google Scholar
  81. Taket A and White L (1993). After OR: an agenda for postmodernism and poststructuralism in OR. J Opl Res Soc 44: 867–881.CrossRefGoogle Scholar
  82. Taket A and White L (1994). The death of the expert. J Opl Res Soc 45: 733–748.CrossRefGoogle Scholar
  83. Taket A and White L (1996). Pragmatic pluralism—an explication. Systems Pract 9(6): 571–586.CrossRefGoogle Scholar
  84. Taket A and White L (1997). Working with heterogeneity: a pluralist strategy for evaluation. Systems Res Behav Sci 14(2): 101–111.CrossRefGoogle Scholar
  85. Taket A and White L (2000). Partnership and Participation: Decision-Making in the Multiagency Setting. Wiley: Chichester.Google Scholar
  86. Taket A and White L (2004). Playing with PANDA: The CybOrg and the rhizome. In: Midgley G and Ochoa-Arias AE (Eds). Community Operational Research: OR and Systems Thinking for Community Development. Kluwer/Plenum: New York.Google Scholar
  87. Ulrich W (1983). Critical Heuristics of Social Planning: A New Approach to Practical Philosophy. Haupt: Bern.Google Scholar
  88. Ulrich W (1993). Some difficulties of ecological thinking considered from a critical systems perspective: a plea for critical holism. Systems Pract 6: 583–611.CrossRefGoogle Scholar
  89. Weil S (1998). Rhetorics and realities in public service organizations: systemic practice and organizational learning as critically reflexive action research (CRAR). Systemic Pract Action Res 11: 37–62.CrossRefGoogle Scholar
  90. Whittaker A, Gardner S and Kershaw J (1991). Service Evaluation by People with Learning Difficulties. Kings Fund: London.Google Scholar
  91. Wolstenholme EF (1990). System Enquiry: A System Dynamics Approach. Wiley: Chichester.Google Scholar
  92. Wright DS, Taylor A, Davies DR, Sluckin W, Lee SGM and Reason JT (1970). Introducing Psychology: An Experimental Approach. Penguin: Harmondsworth.Google Scholar
  93. Yuchtman E and Seashore SE (1967). A system resource approach to organizational effectiveness. Am Soc Rev 32: 891–903.CrossRefGoogle Scholar
  94. Zhu J (2002). Quantitative Models for Performance Evaluation and Benchmarking: Data Envelopment Analysis with Spreadsheets and DEA Excel Solver. Kluwer: Boston.Google Scholar

Copyright information

© Palgrave Macmillan Ltd 2006

Authors and Affiliations

  • A Boyd
    • 1
  • T Geerling
    • 2
  • W J Gregory
    • 3
  • C Kagan
    • 4
  • G Midgley
    • 3
    • 5
    Email author
  • P Murray
    • 2
  • M P Walsh
    • 6
  1. 1.University of ManchesterManchesterUK
  2. 2.University of HullHullUK
  3. 3.Institute of Environmental Science and Research (ESR) Ltd.ChristchurchNew Zealand
  4. 4.Manchester Metropolitan UniversityManchesterUK
  5. 5.Victoria UniversityWellingtonNew Zealand
  6. 6.North Star (UK) Ltd.LeconfieldUK

Personalised recommendations