Skip to main content
SpringerLink
Log in

Systemic evaluation: a participative, multi-method approach

  • Case-Oriented Paper
  • Published:
Journal of the Operational Research Society

Abstract

This paper presents some new ideas on systemic evaluation developed in the context of a project to support capacity building for the evaluation of community health services. Emphases are placed on the need for stakeholder participation; dialogue on the boundaries of evaluations; considering multiple values; and ensuring that marginalized people and issues are properly accounted for. Further developing the work of previous authors, three different approaches to evaluation are outlined, each of which can be applied participatively: goal-based (where goals are set and their achievement is measured); stakeholder (where there are no pre-set goals, and different people's experiences and stories are surveyed to reveal significant issues); and organizational (where organizational processes are compared with models of good practice). There is a logical relationship between these: a stakeholder evaluation can lead to the setting of community-sensitive goals, the achievement of which can be measured through goal-based evaluation, and pursuit of the goals can be enhanced by organizational evaluation. A participative, flexible and responsive evaluation practice will often need to draw upon aspects of all three approaches. Initial feedback from health practitioners suggests that this way of thinking about systemic evaluation may prove useful in a range of situations faced by people in the statutory, voluntary and community sectors.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5

Similar content being viewed by others

References

  • Ackoff RL (1981). Creating the Corporate Future. Wiley: New York.

    Google Scholar 

  • Baker V, Foote J, Gregor J, Houston D and Midgley G (2004). Boundary critique and community involvement in small watershed management. In: Dew K and Fitzgerald R (Eds). Challenging Science: Science and Society Issues in New Zealand. Dunmore Press: Palmerston North.

    Google Scholar 

  • Beer S (1985). Diagnosing the System for Organizations. Wiley: Chichester.

    Google Scholar 

  • Boyd A (2002). Capacity-Building for Evaluation: Follow Up to the HAZE Project. Centre for Systems Studies, Business School, University of Hull: Hull.

    Google Scholar 

  • Boyd A, Brown M and Midgley G (1999). Home and Away: Developing Services with Young People Missing from Home or Care. Centre for Systems Studies: Hull.

    Google Scholar 

  • Boyd A, Brown M and Midgley G (2004). Systemic intervention for community OR: developing services with young people (under 16) living on the streets. In: Midgley G and Ochoa-Arias AE (Eds). Community Operational Research: OR and Systems Thinking for Community Development. Kluwer/Plenum: New York.

    Google Scholar 

  • Boyd A, Geerling T, Gregory W, Kagan C, Midgley G, Murray P and Walsh M (2001). Capacity-Building for Evaluation: A Report on the HAZE Project to the Manchester, Salford and Trafford Health Action Zone. Centre for Systems Studies, Business School, University of Hull: Hull.

    Google Scholar 

  • Brocklesby J and Cummings S (1996). Foucault plays Habermas: an alternative philosophical underpinning for critical systems thinking (CST). J Opl Res Soc 47: 741–754.

    Article  Google Scholar 

  • Brockwell PJ and Davis RA (2002). Introduction to Time Series and Forecasting, 2nd edn. Springer-Verlag: New York.

    Book  Google Scholar 

  • Brown M (1996). A framework for assessing participation. In: Flood RL and Romm NRA (Eds). Critical Systems Thinking: Current Research and Practice. Plenum: New York.

    Google Scholar 

  • Chambers R (1997). Whose Reality Counts? Putting the Last First. Intermediate Technology Publications: London.

    Book  Google Scholar 

  • Charnes A, Cooper W, Lewin AY and Seiford LM (eds) (1995). Data Envelopment Analysis: Theory, Methodology and Applications. Kluwer: London.

  • Checkland P (1981). Systems Thinking, Systems Practice. Wiley: Chichester.

    Google Scholar 

  • Checkland P and Scholes J (1990). Soft Systems Methodology in Action. Wiley: Chichester.

    Google Scholar 

  • Churchman CW (1970). Operations research as a profession. Mngt Sci 17: B37–B53.

    Article  Google Scholar 

  • Cilliers P (1998). Complexity and Post-Modernism. Routledge: London.

    Google Scholar 

  • Clemson M and Jackson MC (1988). Evaluating organizations with multiple goals. OR Insight 1(2): 2–5.

    Article  Google Scholar 

  • Codling S (1995). Best Practice Benchmarking. Gower: Aldershot.

    Google Scholar 

  • Cohen C and Midgley G (1994). The North Humberside Diversion from Custody Project for Mentally Disordered Offenders: Research Report. Centre for Systems Studies: Hull.

    Google Scholar 

  • Cooper WW, Seiford LM and Zhu J (eds) (2004). Handbook on Data Envelopment Analysis.Kluwer:Boston.

  • Drury C (1985). Management and Cost Accounting. Van Nostrand Reinhold (UK): Wokingham.

    Google Scholar 

  • Dutt PK (1994). Problem contexts—a consultant's perspective. Systems Pract 7: 539–550.

    Article  Google Scholar 

  • Eden C (1989). Using cognitive mapping for strategic options development and analysis. In: Rosenhead J. (Ed). Rational Analysis for a Problematic World: Problem Structuring Methods for Complexity, Uncertainty and Conflict. Wiley: Chichester.

    Google Scholar 

  • Fishman GS (2001). Discrete-Event Simulation: Modeling, Programming, and Analysis. Springer-Verlag: Berlin.

    Book  Google Scholar 

  • Flood RL (1995). Solving Problem Solving. Wiley: Chichester.

    Google Scholar 

  • Flood RL and Jackson MC (eds) (1991a). Critical Systems Thinking: Directed Readings. Wiley: Chichester.

  • Flood RL and Jackson MC (1991b). Creative Problem Solving: Total Systems Intervention. Wiley: Chichester.

    Google Scholar 

  • Foucault M (1976). Two lectures. In: Gordon C. (Ed). Michel Foucault: Power/Knowledge. Harvester Wheatsheaf: London.

    Google Scholar 

  • Friend JK and Hickling A (1987). Planning under Pressure: The Strategic Choice Approach. Pergamon: Oxford.

    Google Scholar 

  • Gregory AJ (1995). Organizational evaluation: a complementarist approach. PhD thesis, University of Hull.

  • Gregory AJ (1996a). The road to integration: reflections on the development of organizational evaluation theory and practice. Omega 24(3): 295–307.

    Article  Google Scholar 

  • Gregory AJ (1996b). Long-term evaluation systems. OR Insight 9(1): 29–32.

    Article  Google Scholar 

  • Gregory AJ (1997). Evaluation practice and the tricky issue of coercive contexts. Systems Pract 10: 589–609.

    Article  Google Scholar 

  • Gregory AJ and Jackson MC (1992a). Evaluation methodologies: a system for use. J Opl Res Soc 43: 19–28.

    Article  Google Scholar 

  • Gregory AJ and Jackson MC (1992b). Evaluating organizations: a systems and contingency approach. Systems Pract 5: 37–60.

    Article  Google Scholar 

  • Gregory WJ (1992). Critical systems thinking and pluralism: a new constellation. PhD thesis, City University, London.

  • Gregory WJ (1996). Discordant pluralism: a new strategy for critical systems thinking. Systems Pract 9: 605–625.

    Article  Google Scholar 

  • Guba E and Lincoln Y (1989). Fourth Generation Evaluation. Sage: London.

    Google Scholar 

  • Hearn J and Parkin W (1993). Organizations, multiple oppressions and postmodernism. In: Hassard J and Parker M (Eds). Postmodernism and Organizations. Sage: London.

    Google Scholar 

  • Hughes A and Halsall DN (2002). Comparison of the 14 deadly diseases and the business excellence model. Total Qual Mngt 13(2): 255–263.

    Article  Google Scholar 

  • Jackson MC (1991). The origins and nature of critical systems thinking. Systems Pract 4: 131–149.

    Article  Google Scholar 

  • Jackson MC (2000). Systems Approaches to Management. Kluwer/Plenum: New York.

    Google Scholar 

  • Jackson MC and Keys P (1984). Towards a system of systems methodologies. J Opl Res Soc 35: 473–486.

    Article  Google Scholar 

  • Jackson MC and Medjedoub S (1988). Designing evaluation systems: theoretical groundings and a practical intervention. In: Trappl R. (Ed). Cybernetics and Systems. Reidel: Dordrecht.

    Google Scholar 

  • Jackson N and Carter P (1991). In defence of paradigm incommensurability. Organ Stud 12: 109–127.

    Article  Google Scholar 

  • Mason RO and Mitroff II (1981). Challenging Strategic Planning Assumptions. Wiley: New York.

    Google Scholar 

  • Maturana HR and Varela FJ (1980). Autopoiesis and Cognition: The Realization of the Living. Reidel: Dordrecht.

    Book  Google Scholar 

  • Midgley G (1988). A systems analysis and evaluation of microjob: a vocational rehabilitation and information technology training centre for people with disabilities. MPhil thesis, City University, London.

  • Midgley G (1990). Creative methodology design. Systemist 12: 108–113.

    Google Scholar 

  • Midgley G (1992). The sacred and profane in critical systems thinking. Systems Pract 5: 5–16.

    Article  Google Scholar 

  • Midgley G (1994). Ecology and the poverty of humanism: a critical systems perspective. Systems Res 11: 67–76.

    Article  Google Scholar 

  • Midgley G (1996). What is this thing called CST? In: Flood RL and Romm NRA (Eds). Critical Systems Thinking: Current Research and Practice. Plenum: New York.

    Google Scholar 

  • Midgley G (1997a). Dealing with coercion: critical systems heuristics and beyond. Systems Pract 10: 37–57.

    Article  Google Scholar 

  • Midgley G (1997b). Developing the methodology of TSI: from the oblique use of methods to creative design. Systems Pract 10: 305–319.

    Article  Google Scholar 

  • Midgley G (1999). Ethical Dilemmas: a reply to Richard Ormerod. J Opl Res Soc 50: 549–553.

    Article  Google Scholar 

  • Midgley G (2000). Systemic Intervention: Philosophy, Methodology, and Practice. Kluwer/Plenum: New York.

    Book  Google Scholar 

  • Midgley G and Floyd M (1988). Microjob: A Computer Training Service for People with Disabilities. Rehabilitation Resource Centre: London.

    Google Scholar 

  • Midgley G and Floyd M (1990). Vocational training in the use of new technologies for people with disabilities. Behav Inform Technol 9: 409–424.

    Article  Google Scholar 

  • Midgley G, Munlo I and Brown M (1998). The theory and practice of boundary critique: developing housing services for older people. J Opl Res Soc 49: 467–478.

    Article  Google Scholar 

  • Midgley G and Ochoa-Arias AE (2004). An introduction to community operational research. In: Midgley G and Ochoa-Arias AE (Eds). Community Operational Research: OR and Systems Thinking for Community Development. Kluwer/Plenum: New York.

    Chapter  Google Scholar 

  • Mingers JC (1992). What are real friends for? A reply to Mike Jackson. J Opl Res Soc 43: 732–735.

    Article  Google Scholar 

  • Mingers JC (1997). Towards critical pluralism. In: Mingers J and Gill A (Eds). Multimethodology: The Theory and Practice of Combining Management Science Methodologies. Wiley: Chichester.

    Google Scholar 

  • Ochoa-Arias AE (2004). An interpretive systemic exploration of community action in Venezuela. In: Midgley G and Ochoa-Arias AE (Eds). Community Operational Research: OR and Systems Thinking for Community Development. Kluwer/Plenum: New York.

    Google Scholar 

  • Ormerod RJ (1999). Viewpoint—ethical dilemmas. J Opl Res Soc 50: 546–548.

    Article  Google Scholar 

  • Owen H (1997). Open Space Technology: A User's Guide, 2nd edn. Berrett-Koehler: San Francisco.

    Google Scholar 

  • Pratt J, Gordon P and Plamping D (1999). Working Whole Systems. King's Fund: London.

    Google Scholar 

  • Prigogine I and Stengers I (1984). Order out of Chaos: Man's New Dialogue with Nature. Fontana: London.

    Google Scholar 

  • Probyn E (1990). Travels in the postmodern: making sense of the local. In: Nicholson L.J. (Ed). Feminism/Postmodernism. Routledge: London.

    Google Scholar 

  • Puchta C and Potter J (2004). Focus Group Practice. Cassell: London.

    Book  Google Scholar 

  • Ritchie C, Taket A and Bryant J (eds) (1994). Community Works: 26 Case Studies showing Community Operational Research in Action. Pavic Press: Sheffield.

  • Rosenau PM (1992). Post-Modernism and the Social Sciences: Insights, Inroads, and Intrusions. Princeton University Press: Princeton, NJ.

    Google Scholar 

  • Rosenhead J, (ed) (1989). Rational Analysis for a Problematic World: Problem Structuring Methods for Complexity, Uncertainty and Conflict. Wiley: Chichester.

  • Rosenhead J and Mingers J (eds) (2001). Rational Analysis for a Problematic World Revisited: Problem Structuring Methods for Complexity, Uncertainty and Conflict. Wiley: Chichester.

  • Schecter D (1991). Critical systems thinking in the 1980s: a connective summary. In: Flood RL and Jackson MC (Eds). Critical Systems Thinking: Directed Readings. Wiley: Chichester.

    Google Scholar 

  • Shaddish WR, Cook TD and Leviton LC (1991). Foundations of Program Evaluation: Theories of Practice. Sage: London.

    Google Scholar 

  • Spaul MWJ (1997). Multimethodology and critical theory: an intersection of interests? In: Mingers J and Gill A (Eds). Multimethodology: The Theory and Practice of Combining Management Science Methodologies. John Wiley: Chichester.

    Google Scholar 

  • Stacey RD, Griffin D and Shaw P (2000). Complexity and Management: Fad or Radical Challenge to Systems Thinking? Routledge: London.

    Google Scholar 

  • Stake RE (1980). Program evaluation, particularly responsive evaluation. In: Dockrell WB and Hamilton D (Eds). Rethinking Educational Research. Hodder and Stoughton: London 72–87.

    Google Scholar 

  • Stewart DW and Shamdasani PN (1990). Focus Groups: Theory and Practice. Sage: Thousand Oaks, CA.

    Google Scholar 

  • Sudman S and Bradburn NM (1982). Asking Questions: A Practical Guide to Questionnaire Design. Jossey Bass: San Francisco.

    Google Scholar 

  • Taket A and White L (1993). After OR: an agenda for postmodernism and poststructuralism in OR. J Opl Res Soc 44: 867–881.

    Article  Google Scholar 

  • Taket A and White L (1994). The death of the expert. J Opl Res Soc 45: 733–748.

    Article  Google Scholar 

  • Taket A and White L (1996). Pragmatic pluralism—an explication. Systems Pract 9(6): 571–586.

    Article  Google Scholar 

  • Taket A and White L (1997). Working with heterogeneity: a pluralist strategy for evaluation. Systems Res Behav Sci 14(2): 101–111.

    Article  Google Scholar 

  • Taket A and White L (2000). Partnership and Participation: Decision-Making in the Multiagency Setting. Wiley: Chichester.

    Google Scholar 

  • Taket A and White L (2004). Playing with PANDA: The CybOrg and the rhizome. In: Midgley G and Ochoa-Arias AE (Eds). Community Operational Research: OR and Systems Thinking for Community Development. Kluwer/Plenum: New York.

    Google Scholar 

  • Ulrich W (1983). Critical Heuristics of Social Planning: A New Approach to Practical Philosophy. Haupt: Bern.

    Google Scholar 

  • Ulrich W (1993). Some difficulties of ecological thinking considered from a critical systems perspective: a plea for critical holism. Systems Pract 6: 583–611.

    Article  Google Scholar 

  • Weil S (1998). Rhetorics and realities in public service organizations: systemic practice and organizational learning as critically reflexive action research (CRAR). Systemic Pract Action Res 11: 37–62.

    Article  Google Scholar 

  • Whittaker A, Gardner S and Kershaw J (1991). Service Evaluation by People with Learning Difficulties. Kings Fund: London.

    Google Scholar 

  • Wolstenholme EF (1990). System Enquiry: A System Dynamics Approach. Wiley: Chichester.

    Google Scholar 

  • Wright DS, Taylor A, Davies DR, Sluckin W, Lee SGM and Reason JT (1970). Introducing Psychology: An Experimental Approach. Penguin: Harmondsworth.

    Google Scholar 

  • Yuchtman E and Seashore SE (1967). A system resource approach to organizational effectiveness. Am Soc Rev 32: 891–903.

    Article  Google Scholar 

  • Zhu J (2002). Quantitative Models for Performance Evaluation and Benchmarking: Data Envelopment Analysis with Spreadsheets and DEA Excel Solver. Kluwer: Boston.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Manchester, Manchester, UK

    A Boyd

  2. University of Hull, Hull, UK

    T Geerling & P Murray

  3. Institute of Environmental Science and Research (ESR) Ltd., Christchurch, New Zealand

    W J Gregory & G Midgley

  4. Manchester Metropolitan University, Manchester, UK

    C Kagan

  5. Victoria University, Wellington, New Zealand

    G Midgley

  6. North Star (UK) Ltd., Leconfield, UK

    M P Walsh

Authors
  1. A Boyd
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. T Geerling
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. W J Gregory
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. C Kagan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. G Midgley
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. P Murray
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. M P Walsh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to G Midgley.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Boyd, A., Geerling, T., Gregory, W. et al. Systemic evaluation: a participative, multi-method approach. J Oper Res Soc 58, 1306–1320 (2007). https://doi.org/10.1057/palgrave.jors.2602281

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1057/palgrave.jors.2602281

Keywords

Search

Navigation