Evaluating Effectiveness of Complex System Interventions

  • Jack Chen


Following the landmark reports from the Institute of Medicine (IOM)1, 2 and other studies, numerous intervention programs have been introduced to improve patients’ safety.2 6 The evaluation of the impact of such complex system interventions continues to be a challenge. There is little robust evidence about the effectiveness of intervention programs aimed at system improvement.7 Rigorous evaluation of the systems we use for delivering healthcare presents a different challenge from the ones used for evaluating simple interventions such as a new drug or new medical technology.


Evaluating Complex System Interventions 


  1. 1.
    Donaldson MS, Kohn LT, Corrigan J. To Err is Human. Building a Safer Health System. Washington, DC: National Academies Press; 2000.Google Scholar
  2. 2.
    Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academies Press; 2001.Google Scholar
  3. 3.
    Aspden P, Institute of Medicine (Committee on Data Standards for Patient Safety). Patient Safety: Achieving a New Standard for Care. Washington, DC: National Academies Press; 2004.Google Scholar
  4. 4.
    Byers JF, White SV. Patient Safety: Principles and Practice. New York: Springer; 2004.Google Scholar
  5. 5.
    Child AP, Institute of Medicine (Committee on the Work Environment for Nurses and Patient Safety). Keeping Patients Safe: Transforming the Work Environment of Nurses. Washington, DC: National Academies Press; 2004.Google Scholar
  6. 6.
    United States, Congress, House, Committee on Energy and Commerce. Patient Safety and Quality Improvement Act Report (To Accompany H.R. 663) (Including Cost Estimate of the Congressional Budget Office). Washington, DC: U.S. G.P.O; 2003.Google Scholar
  7. 7.
    John O. Safety Deficiencies in Healthcare – A Review of Research. Stockholm: Karolinska Institute Medical Management Centre; 2004.Google Scholar
  8. 8.
    Grimshaw J, Eccles M, Thomas R, et al. Toward evidence-based quality improvement. Evidence (and its limitations) of the effectiveness of guideline dissemination and implementation strategies 1966–1998. J Gen Intern Med. 2006;21(suppl 2):S14-S20.PubMedGoogle Scholar
  9. 9.
    Grimshaw JM, Thomas RE, MacLennan G, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. 2004;8:iii-iv. 1–72.PubMedGoogle Scholar
  10. 10.
    Doig GS, Simpson F, Finfer S, et al. Effect of evidence-based feeding guidelines on mortality of critically ill adults: a cluster randomised controlled trial. J Am Med Assoc. 2008;300:2731–2741.CrossRefGoogle Scholar
  11. 11.
    Thomas RE, Croal BL, Ramsay C, Eccles M, Grimshaw J. Effect of enhanced feedback and brief educational reminder messages on laboratory test requesting in primary care: A cluster randomised trial. Lancet. 2006;367:1990–1996.CrossRefPubMedGoogle Scholar
  12. 12.
    Doumit G, Gattellari M, Grimshaw J, O’Brien MA. Local opinion leaders: Effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2007;CD000125.Google Scholar
  13. 13.
    Finfer S, Bellomo R, Boyce N, French J, Myburgh J, Norton R. A comparison of albumin and saline for fluid resuscitation in the intensive care unit. N Engl J Med. 2004;350:2247–2256.CrossRefPubMedGoogle Scholar
  14. 14.
    Institute of Medicine National Academy of Sciences. Health Services Research: Workforce and Educational Issues. Washington, DC: National Academies Press; 1995.Google Scholar
  15. 15.
    Lomas J. Health services research: A domain where disciplines and decision makers meet. In: Sibbald WJ, Bion JF, eds. Evaluating Critical Care. Using Health Services Research to Improve Quality. Berlin: Springer; 2000:6–19.Google Scholar
  16. 16.
    Green LW, Kreuter MW. Health Program Planning: An Educational and Ecological Approach. 4th ed. New York: McGraw-Hill; 2005.Google Scholar
  17. 17.
    Windsor RA. Evaluation of Health Promotion, Health Education, and Disease Prevention Programs. 3rd ed. Boston: McGraw-Hill; 2004.Google Scholar
  18. 18.
    Hillman K, Chen J, Brown D. A clinical model for health services research – the medical emergency team. J Crit Care. 2003;18:195-199.CrossRefPubMedGoogle Scholar
  19. 19.
    Hillman K, Chen J, Cretikos M, et al. Introduction of medical emergency team (MET) system – a cluster-randomised controlled trial. Lancet. 2005;365:2091–2097.CrossRefPubMedGoogle Scholar
  20. 20.
    Basanta WE. Changing the culture of patient safety and medical errors: A symposium introduction and overview. J Leg Med. 2003;24:1–6.CrossRefPubMedGoogle Scholar
  21. 21.
    Cohen MM, Eustis MA, Gribbins RE. Changing the culture of patient safety: Leadership’s role in health care quality improvement. Jt Comm J Qual Saf. 2003;29:329–335.PubMedGoogle Scholar
  22. 22.
    Argyris C. On Organizational Learning. 2nd ed. Malden, MA: Blackwell; 1999.Google Scholar
  23. 23.
    Howard JK, Eckhardt SA. Action Research: A Guide for Library Media Specialists. Worthington, OH: Linworth; 2005.Google Scholar
  24. 24.
    Johnson AP. A Short Guide to Action Research. Boston: Pearson/Allyn and Bacon; 2005.Google Scholar
  25. 25.
    Haslett T. Implications of Systems Thinking for Research and Practice in Management. Caulfield East, VIC: Faculty of Business and Economics, Monash University; 1998.Google Scholar
  26. 26.
    Jackson MC. Systems Thinking: Creative Holism for Managers. Chichester: Wiley; 2002.Google Scholar
  27. 27.
    Donner A, Klar N. Design and Analysis of Cluster Randomization Trials in Health Research. London: Arnold; 2000.Google Scholar
  28. 28.
    Campbell MK, Elbourne DR, Altman DG, Group C. CONSORT statement: Extension to cluster randomised trials. BMJ. 2004;328:702–708.CrossRefPubMedGoogle Scholar
  29. 29.
    Kerry M, Bland JM. Statistical notes: sample size in cluster randomization. BMJ. 1998;316:549.PubMedGoogle Scholar
  30. 30.
    Kerry M, Bland JM. Analysis of a trial randomized in clusters. Br Med J. 1998;316:54.Google Scholar
  31. 31.
    Kish L, Frankel M. Inference from complex samples. J R Stat Soc. 1974;36:1–37.Google Scholar
  32. 32.
    Simpson JM, Klar N, Donner A. Accounting for cluster randomization: A review of primary prevention trials, 1990 through 1993. Am J Public Health. 1995;85:1378–1383.CrossRefPubMedGoogle Scholar
  33. 33.
    Campbell MK, Fayers PM, Grimshaw JM. Determinants of the intracluster correlation coefficient in cluster randomized trials: the case of implementation research. Clin Trials. 2005;2:99–107.CrossRefPubMedGoogle Scholar
  34. 34.
    Nathens AB, Jurkovich GJ, Cummings P, Rivara FP, Maier RV. The effect of organized systems of trauma care on motor vehicle crash mortality. JAMA. 2000;283:1990–1994.CrossRefPubMedGoogle Scholar
  35. 35.
    Lecky F, Woodford M, Yates DW. Trends in trauma care in England and Wales 1989–97. UK Trauma Audit and Research Network. Lancet. 2000;355:1771–1775.CrossRefPubMedGoogle Scholar
  36. 36.
    Young L, Donald M, Parr M, Hillman K. The medical emergency team system: a two-hospital comparison. Resuscitation. 2008;77:180–188.CrossRefPubMedGoogle Scholar
  37. 37.
    Parr MJA, Hadfield JH, Flabouris A, Bishop G, Hillman K. The medical emergency team: 12-Month analysis of reasons for activation, immediate outcome and not-for-resuscitation orders. Resuscitation. 2001;50:39–44.CrossRefPubMedGoogle Scholar
  38. 38.
    Kenward G, Castle N, Hodgetts T, Shaikh L. Evaluation of a medical emergency team one year after implementation. Resuscitation. 2004;61:257–263.CrossRefPubMedGoogle Scholar
  39. 39.
    Jones DA, McIntyre T, Baldwin I, Mercer I, Kattula A, Bellomo R. The medical emergency team and end-of-life care: A pilot study. Crit Care Resusc. 2007;9:151–156.PubMedGoogle Scholar
  40. 40.
    Calzavacca P, Licari E, Tee A, et al. A prospective study of factors influencing the outcome of patients after a medical emergency team review. Intensive Care Med. 2008;34:2112–2116.CrossRefPubMedGoogle Scholar
  41. 41.
    Chen J, Flabouris A, Bellomo R, Hillman K, Finfer S. The medical emergency team system and not-for-resuscitation orders: Results from the MERIT study. Resuscitation. 2008;79:391–397.CrossRefPubMedGoogle Scholar
  42. 42.
    Chen J, Hillman K, Bellomo R, Flabouris A, Finfer S, Cretikos M. The impact of introducing medical emergency team system on the documentations of vital signs. Resuscitation. 2009;80:35-43.CrossRefPubMedGoogle Scholar
  43. 43.
    Drummond MF, McGuire A. Economic Evaluation in Health Care Merging Theory with Practice. Oxford: Oxford University Press; 2001.Google Scholar
  44. 44.
    Folland S, Stano M, Goodman AC. The Economics of Health and Health Care. 4th ed. Upper Saddle River, NJ: Pearson Prentice Hall; 2004.Google Scholar
  45. 45.
    Altman DG, Bland JM. Absence of evidence is not evidence of absence. BMJ. 1995;311:485.PubMedGoogle Scholar
  46. 46.
    Chen J, Bellomo R, Flabouris A, Hillman K, Finfer S, MERIT Investigators for the Simpson Centre and ANICS Clinical Trials Group. The relationship between early emergency team calls and serious adverse events. Crit Care Med. 2009;37:148–153.CrossRefPubMedGoogle Scholar
  47. 47.
    Spiegelhalter DJ, Abrams KR, Myles JP. Bayesian Approaches to Clinical Trials and Health-Care Evaluation. New York: Wiley; 2004.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  1. 1.The Simpson Centre for Health Services Research, Liverpool Health ServiceSydneyAustralia

Personalised recommendations