Skip to main content

Advertisement

SpringerLink
Log in

Derivation, Validation and Application of a Pragmatic Risk Prediction Index for Benchmarking of Surgical Outcomes

  • Original Scientific Report
  • Published:
World Journal of Surgery Aims and scope Submit manuscript

Abstract

Background

Despite the existence of multiple validated risk assessment and quality benchmarking tools in surgery, their utility outside of high-income countries is limited. We sought to derive, validate and apply a scoring system that is both (1) feasible, and (2) reliably predicts mortality in a middle-income country (MIC) context.

Methods

A 5-step methodology was used: (1) development of a de novo surgical outcomes database modeled around the American College of Surgeons’ National Surgical Quality Improvement Program (ACS-NSQIP) in South Africa (SA dataset), (2) use of the resultant data to identify all predictors of in-hospital death with more than 90% capture indicating feasibility of collection, (3) use these predictors to derive and validate an integer-based score that reliably predicts in-hospital death in the 2012 ACS-NSQIP, (4) apply the score in the original SA dataset and demonstrate its performance, (5) identify threshold cutoffs of the score to prompt action and drive quality improvement.

Results

Following step one-three above, the 13 point Codman’s score was derived and validated on 211,737 and 109,079 patients, respectively, and includes: age 65 (1), partially or completely dependent functional status (1), preoperative transfusions ≥4 units (1), emergency operation (2), sepsis or septic shock (2) American Society of Anesthesia score ≥3 (3) and operative procedure (1–3). Application of the score to 373 patients in the SA dataset showed good discrimination and calibration to predict an in-hospital death. A Codman Score of 8 is an optimal cutoff point for defining expected and unexpected deaths.

Conclusion

We have designed a novel risk prediction score specific for a MIC context. The Codman Score can prove useful for both (1) preoperative decision-making and (2) benchmarking the quality of surgical care in MIC’s.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Haider AH, Hashmi ZG, Gupta S, Zafar SN, David JS, Efron DT et al (2014) Benchmarking of trauma care worldwide: the potential value of an International Trauma Data Bank (ITDB). World J Surg 38(8):1882–1891. doi:10.1007/s00268-014-2629-5

    Article  PubMed  Google Scholar 

  2. Pearse RM, Moreno RP, Bauer P, Pelosi P, Metnitz P, Spies C et al (2012) Mortality after surgery in Europe: a 7 day cohort study. Lancet 380(9847):1059–1065

    Article  PubMed  PubMed Central  Google Scholar 

  3. Haynes AB, Weiser TG, Berry WR, Lipsitz SR, Breizat AH, Dellinger EP et al (2009) A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med 360(5):491–499

    Article  CAS  PubMed  Google Scholar 

  4. Meara JG, Greenberg SL (2015) Global surgery as an equal partner in health: no longer the neglected stepchild. Lancet Glob Health 27(3 Suppl 2):S1–S2

    Article  Google Scholar 

  5. Farmer DL (2012) NSQIP lite: a potential tool for global comparative effectiveness evaluations. Arch Surg 147(9):803–804

    Article  PubMed  Google Scholar 

  6. Hall BL, Hamilton BH, Richards K, Bilimoria KY, Cohen ME, Ko CY (2009) Does surgical quality improve in the American College of Surgeons National Surgical Quality Improvement Program: an evaluation of all participating hospitals. Ann Surg 250(3):363–376

    PubMed  Google Scholar 

  7. Cohen ME, Liu Y, Ko CY, Hall BL (2015) Improved surgical outcomes for ACS NSQIP hospitals over time: evaluation of hospital cohorts with up to 8 years of participation. Ann Surg 263(2):267–273

    Article  Google Scholar 

  8. Haynes AB, Regenbogen SE, Weiser TG, Lipsitz SR, Dziekan G, Berry WR et al (2011) Surgical outcome measurement for a global patient population: validation of the Surgical Apgar Score in 8 countries. Surgery 149(4):519–524

    Article  PubMed  Google Scholar 

  9. Berwick DM (2015) Measuring surgical outcomes for improvement: was Codman wrong? JAMA 313(5):469–470

    Article  CAS  PubMed  Google Scholar 

  10. Project REDCap (2015). http://www.project-redcap.org/. Accessed 24 June 2015

  11. Johnson RG, Arozullah AM, Neumayer L, Henderson WG, Hosokawa P, Khuri SF (2007) Multivariable predictors of postoperative respiratory failure after general and vascular surgery: results from the patient safety in surgery study. J Am Coll Surg 204(6):1188–1198

    Article  PubMed  Google Scholar 

  12. Andersen J, Lassiter R, Bickler S, Talamini M, Chang D (2012) Brief tool to measure risk-adjusted surgical outcomes in resource-limited hospitals. Arch Surg 147(9):798–803

    Article  Google Scholar 

  13. Cook F (2014) Analytical epidemiology lecture series. Harvard TH Chan School of Public Health, Program in Clinical Effectiveness. https://www.hsph.harvard.edu/clinical-effectiveness/2014

  14. Lee TH, Marcantonio ER, Mangione CM, Thomas EJ, Polanczyk CA, Cook EF et al (1999) Derivation and prospective validation of a simple index for prediction of cardiac risk of major noncardiac surgery. Circulation 100(10):1043–1049

    Article  CAS  PubMed  Google Scholar 

  15. Batterton KA, Schubert CM (2015) A nonparametric fiducial interval for the Youden index in multi-state diagnostic settings. Stat Med 35(1):78–96

    Article  PubMed  Google Scholar 

  16. Rubinfeld I, Farooq M, Velanovich V, Syed Z (2010) Predicting surgical risk: how much data is enough? AMIA Annu Symp Proc. 2010:777–781

    PubMed  PubMed Central  Google Scholar 

  17. Dimick JB, Osborne NH, Hall BL, Ko CY, Birkmeyer JD (2010) Risk adjustment for comparing hospital quality with surgery: how many variables are needed? J Am Coll Surg 210(4):503–508

    Article  PubMed  PubMed Central  Google Scholar 

  18. Ariyaratnam R, Palmqvist CL, Hider P, Laing GL, Stupart D, Wilson L et al (2015) Toward a standard approach to measurement and reporting of perioperative mortality rate as a global indicator for surgery. Surgery 158(1):17–26

    Article  PubMed  Google Scholar 

  19. Henderson WG, Daley J (2009) Design and statistical methodology of the National Surgical Quality Improvement Program: why is it what it is? Am J Surg 198(5 Suppl):S19–S27

    Article  PubMed  Google Scholar 

  20. Haynes SR, Lawler PG (1995) An assessment of the consistency of ASA physical status classification allocation. Anaesthesia 50(3):195–199

    Article  CAS  PubMed  Google Scholar 

  21. Cohen ME, Bilimoria KY, Ko CY, Richards K, Hall BL (2009) Effect of subjective preoperative variables on risk-adjusted assessment of hospital morbidity and mortality. Ann Surg 249(4):682–689

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of General Surgery, Codman Center for Clinical Effectiveness in Surgery, Massachusetts General Hospital, Boston, MA, USA

    Richard T. Spence, David C. Chang, Haytham M. A. Kaafarani & Matthew M. Hutter

  2. Department of Surgery, University of Cape Town, Cape Town, South Africa

    Richard T. Spence & Eugenio Panieri

  3. Harvard Medical School, Boston, MA, USA

    David C. Chang, Haytham M. A. Kaafarani, Geoffrey A. Anderson & Matthew M. Hutter

Authors
  1. Richard T. Spence
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David C. Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haytham M. A. Kaafarani
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Eugenio Panieri
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Geoffrey A. Anderson
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Matthew M. Hutter
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Richard T. Spence.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 19 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Spence, R.T., Chang, D.C., Kaafarani, H.M.A. et al. Derivation, Validation and Application of a Pragmatic Risk Prediction Index for Benchmarking of Surgical Outcomes. World J Surg 42, 533–540 (2018). https://doi.org/10.1007/s00268-017-4177-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00268-017-4177-2

Search

Navigation