Skip to main content
SpringerLink
Log in

Development and Validation of Machine Learning Models to Predict Readmission After Colorectal Surgery

  • Original Article
  • Published:
Journal of Gastrointestinal Surgery

Abstract

Background

Readmission after colorectal surgery is common and often implies complications for patients and costs for hospitals. Previous works have created predictive models using logistic regression for this outcome but have shown limited accuracy. Machine learning has shown promise in improving predictions by identifying non-linear patterns in data. We sought to create a more accurate predictive model for readmission after colorectal surgery using machine learning.

Methods

Patients who underwent colorectal surgery were identified in the National Quality Improvement Program (NSQIP) database including years 2012–2019 and split into training, validation, and test sets. The primary outcome was readmission within 30 days of surgery. Three types of machine learning models were created, including random forest (RF), gradient boosting (XGB), and neural network (NN). A logistic regression (LR) model was also created for comparison. Model performance was evaluated using area under the receiver operating characteristic curve (AUROC).

Results

The dataset included 213,827 patients after application of exclusion criteria. A total of 23,083 (10.8%) of patients experienced readmission. NN obtained an AUROC of 0.751 (95% CI 0.743–0.759), compared with 0.684 (95% CI 0.676–0.693) for LR. RF and XGB performed similarly with AUROCs of 0.749 (95% CI 0.741–0.757) and 0.745 (95% CI 0.737–0.753) respectively. Ileus, index admission length of stay, organ-space surgical site infection present at time of surgery, and ostomy placement were identified as the most contributory variables.

Conclusions

Machine learning approaches outperformed traditional statistical methods in the prediction of readmission after colorectal surgery. After external validation, this improved prediction model could be used to target interventions to reduce readmission rate.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Guinier D, Mantion GA, Alves A, Kwiatkowski F, Slim K, Panis Y (2007) Risk factors of unplanned readmission after colorectal surgery: A prospective, multicenter study. Dis Colon Rectum 50:1316–1323 . https://doi.org/10.1007/S10350-007-0310-X

    Article  PubMed  Google Scholar 

  2. Damle RN, Alavi K (2016) Risk factors for 30-d readmission after colorectal surgery: a systematic review. J Surg Res 200:200–207 . https://doi.org/10.1016/J.JSS.2015.06.052

    Article  PubMed  Google Scholar 

  3. Albright J, Batool F, Cleary RK, Mullard AJ, Kreske E, Ferraro J, Regenbogen SE (2019) Complications after discharge predict readmission after colorectal surgery. Surg Endosc 33:1216 . https://doi.org/10.1007/S00464-018-6398-3

    Article  PubMed  Google Scholar 

  4. Chen SY, Stem M, Gearhart SL, Safar B, Fang SH, Azad NS, Murphy AG, Narang AK, Wolfgang CL, Efron JE (2019) Readmission Adversely Affects Survival in Surgical Rectal Cancer Patients. World J Surg 43:2506–2517 . https://doi.org/10.1007/S00268-019-05053-4

    Article  PubMed  Google Scholar 

  5. Damle RN, Cherng NB, Flahive JM, Davids JS, Maykel JA, Sturrock PR, Sweeney WB, Alavi K (2014) Clinical and financial impact of hospital readmissions after colorectal resection: predictors, outcomes, and costs. Dis Colon Rectum 57:1421–1429 . https://doi.org/10.1097/DCR.0000000000000251

    Article  PubMed  Google Scholar 

  6. Iqbal A, Sakharuk I, Goldstein L, Tan SA, Qiu P, Li Z, Hughes SJ (2018) Readmission After Elective Ileostomy in Colorectal Surgery Is Predictable. JSLS J Soc Laparoendosc Surg 22:

  7. Tatar C, Benlice C, Delaney CP, Holubar SD, Liska D, Steele SR, Gorgun E (2020) Modified frailty index predicts high-risk patients for readmission after colorectal surgery for cancer. In: American Journal of Surgery. Am J Surg, pp 187–190

  8. Hill SS, Harnsberger CR, Crawford AS, Hoang CM, Davids JS, Sturrock PR, Maykel JA, Alavi K (2020) Creation and institutional validation of a readmission risk calculator for elective colorectal surgery. Dis Colon Rectum 10:1436–1445 . https://doi.org/10.1097/DCR.0000000000001674

    Article  Google Scholar 

  9. Liu C, Bhat S, Sharma P, Yuan L, O’Grady G, Bissett I (2021) Risk factors for readmission with dehydration after ileostomy formation: A systematic review and meta-analysis. Color Dis 23:1071–1082 . https://doi.org/10.1111/codi.15566

    Article  Google Scholar 

  10. 11. Jones CE, Hollis RH, Wahl TS, Oriel BS, Itani KMF, Morris MS, Hawn MT (2016) Transitional care interventions and hospital readmissions in surgical populations: a systematic review. Am J Surg 212:327–335 . https://doi.org/10.1016/J.AMJSURG.2016.04.004

    Article  PubMed  Google Scholar 

  11. 12. Shameer K, Johnson KW, Yahi A, Miotto R, Li LI, Ricks D, Jebakaran J, Kovatch P, Sengupta PP, Gelijns A, Moskovitz A, Darrow B, Reich DL, Kasarskis A, Tatonetti NP, Pinney S, Dudley JT (2017) Predictive modeling of hospital readmission rates using electronic medical record-wide machine learning: A case-study using mount sinai heart failure cohort. Pacific Symp Biocomput 0:276–287 . https://doi.org/10.1142/9789813207813_0027

    Article  Google Scholar 

  12. 13. Min X, Yu B, Wang F (2019) Predictive Modeling of the Hospital Readmission Risk from Patients’ Claims Data Using Machine Learning: A Case Study on COPD. Sci Rep 9:1–10 . https://doi.org/10.1038/s41598-019-39071-y

    Article  Google Scholar 

  13. 14. Bolourani S, Tayebi MA, Diao L, Wang P, Patel V, Manetta F, Lee PC (2021) Using machine learning to predict early readmission following esophagectomy. J Thorac Cardiovasc Surg 161:1926-1939.e8 . https://doi.org/10.1016/j.jtcvs.2020.04.172

    Article  PubMed  Google Scholar 

  14. 15. Goyal A, Ngufor C, Kerezoudis P, McCutcheon B, Storlie C, Bydon M (2019) Can machine learning algorithms accurately predict discharge to nonhome facility and early unplanned readmissions following spinal fusion? Analysis of a national surgical registry: Presented at the 2019 AANS/CNS Section on Disorders of the Spine and Peripheral Nerves. J Neurosurg Spine 31:568–578 . https://doi.org/10.3171/2019.3.SPINE181367

    Article  Google Scholar 

  15. 16. JM H, S W, J C, M D, A T, TM P, A E (2020) Can We Improve Prediction of Adverse Surgical Outcomes? Development of a Surgical Complexity Score Using a Novel Machine Learning Technique. J Am Coll Surg 230:43-52.e1 . https://doi.org/10.1016/J.JAMCOLLSURG.2019.09.015

    Article  Google Scholar 

  16. 17. Francis NK, Luther A, Salib E, Allanby L, Messenger D, Allison AS, Smart NJ, Ockrim JB (2015) The use of artificial neural networks to predict delayed discharge and readmission in enhanced recovery following laparoscopic colorectal cancer surgery. Tech Coloproctol 19:419–428 . https://doi.org/10.1007/S10151-015-1319-0

    Article  PubMed  Google Scholar 

  17. 18. Moons KGM, Altman DG, Reitsma JB, Ioannidis JPA, Macaskill P, Steyerberg EW, Vickers AJ, Ransohoff DF, Collins GS (2015) Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 162:W1–W73 . https://doi.org/10.7326/M14-0698

    Article  PubMed  Google Scholar 

  18. ACS NSQIP Participant Use Data File. https://www.facs.org/quality-programs/acs-nsqip/participant-use. Accessed 22 Dec 2021

  19. 20. Chen SY, Stem M, Cerullo M, Canner JK, Gearhart SL, Safar B, Fang SH, Efron JE (2018) Predicting the risk of readmission from dehydration after ileostomy formation: The dehydration readmission after ileostomy prediction score. Dis Colon Rectum 61:1410–1417 . https://doi.org/10.1097/DCR.0000000000001217

    Article  PubMed  PubMed Central  Google Scholar 

  20. 21. Lucas DJ, Haut ER, Hechenbleikner EM, Wick EC, Pawlik TM (2014) Avoiding immortal time bias in the American College of Surgeons National Surgical Quality Improvement Program readmission measure. JAMA Surg 149:875–877 . https://doi.org/10.1001/JAMASURG.2014.115

    Article  PubMed  Google Scholar 

  21. 22. D B, J D, GC V, HMA K (2018) Surgical Risk Is Not Linear: Derivation and Validation of a Novel, User-friendly, and Machine-learning-based Predictive OpTimal Trees in Emergency Surgery Risk (POTTER) Calculator. Ann Surg 268:574–583 . https://doi.org/10.1097/SLA.0000000000002956

    Article  Google Scholar 

  22. 23. Merath K, Hyer JM, Mehta R, Farooq A, Bagante F, Sahara K, Tsilimigras DI, Beal E, Paredes AZ, Wu L, Ejaz A, Pawlik TM (2019) Use of Machine Learning for Prediction of Patient Risk of Postoperative Complications After Liver, Pancreatic, and Colorectal Surgery. J Gastrointest Surg 2019 248 24:1843–1851 . https://doi.org/10.1007/S11605-019-04338-2

    Article  Google Scholar 

  23. Géron A (2019) Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow: Concepts, tools, and techniques to build intelligent systems. O’Reilly Media

  24. 25. Nudel J, Bishara AM, de Geus SWL, Patil P, Srinivasan J, Hess DT, Woodson J (2020) Development and validation of machine learning models to predict gastrointestinal leak and venous thromboembolism after weight loss surgery: an analysis of the MBSAQIP database. Surg Endosc. https://doi.org/10.1007/s00464-020-07378-x

    Article  PubMed  PubMed Central  Google Scholar 

  25. 26. Chen SY, Stem M, Cerullo M, Canner JK, Gearhart SL, Safar B, Fang SH, Efron JE (2018) Predicting the risk of readmission from dehydration after ileostomy formation: The DRIP score. Dis Colon Rectum 61:1410 . https://doi.org/10.1097/DCR.0000000000001217

    Article  PubMed  PubMed Central  Google Scholar 

  26. 27. McKenna NP, Bews KA, Cima RR, Crowson CS, Habermann EB (2020) Development of a Risk Score to Predict Anastomotic Leak After Left-Sided Colectomy: Which Patients Warrant Diversion? J Gastrointest Surg 24:132–143 . https://doi.org/10.1007/s11605-019-04293-y

    Article  PubMed  Google Scholar 

  27. 28. Alqahtani M, Garfinkle R, Zhao K, Vasilevsky CA, Morin N, Ghitulescu G, Faria J, Boutros M (2020) Can we better predict readmission for dehydration following creation of a diverting loop ileostomy: development and validation of a prediction model and web-based risk calculator. Surg Endosc 34:3118–3125 . https://doi.org/10.1007/s00464-019-07069-2

    Article  PubMed  Google Scholar 

  28. 29. Rencuzogullari A, Benlice C, Valente M, Abbas MA, Remzi FH, Gorgun E (2017) Predictors of anastomotic leak in elderly patients after colectomy: nomogram-based assessment from the American College of Surgeons National Surgical Quality Program Procedure-Targeted Cohort. Dis Colon Rectum 60:527–536 . https://doi.org/10.1097/DCR.0000000000000789

    Article  PubMed  Google Scholar 

  29. 30. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nat 2015 5217553 521:436–444 . https://doi.org/10.1038/nature14539

    Article  Google Scholar 

  30. 31. Pollard TJ, Johnson AEW, Raffa JD, Mark RG (2018) tableone: An open source Python package for producing summary statistics for research papers. JAMIA Open 1:26 . https://doi.org/10.1093/JAMIAOPEN/OOY012

    Article  PubMed  PubMed Central  Google Scholar 

  31. 32. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: Machine Learning in Python. J Mach Learn Res 12:2825–2830

    Google Scholar 

  32. 33. Bowers AJ, Zhou X (2019) Receiver Operating Characteristic (ROC) Area Under the Curve (AUC): A Diagnostic Measure for Evaluating the Accuracy of Predictors of Education Outcomes. J Educ Students Placed Risk 24:20–46 . https://doi.org/10.1080/10824669.2018.1523734

    Article  Google Scholar 

  33. Lundberg SM, Allen PG, Lee S-I A Unified Approach to Interpreting Model Predictions

  34. 35. Cologne KG, Keller DS, Liwanag L, Devaraj B, Senagore AJ (2015) Use of the American College of Surgeons NSQIP Surgical Risk Calculator for Laparoscopic Colectomy: How Good Is It and How Can We Improve It? J Am Coll Surg 220:281–286 . https://doi.org/10.1016/J.JAMCOLLSURG.2014.12.007

    Article  PubMed  Google Scholar 

  35. 36. Hyde LZ, Valizadeh N, Al-Mazrou AM, Kiran RP (2019) ACS-NSQIP risk calculator predicts cohort but not individual risk of complication following colorectal resection. Am J Surg 218:131–135 . https://doi.org/10.1016/j.amjsurg.2018.11.017

    Article  PubMed  Google Scholar 

  36. 37. Hyer JM, White S, Cloyd J, Dillhoff M, Tsung A, Pawlik TM, Ejaz A (2020) Can We Improve Prediction of Adverse Surgical Outcomes? Development of a Surgical Complexity Score Using a Novel Machine Learning Technique. J Am Coll Surg 230:43-52.e1 . https://doi.org/10.1016/J.JAMCOLLSURG.2019.09.015

    Article  PubMed  Google Scholar 

  37. 38. Hopkins BS, Yamaguchi JT, Garcia R, Kesavabhotla K, Weiss H, Hsu WK, Smith ZA, Dahdaleh NS (2020) Using machine learning to predict 30-day readmissions after posterior lumbar fusion: An NSQIP study involving 23,264 patients. J Neurosurg Spine 32:399–406 . https://doi.org/10.3171/2019.9.SPINE19860

    Article  Google Scholar 

  38. 39. Zack CJ, Senecal C, Kinar Y, Metzger Y, Bar-Sinai Y, Widmer RJ, Lennon R, Singh M, Bell MR, Lerman A, Gulati R (2019) Leveraging Machine Learning Techniques to Forecast Patient Prognosis After Percutaneous Coronary Intervention. Cardiovasc Interv 12:1304–1311 . https://doi.org/10.1016/J.JCIN.2019.02.035

    Article  Google Scholar 

  39. Hospital Readmissions Reduction Program (HRRP) | CMS. https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/AcuteInpatientPPS/Readmissions-Reduction-Program. Accessed 25 Jan 2022

  40. 41. Luo W, Holubar S, Bordeianou L, Crawford L, Hall B, Hilbert N, Hull T, Hyman N, Keenan M, Kunitake H, Lee E, Lewis W, Maron D, Messaris E, Miller R, Mutch M, Ortenzi G, Ramamoorthy S, Smith R, Steinhagen R, Eisenstein S (2020) P015 ILEOCOLIC RESECTION FOR CROHN’S DISEASE: CURRENT TRENDS FROM ANALYSIS OF 2 YEARS OF NSQIP IBD COLLABORATIVE DATA. Gastroenterology 158:S83–S84 . https://doi.org/10.1053/J.GASTRO.2019.11.212

    Article  Google Scholar 

Download references

Funding

This work was supported by funding from the National Institutes of Health (Program in Translational Medicine T32-CA244125 to UNC/KAC).

Author information

Authors and Affiliations

  1. Department of Surgery, University of North Carolina, 100 Manning Drive, Burnett Womack Building, Suite 4038, Chapel Hill, NC, 27599, USA

    Kevin A. Chen, Karyn B. Stitzenberg, Jonathan Stem, Jose G. Guillem & Muneera R. Kapadia

  2. Joint Department of Biomedical Engineering, University of North Carolina, 10202C Mary Ellen Jones Building, Chapel Hill, NC, 27599, USA

    Chinmaya U. Joisa & Shawn M. Gomez

Authors
  1. Kevin A. Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chinmaya U. Joisa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Karyn B. Stitzenberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jonathan Stem
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jose G. Guillem
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shawn M. Gomez
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Muneera R. Kapadia
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

KA Chen: project development, data management, data analysis, and manuscript writing/editing. ME Berginski: data analysis and manuscript editing. K Stitzenberg: project development and manuscript editing. J Stem: project development and manuscript editing. JG Guillem: project development and manuscript editing. SM Gomez: project development, data analysis, and manuscript editing. MR Kapadia: project development and manuscript writing/editing.

Corresponding author

Correspondence to Muneera R. Kapadia.

Ethics declarations

Conflict of Interest

The authors declare no competing interests.

Additional information

Presentation

This work was presented at SAGES 2022 in Denver, CO (3/17/2022).

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 14 KB)

Supplementary file2 (DOCX 12 KB)

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, K.A., Joisa, C.U., Stitzenberg, K.B. et al. Development and Validation of Machine Learning Models to Predict Readmission After Colorectal Surgery. J Gastrointest Surg 26, 2342–2350 (2022). https://doi.org/10.1007/s11605-022-05443-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11605-022-05443-5

Keywords

Search

Navigation