Journal of Gastrointestinal Surgery

, Volume 22, Issue 7, pp 1258–1267 | Cite as

Prospective Validation of the Iowa Rectal Surgery Risk Calculator

  • Scott K. ShermanEmail author
  • Jennifer E. Hrabe
  • Emily Huang
  • John W. Cromwell
  • John C. Byrn
Original Article



The Iowa Rectal Surgery Risk Calculator estimates risk for proctectomy procedures. The Iowa Calculator performed well on NSQIP 2010–2011 training and 2005–2009 validation datasets, but was not prospectively validated and did not include low anterior resections. This study sought to demonstrate validity on new independent data, to update the calculator to include low anterior resection, and to compare performance to other risk assessment tools.


Non-emergent ACS-NSQIP proctectomy and low anterior resection data from 2010 to 2015 (n = 65,683) were included. The Iowa Calculator generated risk estimates for 30-day morbidity using 2012–2015 data. An Updated Calculator used 2010–2011 training data to include low anterior resection, with validation on 2012–2015 data. NSQIP data provided NSQIP Morbidity Model predictions and a custom web-script collected ACS-NSQIP Online Surgical Risk Calculator predictions for all patients.


Proctectomy morbidity (not including low anterior resection) decreased from 40.4% in 2010–2011 to 37.0% in 2012–2015. Low anterior resection had lower morbidity (22.4% in 2012–15). The Iowa Calculator demonstrated good discrimination and calibration using 2012–2015 data (C-statistic 0.676, deviance + 9.2%). After including low anterior resection, the Updated Iowa Calculator performed well during training (c-statistic 0.696, deviance 0%) and validation (C-statistic 0.706, deviance + 7.9%). The Updated Iowa Calculator had significantly better discrimination and calibration than morbidity predictions from the ACS Online Calculator (C-statistic 0.693, P < 0.001, deviance − 28.1%) and NSQIP General/Vascular Surgery Model (C-statistic 0.703, P < 0.05, deviance − 40.8%).


When applied to new independent data, the Iowa Calculator supplies accurate risk estimates. The Updated Iowa Calculator includes low anterior resection, and both are prospectively validated. Risk estimation by the Iowa Calculators was superior to ACS-provided risk tools.


Rectal surgery Morbidity Complications Risk calculator 



This study was supported by NIH 5T32#CA148062-03 (SKS and JEH). The authors thank Mary E. Belding-Schmitt, RN, BSN for database support.

Author’s Contributions

Sherman: study design, data analysis, data interpretation, writing, critical revision; Hrabe: study design, data interpretation, writing, critical revision; Huang: data interpretation and critical revision; Cromwell: study design, data interpretation, critical revision; Byrn: study design, data interpretation, critical revision.

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they have no conflicts of interest.

Supplementary material

11605_2018_3770_MOESM1_ESM.docx (28 kb)
ESM 1 (DOCX 27 kb)
11605_2018_3770_MOESM2_ESM.pdf (11 kb)
Fig. S1 (PDF 11 kb)
11605_2018_3770_MOESM3_ESM.pdf (6 kb)
Fig. S2A (PDF 5 kb)
11605_2018_3770_MOESM4_ESM.pdf (6 kb)
Fig. S2B (PDF 5 kb)
11605_2018_3770_MOESM5_ESM.pdf (6 kb)
Fig. S2C (PDF 5 kb)
11605_2018_3770_MOESM6_ESM.pdf (6 kb)
Fig. S2D (PDF 5 kb)


Bolded Names Indicate Co-First Authorship.

  1. 1.
    Balentine, C. J. et al. Waist circumference predicts increased complications in rectal cancer surgery. J. Gastrointest. Surg. 2010 Nov; 14(11):1669–1679.CrossRefPubMedGoogle Scholar
  2. 2.
    Ballian, N. et al. Body mass index does not affect postoperative morbidity and oncologic outcomes of total mesorectal excision for rectal adenocarcinoma. Ann. Surg. Oncol. 2010 Jun;17(6):1606–1613.CrossRefPubMedGoogle Scholar
  3. 3.
    Canedo, J. A., Pinto, R. A., McLemore, E. C., Rosen, L. & Wexner, S. D. Restorative proctectomy with ileal pouch-anal anastomosis in obese patients. Dis. Colon rectum 2010 Jul; 53(7):1030–1034.CrossRefPubMedGoogle Scholar
  4. 4.
    Hrabe, J. E., Sherman, S. K., Charlton, M. E., Cromwell, J. W. & Byrn, J. C. Effect of BMI on outcomes in proctectomy. Dis. Colon rectum 2014 May; 57(5):608–615.Google Scholar
  5. 5.
    Sherman, S. K., Hrabe, J. E., Charlton, M. E., Cromwell, J. W. & Byrn, J. C. Development of an improved risk calculator for complications in proctectomy. J. Gastrointest. Surg. 2014 May; 18(5):986–994.CrossRefPubMedPubMedCentralGoogle Scholar
  6. 6.
    Collins, G. S., Reitsma, J. B., Altman, D. G. & Moons, K. G. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Ann. Intern. Med. 2015 Jan; 162(1):55–63.CrossRefPubMedGoogle Scholar
  7. 7.
    Bilimoria, K. Y. et al. Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J. Am. Coll. Surg. 2013 Nov; 217(5):833–842.CrossRefPubMedPubMedCentralGoogle Scholar
  8. 8.
    User Guide for the 2015 ACS-NSQIP Participant Use Data File (PUF). American College of Surgeons. (Oct. 2016).
  9. 9.
    ACS-NSQIP Surgical Risk Calculator. American College of Surgeons., Accessed February–March, 2017.
  10. 10.
    Hochberg, Y. & Benjamini, Y. More powerful procedures for multiple significance testing. Stat. Med. 1990 Jul; 9(7):811–818.CrossRefPubMedGoogle Scholar
  11. 11.
    Robin, X. et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 2011 Mar; 12:77.CrossRefPubMedPubMedCentralGoogle Scholar
  12. 12.
    Hanley, J. A. & McNeil, B. J. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982 Apr; 143(1):29–36.CrossRefPubMedGoogle Scholar
  13. 13.
    Brier, G. Verification of forecasts expressed in terms of probability. Monthly Weather Review. 1950 Jan; 78(1):1–3.CrossRefGoogle Scholar
  14. 14.
    Wilks, D. S. Statistical Methods in the Atmospheric Sciences. Academic Press, San Diego, CA, Mar. 1995.Google Scholar
  15. 15.
    Bouwmeester, W. et al. Reporting and methods in clinical prediction research: a systematic review. PLoS Med. 2012; 9(5):1–12.CrossRefPubMedGoogle Scholar
  16. 16.
    Collins, G. S. et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med. Res. Methodol. 2014 Mar; 14:40. 17.Google Scholar
  17. 17.
    Moons, K. G. et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann. Intern. Med. 2015 Jan; 162(1):1–73.CrossRefGoogle Scholar
  18. 18.
    House, M. G. et al. Survival after hepatic resection for metastatic colorectal cancer: trends in outcomes for 1,600 patients during two decades at a single institution. J. Am. Coll. Surg. 2010 May; 210(5):744–752.CrossRefPubMedGoogle Scholar
  19. 19.
    Molena, D., Mungo, B., Stem, M., Feinberg, R. L. & Lidor, A. O. Prevalence, impact, and risk factors for hospital-acquired conditions after major surgical resection for cancer: a NSQIP analysis. J. Gastrointest. Surg. 2015 Jan; 19(1):142–151.CrossRefPubMedGoogle Scholar
  20. 20.
    Nygren, J. et al. Guidelines for perioperative care in elective rectal/pelvic surgery: Enhanced Recovery After Surgery (ERAS) Society recommendations. World J. Surg. 2013 Feb; 37(2):285–305.CrossRefPubMedGoogle Scholar
  21. 21.
    Rosenberger, L. H., Politano, A. D. & Sawyer, R. G. The surgical care improvement project and prevention of post-operative infection, including surgical site infection. Surg. Infect. (Larchmt) 2011 Jun; 12(3):163–168.Google Scholar
  22. 22.
    Arriaga, A. F. et al. The better colectomy project: association of evidence-based best-practice adherence rates to outcomes in colorectal surgery. Ann. Surg. 2009 Oct; 250(4):507–513.PubMedGoogle Scholar
  23. 23.
    Ohman, K. A. et al. Combination of Oral Antibiotics and Mechanical Bowel Preparation Reduces Surgical Site Infection in Colorectal Surgery. J. Am. Coll. Surg. 2017 Oct; 225(4):465–471.CrossRefPubMedGoogle Scholar
  24. 24.
    Cannon, J. A. et al. Preoperative oral antibiotics reduce surgical site infection following elective colorectal resections. Dis. Colon rectum 2012 Nov; 55(11):1160–1166.CrossRefPubMedGoogle Scholar
  25. 25.
    Connolly, T. M., Foppa, C., Kazi, E., Denoya, P. I. & Bergamaschi, R. Impact of a surgical site infection reduction strategy after colorectal resection. Colorectal Dis. 2016 Sep; 18(9):910–918.CrossRefPubMedGoogle Scholar
  26. 26.
    Cook, N. R. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 2007 Feb; 115(7), 928–935.CrossRefPubMedGoogle Scholar
  27. 27.
    Cohen, M. E., Liu, Y., Ko, C. Y. & Hall, B. L. An Examination of American College of Surgeons NSQIP Surgical Risk Calculator Accuracy. J. Am. Coll. Surg. 2017 May; 224(5):787–795.CrossRefPubMedGoogle Scholar
  28. 28.
    Liu, Y., Cohen, M. E., Hall, B. L., Ko, C. Y. & Bilimoria, K. Y. Evaluation and Enhancement of Calibration in the American College of Surgeons NSQIP Surgical Risk Calculator. J. Am. Coll. Surg. 2016 Aug; 223(2):231–239.CrossRefPubMedGoogle Scholar
  29. 29.
    Kramer, A. A. & Zimmerman, J. E. Assessing the calibration of mortality benchmarks in critical care: The Hosmer-Lemeshow test revisited. Crit. Care Med. 2007 Sep; 35(9):2052–2056. 30.Google Scholar
  30. 30.
    Paul, P., Pennell, M. L. & Lemeshow, S. Standardizing the power of the Hosmer-Lemeshow goodness of fit test in large data sets. Stat. Med. 2013 Jan; 32(1):67–80.CrossRefPubMedGoogle Scholar
  31. 31.
    Paruch, J. L., Ko, C. Y. & Bilimoria, K. Y. An opportunity to improve informed consent and shared decision making: the role of the ACS NSQIP Surgical Risk Calculator in oncology. Ann. Surg. Oncol. 2014 Jan; 21(1):5–7.CrossRefPubMedGoogle Scholar
  32. 32.
    Wanderer, J. P. & Ehrenfeld, J. M. Toward External Validation and Routine Clinical Use of the American College of Surgeons NSQIP Surgical Risk Calculator. J. Am. Coll. Surg. 2016 Oct; 223(4):674.CrossRefPubMedGoogle Scholar
  33. 33.
    Liu, Y., Cohen, M. E., Ko, C. Y., Bilimoria, K. Y. & Hall, B. L. Considerations in Releasing Equations for the American College of Surgeons NSQIP Surgical Risk Calculator: In Reply to Wanderer and Ehrenfeld. J. Am. Coll. Surg. 2016 Oct; 223(4):674–675.Google Scholar

Copyright information

© The Society for Surgery of the Alimentary Tract 2018

Authors and Affiliations

  • Scott K. Sherman
    • 1
    Email author
  • Jennifer E. Hrabe
    • 2
  • Emily Huang
    • 1
  • John W. Cromwell
    • 2
  • John C. Byrn
    • 3
  1. 1.Department of SurgeryUniversity of ChicagoChicagoUSA
  2. 2.Department of SurgeryUniversity of Iowa Carver College of MedicineIowa CityUSA
  3. 3.Department of SurgeryUniversity of MichiganAnn ArborUSA

Personalised recommendations