Skip to main content
SpringerLink
Log in

Evaluating the ACS NSQIP Risk Calculator in Primary Pancreatic Neuroendocrine Tumor: Results from the US Neuroendocrine Tumor Study Group

  • Original Article
  • Published:
Journal of Gastrointestinal Surgery Aims and scope

Abstract

Background

In a changing health care environment where patient outcomes will be more closely scrutinized, the ability to predict surgical complications is becoming increasingly important. The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) online risk calculator is a popular tool to predict surgical risk. This paper aims to assess the applicability of the ACS NSQIP calculator to patients undergoing surgery for pancreatic neuroendocrine tumors (PNETs).

Methods

Using the US Neuroendocrine Tumor Study Group (USNET-SG), 890 patients who underwent pancreatic procedures between 1/1/2000–12/31/2016 were evaluated. Predicted and actual outcomes were compared using C-statistics and Brier scores.

Results

The most commonly performed procedure was distal pancreatectomy, followed by standard and pylorus-preserving pancreaticoduodenectomy. For the entire group of patients studied, C-statistics were highest for discharge destination (0.79) and cardiac complications (0.71), and less than 0.7 for all other complications. The Brier scores for surgical site infection (0.1441) and discharge to nursing/rehabilitation facility (0.0279) were below the Brier score cut-off, while the rest were equal to or above and therefore not useful for interpretation.

Conclusion

This work indicates that the ACS NSQIP risk calculator is a valuable tool that should be used with caution and in coordination with clinical assessment for PNET clinical decision-making.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Ries LAG, Young JL, Keel GE, Eisner MP, Lin YD, Horner M-JD et al. Cancer Survival Among Adults: U.S. SEER Program, 1988-2001. Patient and Tumor Characteristics. 2007.

  2. Metz DC, Jensen RT. Gastrointestinal neuroendocrine tumors: pancreatic endocrine tumors. Gastroenterology. 2008;135(5):1469–92. doi:https://doi.org/10.1053/j.gastro.2008.05.047.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Fischer L, Kleeff J, Esposito I, Hinz U, Zimmermann A, Friess H et al. Clinical outcome and long-term survival in 118 consecutive patients with neuroendocrine tumours of the pancreas. Br J Surg. 2008;95(5):627–35. doi:https://doi.org/10.1002/bjs.6051.

    Article  CAS  PubMed  Google Scholar 

  4. McPhee JT et al. Perioperative mortality for pancreatectomy: A national perspective. Ann. Surg. 2007; 246:246–253.

    Article  Google Scholar 

  5. Goldman L, Caldera DL, Nussbaum SR, Southwick FS, Krogstad D, Murray B et al. Multifactorial index of cardiac risk in noncardiac surgical procedures. N Engl J Med. 1977;297(16):845–50. doi:https://doi.org/10.1056/NEJM197710202971601.

    Article  CAS  PubMed  Google Scholar 

  6. Lee TH, Marcantonio ER, Mangione CM, Thomas EJ, Polanczyk CA, Cook EF et al. Derivation and prospective validation of a simple index for prediction of cardiac risk of major noncardiac surgery. Circulation. 1999;100(10):1043–9.

    Article  CAS  Google Scholar 

  7. Hill JS, Zhou Z, Simons JP, Ng SC, McDade TP, Whalen GF et al. A simple risk score to predict in-hospital mortality after pancreatic resection for cancer. Ann Surg Oncol. 2010;17(7):1802–7. doi:https://doi.org/10.1245/s10434-010-0947-x.

    Article  PubMed  Google Scholar 

  8. Are C, Afuh C, Ravipati L, Sasson A, Ullrich F, Smith L. Preoperative nomogram to predict risk of perioperative mortality following pancreatic resections for malignancy. J Gastrointest Surg. 2009;13(12):2152–62. doi:https://doi.org/10.1007/s11605-009-1051-z.

    Article  PubMed  Google Scholar 

  9. Parikh P, Shiloach M, Cohen ME, Bilimoria KY, Ko CY, Hall BL et al. Pancreatectomy risk calculator: an ACS-NSQIP resource. HPB (Oxford). 2010;12(7):488–97. doi:https://doi.org/10.1111/j.1477-2574.2010.00216.x.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Bilimoria KY, Liu Y, Paruch JL, Zhou L, Kmiecik TE, Ko CY et al. Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg. 2013;217(5):833–42.e1–3. doi:https://doi.org/10.1016/j.jamcollsurg.2013.07.385.

    Article  PubMed  Google Scholar 

  11. Kohut A, Orfanelli T, Poggio JL, Gibbon D, Buckley De Meritens A, Richard S. Morbidity and Mortality Risk Assessment in Gynecologic Oncology Surgery Using the American College of Surgeons National Surgical Quality Improvement Program Database. Int J Gynecol Cancer. 2018;28(4):840–7. doi:https://doi.org/10.1097/IGC.0000000000001234.

    Article  PubMed  Google Scholar 

  12. Vosler PS, Orsini M, Enepekides DJ, Higgins KM. Predicting complications of major head and neck oncological surgery: an evaluation of the ACS NSQIP surgical risk calculator. J Otolaryngol Head Neck Surg. 2018;47(1):21. doi:https://doi.org/10.1186/s40463-018-0269-8.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Beal EW, Saunders ND, Kearney JF, Lyon E, Wei L, Squires MH et al. Accuracy of the ACS NSQIP Online Risk Calculator Depends on How You Look at It: Results from the United States Gastric Cancer Collaborative. Am Surg. 2018;84(3):358–64.

    PubMed  Google Scholar 

  14. Margolick J, Wiseman SM. Risk of major complications following thyroidectomy and parathyroidectomy: Utility of the NSQIP surgical risk calculator. Am J Surg. 2018;215(5):936–41. doi:https://doi.org/10.1016/j.amjsurg.2018.01.006.

    Article  PubMed  Google Scholar 

  15. Golan S, Adamsky MA, Johnson SC, Barashi NS, Smith ZL, Rodriguez MV et al. National Surgical Quality Improvement Program surgical risk calculator poorly predicts complications in patients undergoing radical cystectomy with urinary diversion. Urol Oncol. 2018;36(2):77.e1-.e7. doi:https://doi.org/10.1016/j.urolonc.2017.09.015.

    Article  Google Scholar 

  16. Jiang HY, Kohtakangas EL, Asai K & Shum JB. Predictive Power of the NSQIP Risk Calculator for Early Post-Operative Outcomes After Whipple: Experience from a Regional Center in Northern Ontario. J Gastrointest. Cancer. 2018;49:288–294.

  17. Beal EW, Lyon E, Kearney J, Wei L, Ethun CG, Black SM et al. Evaluating the American College of Surgeons National Surgical Quality Improvement project risk calculator: results from the U.S. Extrahepatic Biliary Malignancy Consortium. HPB (Oxford). 2017;19(12):1104–11. doi:https://doi.org/10.1016/j.hpb.2017.08.009.

    Article  Google Scholar 

  18. Xia J, Broadhurst DI, Wilson M, Wishart DS. Translational biomarker discovery in clinical metabolomics: an introductory tutorial. Metabolomics. 2013:9, 280–299.

    Article  Google Scholar 

  19. Cohen ME, Ko CY, Bilimoria KY, Zhou L, Huffman K, Wang X et al. Optimizing ACS NSQIP modeling for evaluation of surgical quality and risk: patient risk adjustment, procedure mix adjustment, shrinkage adjustment, and surgical focus. J Am Coll Surg. 2013;217(2):336–46.e1. doi:https://doi.org/10.1016/j.jamcollsurg.2013.02.027.

    Article  Google Scholar 

  20. Merkow RP, Hall BL, Cohen ME, Dimick JB, Wang E, Chow WB et al. Relevance of the c-statistic when evaluating risk-adjustment models in surgery. J Am Coll Surg. 2012;214(5):822–30. doi:https://doi.org/10.1016/j.jamcollsurg.2011.12.041.

    Article  Google Scholar 

  21. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128–38. doi:https://doi.org/10.1097/EDE.0b013e3181c30fb2.

    Article  Google Scholar 

  22. Roberts KJ, Hodson J, Mehrzad H, Marudanayagam R, Sutcliffe RP, Muiesan P et al. A preoperative predictive score of pancreatic fistula following pancreatoduodenectomy. HPB (Oxford). 2014;16(7):620–8. doi:https://doi.org/10.1111/hpb.12186.

    Article  Google Scholar 

  23. Yamamoto Y, Sakamoto Y, Nara S, Esaki M, Shimada K, Kosuge T. A preoperative predictive scoring system for postoperative pancreatic fistula after pancreaticoduodenectomy. World J Surg. 2011;35(12):2747–55. doi:https://doi.org/10.1007/s00268-011-1253-x.

    Article  PubMed  Google Scholar 

  24. Vin Y, Sima CS, Getrajdman GI, Brown KT, Covey A, Brennan MF et al. Management and outcomes of postpancreatectomy fistula, leak, and abscess: results of 908 patients resected at a single institution between 2000 and 2005. J Am Coll Surg. 2008;207(4):490–8. doi:https://doi.org/10.1016/j.jamcollsurg.2008.05.003.

    Article  PubMed  Google Scholar 

  25. Kneuertz PJ, Pitt HA, Bilimoria KY, Smiley JP, Cohen ME, Ko CY et al. Risk of morbidity and mortality following hepato-pancreato-biliary surgery. J Gastrointest Surg. 2012;16(9):1727–35. doi:https://doi.org/10.1007/s11605-012-1938-y.

    Article  PubMed  Google Scholar 

Download references

Authorship

AD, EWB, AGL, GP, EM, FGR, ZK, SRK, VRR, RCF, BAK, KI, PMS, HN, MB, SKM, TMP, CRS, and MD contributed to the conception and design of this work and acquisition of data. EWB, AD, CRS, and MD performed data analysis and interpretation of the data and prepared the manuscript. AD, EWB, AGL, GP, EM, FGR, ZK, SRK, VRR, RCF, BAK, KI, PMS, HN, MB, SKM, TMP, CRS, and MD revised the manuscript critically for important intellectual content, approved the final version to be published, and agree to be accountable for all aspects of the work.

Author information

Authors and Affiliations

  1. Division of Surgical Oncology, The Ohio State University Wexner Medical Center and James Comprehensive Cancer Center, Columbus, OH, USA

    Apeksha Dave, Eliza W. Beal, Timothy M. Pawlik, Carl R. Schmidt & Mary E. Dillhoff

  2. Division of Surgical Oncology, Department of Surgery, Winship Cancer Institute, Emory University, Atlanta, GA, USA

    Alexandra G. Lopez-Aguiar & Shishir K. Maithel

  3. Department of Surgery, Stanford University, Palo Alto, CA, USA

    George Poultsides & Eleftherios Makris

  4. Department of Surgery, Virginia Mason Medical Center, Seattle, WA, USA

    Flavio G. Rocha & Zaheer Kanji

  5. Department of Surgery, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA

    Sean Ronnekleiv-Kelly & Victoria R. Rendell

  6. Department of Surgery, Washington University School of Medicine, St. Louis, MI, USA

    Ryan C. Fields & Bradley A. Krasnick

  7. Division of Surgical Oncology, Department of Surgery, Vanderbilt University, Nashville, TN, USA

    Kamran Idrees & Paula Marincola Smith

  8. Division of Hepatopancreatobiliary and Advanced Gastrointestinal Surgery, Department of Surgery, University of Michigan, Ann Arbor, MI, USA

    Hari Nathan & Megan Beems

  9. Department of Surgery, Division of Surgical Oncology, The Ohio State University Wexner Medical Center, 320 W. 10th Ave, M256 Starling Loving Hall, Columbus, OH, 43210-1267, USA

    Mary E. Dillhoff

Authors
  1. Apeksha Dave
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eliza W. Beal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alexandra G. Lopez-Aguiar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. George Poultsides
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Eleftherios Makris
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Flavio G. Rocha
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Zaheer Kanji
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Sean Ronnekleiv-Kelly
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Victoria R. Rendell
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Ryan C. Fields
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Bradley A. Krasnick
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Kamran Idrees
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Paula Marincola Smith
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Hari Nathan
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Megan Beems
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Shishir K. Maithel
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Timothy M. Pawlik
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. Carl R. Schmidt
    View author publications

    You can also search for this author in PubMed Google Scholar

  19. Mary E. Dillhoff
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mary E. Dillhoff.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This manuscript was presented as a poster at the Americas Hepato-Pancreato-Biliary Association meeting in Miami, FL March 20–24, 2018.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dave, A., Beal, E.W., Lopez-Aguiar, A.G. et al. Evaluating the ACS NSQIP Risk Calculator in Primary Pancreatic Neuroendocrine Tumor: Results from the US Neuroendocrine Tumor Study Group. J Gastrointest Surg 23, 2225–2231 (2019). https://doi.org/10.1007/s11605-019-04120-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11605-019-04120-4

Keywords

Search

Navigation