Evaluating the ACS NSQIP Risk Calculator in Primary Pancreatic Neuroendocrine Tumor: Results from the US Neuroendocrine Tumor Study Group
In a changing health care environment where patient outcomes will be more closely scrutinized, the ability to predict surgical complications is becoming increasingly important. The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) online risk calculator is a popular tool to predict surgical risk. This paper aims to assess the applicability of the ACS NSQIP calculator to patients undergoing surgery for pancreatic neuroendocrine tumors (PNETs).
Using the US Neuroendocrine Tumor Study Group (USNET-SG), 890 patients who underwent pancreatic procedures between 1/1/2000–12/31/2016 were evaluated. Predicted and actual outcomes were compared using C-statistics and Brier scores.
The most commonly performed procedure was distal pancreatectomy, followed by standard and pylorus-preserving pancreaticoduodenectomy. For the entire group of patients studied, C-statistics were highest for discharge destination (0.79) and cardiac complications (0.71), and less than 0.7 for all other complications. The Brier scores for surgical site infection (0.1441) and discharge to nursing/rehabilitation facility (0.0279) were below the Brier score cut-off, while the rest were equal to or above and therefore not useful for interpretation.
This work indicates that the ACS NSQIP risk calculator is a valuable tool that should be used with caution and in coordination with clinical assessment for PNET clinical decision-making.
KeywordsACS NSQIP risk calculator Pancreatic neuroendocrine tumor PNET Pan-NET
AD, EWB, AGL, GP, EM, FGR, ZK, SRK, VRR, RCF, BAK, KI, PMS, HN, MB, SKM, TMP, CRS, and MD contributed to the conception and design of this work and acquisition of data. EWB, AD, CRS, and MD performed data analysis and interpretation of the data and prepared the manuscript. AD, EWB, AGL, GP, EM, FGR, ZK, SRK, VRR, RCF, BAK, KI, PMS, HN, MB, SKM, TMP, CRS, and MD revised the manuscript critically for important intellectual content, approved the final version to be published, and agree to be accountable for all aspects of the work.
- 1.Ries LAG, Young JL, Keel GE, Eisner MP, Lin YD, Horner M-JD et al. Cancer Survival Among Adults: U.S. SEER Program, 1988-2001. Patient and Tumor Characteristics. 2007.Google Scholar
- 4.McPhee JT et al. Perioperative mortality for pancreatectomy: A national perspective. Ann. Surg. 2007; 246:246–253.Google Scholar
- 10.Bilimoria KY, Liu Y, Paruch JL, Zhou L, Kmiecik TE, Ko CY et al. Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg. 2013;217(5):833–42.e1–3. doi: https://doi.org/10.1016/j.jamcollsurg.2013.07.385.CrossRefGoogle Scholar
- 11.Kohut A, Orfanelli T, Poggio JL, Gibbon D, Buckley De Meritens A, Richard S. Morbidity and Mortality Risk Assessment in Gynecologic Oncology Surgery Using the American College of Surgeons National Surgical Quality Improvement Program Database. Int J Gynecol Cancer. 2018;28(4):840–7. doi: https://doi.org/10.1097/IGC.0000000000001234.CrossRefGoogle Scholar
- 13.Beal EW, Saunders ND, Kearney JF, Lyon E, Wei L, Squires MH et al. Accuracy of the ACS NSQIP Online Risk Calculator Depends on How You Look at It: Results from the United States Gastric Cancer Collaborative. Am Surg. 2018;84(3):358–64.Google Scholar
- 15.Golan S, Adamsky MA, Johnson SC, Barashi NS, Smith ZL, Rodriguez MV et al. National Surgical Quality Improvement Program surgical risk calculator poorly predicts complications in patients undergoing radical cystectomy with urinary diversion. Urol Oncol. 2018;36(2):77.e1-.e7. doi: https://doi.org/10.1016/j.urolonc.2017.09.015.CrossRefGoogle Scholar
- 16.Jiang HY, Kohtakangas EL, Asai K & Shum JB. Predictive Power of the NSQIP Risk Calculator for Early Post-Operative Outcomes After Whipple: Experience from a Regional Center in Northern Ontario. J Gastrointest. Cancer. 2018;49:288–294.Google Scholar
- 17.Beal EW, Lyon E, Kearney J, Wei L, Ethun CG, Black SM et al. Evaluating the American College of Surgeons National Surgical Quality Improvement project risk calculator: results from the U.S. Extrahepatic Biliary Malignancy Consortium. HPB (Oxford). 2017;19(12):1104–11. doi: https://doi.org/10.1016/j.hpb.2017.08.009.CrossRefGoogle Scholar
- 18.Xia J, Broadhurst DI, Wilson M, Wishart DS. Translational biomarker discovery in clinical metabolomics: an introductory tutorial. Metabolomics. 2013:9, 280–299.Google Scholar
- 19.Cohen ME, Ko CY, Bilimoria KY, Zhou L, Huffman K, Wang X et al. Optimizing ACS NSQIP modeling for evaluation of surgical quality and risk: patient risk adjustment, procedure mix adjustment, shrinkage adjustment, and surgical focus. J Am Coll Surg. 2013;217(2):336–46.e1. doi: https://doi.org/10.1016/j.jamcollsurg.2013.02.027.
- 20.Merkow RP, Hall BL, Cohen ME, Dimick JB, Wang E, Chow WB et al. Relevance of the c-statistic when evaluating risk-adjustment models in surgery. J Am Coll Surg. 2012;214(5):822–30. doi: https://doi.org/10.1016/j.jamcollsurg.2011.12.041.
- 21.Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128–38. doi: https://doi.org/10.1097/EDE.0b013e3181c30fb2.
- 22.Roberts KJ, Hodson J, Mehrzad H, Marudanayagam R, Sutcliffe RP, Muiesan P et al. A preoperative predictive score of pancreatic fistula following pancreatoduodenectomy. HPB (Oxford). 2014;16(7):620–8. doi: https://doi.org/10.1111/hpb.12186.
- 24.Vin Y, Sima CS, Getrajdman GI, Brown KT, Covey A, Brennan MF et al. Management and outcomes of postpancreatectomy fistula, leak, and abscess: results of 908 patients resected at a single institution between 2000 and 2005. J Am Coll Surg. 2008;207(4):490–8. doi: https://doi.org/10.1016/j.jamcollsurg.2008.05.003.CrossRefGoogle Scholar