Abstract
Background
Rectal tumors display varying degrees of response to total neoadjuvant therapy (TNT). We evaluated the performance of a convolutional neural network (CNN) in interpreting endoscopic images of either a non-complete response to TNT or local regrowth during watch-and-wait surveillance.
Methods
Endoscopic images from stage II/III rectal cancers treated with TNT from 2012 to 2020 at a single institution were retrospectively reviewed. Images were labelled as Tumor or No Tumor based on endoscopy timing (before, during, or after treatment) and the tumor’s endoluminal response. A CNN was trained using ResNet-50 architecture. The area under the curve (AUC) was analyzed during training and for two test sets. The main test set included images of tumors treated with TNT. The other contained images of local regrowth. The model’s performance was compared to sixteen surgeons and surgical trainees who evaluated 119 images for evidence of tumor. Fleiss’ kappa was calculated by respondent experience level.
Results
A total of 2717 images from 288 patients were included; 1407 (51.8%) contained tumor. The AUC was 0.99, 0.98, and 0.92 for training, main test, and local regrowth test sets. The model performed on par with surgeons of all experience levels for the main test set. Interobserver agreement was good (\(k\) = 0.71–0.81). All groups outperformed the model in identifying tumor from images of local regrowth. Interobserver agreement was fair to moderate (\(k\)= 0.24–0.52).
Conclusions
A highly accurate CNN matched the performance of colorectal surgeons in identifying a noncomplete response to TNT. However, the model demonstrated suboptimal accuracy when analyzing images of local regrowth.
Similar content being viewed by others
References
Langenfeld SJ, Davis BR, Vogel JD, et al. The American Society of Colon and Rectal Surgeons clinical practice guidelines for the management of rectal cancer 2023 supplement. Dis Colon Rectum. 2024;67(1):18–31. https://doi.org/10.1097/DCR.0000000000003057.
Garcia-Aguilar J, Patil S, Gollub MJ, et al. Organ preservation in patients with rectal adenocarcinoma treated with total neoadjuvant therapy. J Clin Oncol. 2022;40(23):2546–56. https://doi.org/10.1200/JCO.22.00032.
Verheij FS, Omer DM, Williams H, et al. Long-term results of organ preservation in patients with rectal adenocarcinoma treated with total neoadjuvant therapy: the randomized phase II OPRA trial. J Clin Oncol. 2023. https://doi.org/10.1200/JCO.23.01208.
Maas M, Lambregts DMJ, Nelemans PJ, et al. Assessment of clinical complete response after chemoradiation for rectal cancer with digital rectal examination, endoscopy, and MRI: Selection for organ-saving treatment. Ann Surg Oncol. 2015;22(12):3873–80. https://doi.org/10.1245/s10434-015-4687-9.
Ko HM, Choi YH, Lee JE, Lee KH, Kim JY, Kim JS. Combination assessment of clinical complete response of patients with rectal cancer following chemoradiotherapy with endoscopy and magnetic resonance imaging. Ann Coloproctol. 2019;35(4):202–8. https://doi.org/10.3393/ac.2018.10.15.
Van Der Paardt MP, Zagers MB, Beets-Tan RGH, Stoker J, Bipat S. Patients who undergo preoperative chemoradiotherapy for locally advanced rectal cancer restaged by using diagnostic MR imaging: a systematic review and meta-analysis. Radiology. 2013. https://doi.org/10.1148/radiol.13122833/-/DC1.
Kawai K, Ishihara S, Nozawa H, et al. Prediction of pathological complete response using endoscopic findings and outcomes of patients who underwent watchful waiting after chemoradiotherapy for rectal cancer. Dis Colon Rectum. 2017;60(4):368–75. https://doi.org/10.1097/DCR.0000000000000742.
Smith FM, Chang KH, Sheahan K, Hyland J, O’Connell PR, Winter DC. The surgical significance of residual mucosal abnormalities in rectal cancer following neoadjuvant chemoradiotherapy. Br J Surg. 2012;99(7):993–1001. https://doi.org/10.1002/bjs.8700.
Smith FM, Wiland H, Mace A, Pai RK, Kalady MF. Clinical criteria underestimate complete pathological response in rectal cancer treated with neoadjuvant chemoradiotherapy. Dis Colon Rectum. 2014;57(3):311–5. https://doi.org/10.1097/DCR.0b013e3182a84eba.
Williams H, Thompson H, Lin S, et al. Endoscopic predictors of residual tumor after total neoadjuvant therapy: a post hoc analysis from the organ preservation in rectal adenocarcinoma (OPRA) trial. Dis Colon Rectum. 2023. https://doi.org/10.1097/DCR.0000000000003096.
Chadi SA, Malcomson L, Ensor J, et al. Factors affecting local regrowth after watch and wait for patients with a clinical complete response following chemoradiotherapy in rectal cancer (InterCoRe consortium): an individual participant data meta-analysis. Lancet Gastroenterol Hepatol. 2018;3(12):825–36.
Dattani M, Heald RJ, Goussous G, et al. Oncological and survival outcomes in watch and wait patients with a clinical complete response after neoadjuvant chemoradiotherapy for rectal cancer a systematic review and pooled analysis. Ann Surg. 2018;268(6):955–67. https://doi.org/10.1097/SLA.0000000000002761.
Martens MH, Maas M, Heijnen LA, et al. Long-term outcome of an organ preservation program after neoadjuvant treatment for rectal cancer. J Natl Cancer Inst. 2016;108(12):1–10. https://doi.org/10.1093/jnci/djw171.
Lecun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521(7553):436–44. https://doi.org/10.1038/nature14539.
Wu L, Zhou W, Wan X, et al. A deep neural network improves endoscopic detection of early gastric cancer without blind spots. Endoscopy. 2019;51(6):522–31. https://doi.org/10.1055/a-0855-3532.
Jin EH, Lee D, Bae JH, et al. Improved accuracy in optical diagnosis of colorectal polyps using convolutional neural networks with visual explanations. Gastroenterology. 2020;158(8):2169-79.e8. https://doi.org/10.1053/j.gastro.2020.02.036.
Byrne MF, Chapados N, Soudan F, et al. Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model. Gut. 2019;68(1):94–100. https://doi.org/10.1136/gutjnl-2017-314547.
Dong Z, Wang J, Li Y, et al. Explainable artificial intelligence incorporated with domain knowledge diagnosing early gastric neoplasms under white light endoscopy. NPJ Digit Med. 2023. https://doi.org/10.1038/s41746-023-00813-y.
Thompson HM, Kim JK, Jimenez-Rodriguez RM, Garcia-Aguilar J, Veeraraghavan H, Thompson H. Deep learning-based model for identifying tumor in endoscopic images from patients with locally advanced rectal cancer treated with total neoadjuvant therapy. Dis Colon Rectum. 2023. https://doi.org/10.1097/DCR.0000000000002295.
Haak HE, Gao X, Maas M, et al. The use of deep learning on endoscopic images to assess the response of rectal cancer after chemoradiation. Surg Endosc. 2022;36(5):3592–600. https://doi.org/10.1007/s00464-021-08685-7.
Felder SI, Patil S, Kennedy E, Garcia-Aguilar J. Endoscopic feature and response reproducibility in tumor assessment after neoadjuvant therapy for rectal adenocarcinoma. Ann Surg Oncol. 2021;2:8. https://doi.org/10.1245/s10434.
He K, Zhang X, Ren S, Sun J. Deep. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR); 2016.
Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L. ImageNet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. 2009:248–55.
Yang CB, Kim SH, Lim YJ. Preparation of image databases for artificial intelligence algorithm development in gastrointestinal endoscopy. Clin Endosc. 2022;55(5):594–604. https://doi.org/10.5946/ce.2021.229.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74.
Chino A, Konishi T, Ogura A, Kawachi H, Osumi H. Endoscopic criteria to evaluate tumor response of rectal cancer to neoadjuvant chemoradiotherapy using magnifying chromoendoscopy. Eur J Surg Oncol. 2018;44(8):1247–53.
Stijns RCH, Leijtens J, de Graaf E, et al. Endoscopy and MRI for restaging early rectal cancer after neoadjuvant treatment. Colorectal Dis. 2022. https://doi.org/10.1111/codi.16341.
Min JK, Kwak MS, Cha JM. Overview of deep learning in gastrointestinal endoscopy. Gut Liver. 2019;13(4):388–93. https://doi.org/10.5009/gnl18384.
Mohan BP, Khan SR, Kassab LL, Ponnada S, Dulai PS, Kochhar GS. Accuracy of convolutional neural network-based artificial intelligence in diagnosis of gastrointestinal lesions based on endoscopic images: a systematic review and meta-analysis. Endosc Int Open. 2020;08(11):E1584–94. https://doi.org/10.1055/a-1236-3007.
Acknowledgments
The authors acknowledge the Memorial Sloan Kettering Colorectal Oncology Surgery fellows for their assistance with this project. This study was funded in part by grant P30 CA008748 from the National Cancer Institute.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Disclosures
JGA receives honorariums from Johnson & Johnson, Medtronic and Intuitive Surgical. Owns stock in Intuitive Surgical. JJS received travel support for fellow education from Intuitive Surgical (August 19/20 2015); served as a clinical advisor for Guardant Health (March 19/20 2019); served as a clinical advisor for Foundation Medicine (5 April 2022); served as a consultant and speaker for Johnson and Johnson (8-10 May 2022); serves as a clinical advisor and consultant for GSK (2023). HV receives funding from Elekta, Inc. for research unrelated to this manuscript.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Williams, H., Thompson, H.M., Lee, C. et al. Assessing Endoscopic Response in Locally Advanced Rectal Cancer Treated with Total Neoadjuvant Therapy: Development and Validation of a Highly Accurate Convolutional Neural Network. Ann Surg Oncol (2024). https://doi.org/10.1245/s10434-024-15311-y
Received:
Accepted:
Published:
DOI: https://doi.org/10.1245/s10434-024-15311-y