Abstract
Purpose
To evaluate an MRI-based radiomic texture classifier alone and combined with radiologist qualitative assessment in predicting pathological complete response (pCR) using restaging MRI with internal training and external validation.
Methods
Consecutive patients with locally advanced rectal cancer (LARC) who underwent neoadjuvant therapy followed by total mesorectal excision from March 2012 to February 2016 (Memorial Sloan Kettering Cancer Center/internal dataset, n = 114, 41% female, median age = 55) and July 2014 to October 2015 (Instituto do Câncer do Estado de São Paulo/external dataset, n = 50, 52% female, median age = 64.5) were retrospectively included. Two radiologists (R1, senior; R2, junior) independently evaluated restaging MRI, classifying patients (radiological complete response vs radiological partial response). Model A (n = 33 texture features), model B (n = 91 features including texture, shape, and edge features), and two combination models (model A + B + R1, model A + B + R2) were constructed. Pathology served as the reference standard for neoadjuvant treatment response. Comparison of the classifiers’ AUCs on the external set was done using DeLong’s test.
Results
Models A and B had similar discriminative ability (P = 0.3; Model B AUC = 83%, 95% CI 70%–97%). Combined models increased inter-reader agreement compared with radiologist-only interpretation (κ = 0.82, 95% CI 0.70–0.89 vs k = 0.25, 95% CI 0.11–0.61). The combined model slightly increased junior radiologist specificity, positive predictive value, and negative predictive values (93% vs 90%, 57% vs 50%, and 91% vs 90%, respectively).
Conclusion
We developed and externally validated a combined model using radiomics and radiologist qualitative assessment, which improved inter-reader agreement and slightly increased the diagnostic performance of the junior radiologist in predicting pCR after neoadjuvant treatment in patients with LARC.
Graphical abstract
Similar content being viewed by others
Data/code availability
The datasets used and analyzed in this study are not publicly available due to patient privacy requirements but are available upon reasonable request from the corresponding author. All R code used in the analysis will be made available through the author’s GitHub repository.
References
Habr-Gama A, Perez RO, Nadalin W, Sabbaga J, Ribeiro U, Jr., Silva e Sousa AH, Jr., Campos FG, Kiss DR, Gama-Rodrigues J. Operative versus nonoperative treatment for stage 0 distal rectal cancer following chemoradiation therapy: long-term results. Ann Surg 2004;240(4):711–717; discussion 717–718. https://doi.org/10.1097/01.sla.0000141194.27992.32
Patel UB, Brown G, Rutten H, West N, Sebag-Montefiore D, Glynne-Jones R, Rullier E, Peeters M, Van Cutsem E, Ricci S, Van de Velde C, Kjell P, Quirke P. Comparison of magnetic resonance imaging and histopathological response to chemoradiotherapy in locally advanced rectal cancer. Ann Surg Oncol 2012;19(9):2842-2852. https://doi.org/10.1245/s10434-012-2309-3
Sclafani F, Brown G, Cunningham D, Wotherspoon A, Mendes LST, Balyasnikova S, Evans J, Peckitt C, Begum R, Tait D, Tabernero J, Glimelius B, Rosello S, Thomas J, Oates J, Chau I. Comparison between MRI and pathology in the assessment of tumour regression grade in rectal cancer. Br J Cancer 2017;117(10):1478-1485. https://doi.org/10.1038/bjc.2017.320
Siddiqui MR, Gormly KL, Bhoday J, Balyansikova S, Battersby NJ, Chand M, Rao S, Tekkis P, Abulafi AM, Brown G. Interobserver agreement of radiologists assessing the response of rectal cancers to preoperative chemoradiation using the MRI tumour regression grading (mrTRG). Clin Radiol 2016;71(9):854-862. https://doi.org/10.1016/j.crad.2016.05.005
Nahas SC, Rizkallah Nahas CS, Sparapan Marques CF, Ribeiro U, Jr., Cotti GC, Imperiale AR, Capareli FC, Chih Chen AT, Hoff PM, Cecconello I. Pathologic Complete Response in Rectal Cancer: Can We Detect It? Lessons Learned From a Proposed Randomized Trial of Watch-and-Wait Treatment of Rectal Cancer. Dis Colon Rectum 2016;59(4):255-263. https://doi.org/10.1097/DCR.0000000000000558
De Cecco CN, Ganeshan B, Ciolina M, Rengo M, Meinel FG, Musio D, De Felice F, Raffetto N, Tombolini V, Laghi A. Texture analysis as imaging biomarker of tumoral response to neoadjuvant chemoradiotherapy in rectal cancer patients studied with 3-T magnetic resonance. Invest Radiol 2015;50(4):239-245. https://doi.org/10.1097/RLI.0000000000000116
De Cecco CN, Ciolina M, Caruso D, Rengo M, Ganeshan B, Meinel FG, Musio D, De Felice F, Tombolini V, Laghi A. Performance of diffusion-weighted imaging, perfusion imaging, and texture analysis in predicting tumoral response to neoadjuvant chemoradiotherapy in rectal cancer patients studied with 3T MR: initial experience. Abdom Radiol (NY) 2016;41(9):1728-1735. https://doi.org/10.1007/s00261-016-0733-8
Meng Y, Zhang C, Zou S, Zhao X, Xu K, Zhang H, Zhou C. MRI texture analysis in predicting treatment response to neoadjuvant chemoradiotherapy in rectal cancer. Oncotarget 2018;9(15):11999-12008. https://doi.org/10.18632/oncotarget.23813
Aker M, Ganeshan B, Afaq A, Wan S, Groves AM, Arulampalam T. Magnetic Resonance Texture Analysis in Identifying Complete Pathological Response to Neoadjuvant Treatment in Locally Advanced Rectal Cancer. Dis Colon Rectum 2019;62(2):163-170. https://doi.org/10.1097/DCR.0000000000001224
Horvat N, Veeraraghavan H, Khan M, Blazic I, Zheng J, Capanu M, Sala E, Garcia-Aguilar J, Gollub MJ, Petkovska I. MR Imaging of Rectal Cancer: Radiomics Analysis to Assess Treatment Response after Neoadjuvant Therapy. Radiology 2018;287(3):833-843. https://doi.org/10.1148/radiol.2018172300
Cusumano D, Dinapoli N, Boldrini L, Chiloiro G, Gatta R, Masciocchi C, Lenkowicz J, Casa C, Damiani A, Azario L, Van Soest J, Dekker A, Lambin P, De Spirito M, Valentini V. Fractal-based radiomic approach to predict complete pathological response after chemo-radiotherapy in rectal cancer. Radiol Med 2018;123(4):286-295. https://doi.org/10.1007/s11547-017-0838-3
Cui Y, Yang X, Shi Z, Yang Z, Du X, Zhao Z, Cheng X. Radiomics analysis of multiparametric MRI for prediction of pathological complete response to neoadjuvant chemoradiotherapy in locally advanced rectal cancer. Eur Radiol 2019;29(3):1211-1220. https://doi.org/10.1007/s00330-018-5683-9
Bibault JE, Giraud P, Housset M, Durdux C, Taieb J, Berger A, Coriat R, Chaussade S, Dousset B, Nordlinger B, Burgun A. Deep Learning and Radiomics predict complete response after neo-adjuvant chemoradiation for locally advanced rectal cancer. Sci Rep 2018;8(1):12611. https://doi.org/10.1038/s41598-018-30657-6
Liu Z, Zhang XY, Shi YJ, Wang L, Zhu HT, Tang Z, Wang S, Li XT, Tian J, Sun YS. Radiomics Analysis for Evaluation of Pathological Complete Response to Neoadjuvant Chemoradiotherapy in Locally Advanced Rectal Cancer. Clin Cancer Res 2017;23(23):7253-7262. https://doi.org/10.1158/1078-0432.CCR-17-1038
Nie K, Shi L, Chen Q, Hu X, Jabbour SK, Yue N, Niu T, Sun X. Rectal Cancer: Assessment of Neoadjuvant Chemoradiation Outcome based on Radiomics of Multiparametric MRI. Clin Cancer Res 2016;22(21):5256-5264. https://doi.org/10.1158/1078-0432.CCR-15-2997
Petkovska I, Tixier F, Ortiz EJ, Golia Pernicka JS, Paroder V, Bates DD, Horvat N, Fuqua J, Schilsky J, Gollub MJ, Garcia-Aguilar J, Veeraraghavan H. Clinical utility of radiomics at baseline rectal MRI to predict complete response of rectal cancer after chemoradiation therapy. Abdom Radiol (NY) 2020;45(11):3608-3617. https://doi.org/10.1007/s00261-020-02502-w
Di Re AM, Sun Y, Sundaresan P, Hau E, Toh JWT, Gee H, Or M, Haworth A. MRI radiomics in the prediction of therapeutic response to neoadjuvant therapy for locoregionally advanced rectal cancer: a systematic review. Expert Rev Anticancer Ther 2021:1-25. https://doi.org/10.1080/14737140.2021.1860762
Um H, Tixier F, Bermudez D, Deasy JO, Young RJ, Veeraraghavan H. Impact of image preprocessing on the scanner dependence of multi-parametric MRI radiomic features and covariate shift in multi-institutional glioblastoma datasets. Phys Med Biol 2019;64(16):165011. https://doi.org/10.1088/1361-6560/ab2f44
Bulens P, Couwenberg A, Intven M, Debucquoy A, Vandecaveye V, Van Cutsem E, D'Hoore A, Wolthuis A, Mukherjee P, Gevaert O, Haustermans K. Predicting the tumor response to chemoradiotherapy for rectal cancer: Model development and external validation using MRI radiomics. Radiother Oncol 2020;142:246-252. https://doi.org/10.1016/j.radonc.2019.07.033
Dinapoli N, Barbaro B, Gatta R, Chiloiro G, Casa C, Masciocchi C, Damiani A, Boldrini L, Gambacorta MA, Dezio M, Mattiucci GC, Balducci M, van Soest J, Dekker A, Lambin P, Fiorino C, Sini C, De Cobelli F, Di Muzio N, Gumina C, Passoni P, Manfredi R, Valentini V. Magnetic Resonance, Vendor-independent, Intensity Histogram Analysis Predicting Pathologic Complete Response After Radiochemotherapy of Rectal Cancer. Int J Radiat Oncol Biol Phys 2018;102(4):765-774. https://doi.org/10.1016/j.ijrobp.2018.04.065
van Griethuysen JJM, Lambregts DMJ, Trebeschi S, Lahaye MJ, Bakers FCH, Vliegen RFA, Beets GL, Aerts H, Beets-Tan RGH. Radiomics performs comparable to morphologic assessment by expert radiologists for prediction of response to neoadjuvant chemoradiotherapy on baseline staging MRI in rectal cancer. Abdom Radiol (NY) 2020;45(3):632-643. https://doi.org/10.1007/s00261-019-02321-8
Nyul LG, Udupa JK, Zhang X. New variants of a method of MRI scale standardization. IEEE Trans Med Imaging 2000;19(2):143-150. https://doi.org/10.1109/42.836373
Robitaille N, Mouiha A, Crepeault B, Valdivia F, Duchesne S, The Alzheimer's Disease Neuroimaging I. Tissue-based MRI intensity standardization: application to multicentric datasets. Int J Biomed Imaging 2012;2012:347120. https://doi.org/10.1155/2012/347120
Yoo TS, Ackerman MJ, Lorensen WE, Schroeder W, Chalana V, Aylward S, Metaxas D, Whitaker R. Engineering and algorithm design for an image processing Api: a technical report on ITK--the Insight Toolkit. Stud Health Technol Inform 2002;85:586-592.
Zwanenburg A, Vallieres M, Abdalah MA, Aerts H, Andrearczyk V, Apte A, Ashrafinia S, Bakas S, Beukinga RJ, Boellaard R, Bogowicz M, Boldrini L, Buvat I, Cook GJR, Davatzikos C, Depeursinge A, Desseroit MC, Dinapoli N, Dinh CV, Echegaray S, El Naqa I, Fedorov AY, Gatta R, Gillies RJ, Goh V, Gotz M, Guckenberger M, Ha SM, Hatt M, Isensee F, Lambin P, Leger S, Leijenaar RTH, Lenkowicz J, Lippert F, Losnegard A, Maier-Hein KH, Morin O, Muller H, Napel S, Nioche C, Orlhac F, Pati S, Pfaehler EAG, Rahmim A, Rao AUK, Scherer J, Siddique MM, Sijtsema NM, Socarras Fernandez J, Spezi E, Steenbakkers R, Tanadini-Lang S, Thorwarth D, Troost EGC, Upadhaya T, Valentini V, van Dijk LV, van Griethuysen J, van Velden FHP, Whybra P, Richter C, Lock S. The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping. Radiology 2020;295(2):328-338. https://doi.org/10.1148/radiol.2020191145
Breiman L. Random forests. Mach Learn 2001;45(1):5-32. https://doi.org/10.1023/A:1010933404324
Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. SMOTE: Synthetic minority over-sampling technique. J Artif Intell Res 2002;16:321-357. DOI https://doi.org/10.1613/jair.953
Gitto S, Cuocolo R, Annovazzi A, Anelli V, Acquasanta M, Cincotta A, Albano D, Chianca V, Ferraresi V, Messina C, Zoccali C, Armiraglio E, Parafioriti A, Sciuto R, Luzzati A, Biagini R, Imbriaco M, Sconfienza LM. CT radiomics-based machine learning classification of atypical cartilaginous tumours and appendicular chondrosarcomas. EBioMedicine 2021;68:103407. https://doi.org/10.1016/j.ebiom.2021.103407
Hu J, Zhao Y, Li M, Liu J, Wang F, Weng Q, Wang X, Cao D. Machine learning-based radiomics analysis in predicting the meningioma grade using multiparametric MRI. Eur J Radiol 2020;131:109251. https://doi.org/10.1016/j.ejrad.2020.109251
Min X, Li M, Dong D, Feng Z, Zhang P, Ke Z, You H, Han F, Ma H, Tian J, Wang L. Multi-parametric MRI-based radiomics signature for discriminating between clinically significant and insignificant prostate cancer: Cross-validation of a machine learning method. Eur J Radiol 2019;115:16-21. https://doi.org/10.1016/j.ejrad.2019.03.010
Fehr D, Veeraraghavan H, Wibmer A, Gondo T, Matsumoto K, Vargas HA, Sala E, Hricak H, Deasy JO. Automatic classification of prostate cancer Gleason scores from multiparametric magnetic resonance images. Proc Natl Acad Sci U S A 2015;112(46):E6265-6273. https://doi.org/10.1073/pnas.1505935112
Kittler J, Hatef M, Duin RPW, Matas J. On combining classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence 1998;20(3):226-239. https://doi.org/10.1109/34.667881
Lin LI. A concordance correlation coefficient to evaluate reproducibility. Biometrics 1989;45(1):255-268.
Kalpathy-Cramer J, Mamomov A, Zhao B, Lu L, Cherezov D, Napel S, Echegaray S, Rubin D, McNitt-Gray M, Lo P, Sieren JC, Uthoff J, Dilger SK, Driscoll B, Yeung I, Hadjiiski L, Cha K, Balagurunathan Y, Gillies R, Goldgof D. Radiomics of Lung Nodules: A Multi-Institutional Study of Robustness and Agreement of Quantitative Imaging Features. Tomography 2016;2(4):430-437. https://doi.org/10.18383/j.tom.2016.00235
Zhao B, Tan Y, Tsai WY, Qi J, Xie C, Lu L, Schwartz LH. Reproducibility of radiomics for deciphering tumor phenotype with imaging. Sci Rep 2016;6:23428. https://doi.org/10.1038/srep23428
Lakhman Y, Veeraraghavan H, Chaim J, Feier D, Goldman DA, Moskowitz CS, Nougaret S, Sosa RE, Vargas HA, Soslow RA, Abu-Rustum NR, Hricak H, Sala E. Differentiation of Uterine Leiomyosarcoma from Atypical Leiomyoma: Diagnostic Accuracy of Qualitative MR Imaging Features and Feasibility of Texture Analysis. Eur Radiol 2017;27(7):2903-2915. https://doi.org/10.1007/s00330-016-4623-9
Tixier F, Um H, Bermudez D, Iyer A, Apte A, Graham MS, Nevel KS, Deasy JO, Young RJ, Veeraraghavan H. Preoperative MRI-radiomics features improve prediction of survival in glioblastoma patients over MGMT methylation status alone. Oncotarget 2019;10(6):660-672. https://doi.org/10.18632/oncotarget.26578
Bhatia A, Birger M, Veeraraghavan H, Um H, Tixier F, McKenney AS, Cugliari M, Caviasco A, Bialczak A, Malani R, Flynn J, Zhang Z, Yang TJ, Santomasso BD, Shoushtari AN, Young RJ. MRI radiomic features are associated with survival in melanoma brain metastases treated with immune checkpoint inhibitors. Neuro Oncol 2019;21(12):1578-1586. https://doi.org/10.1093/neuonc/noz141
Horvat N, Bates DDB, Petkovska I. Novel imaging techniques of rectal cancer: what do radiomics and radiogenomics have to offer? A literature review. Abdom Radiol (NY) 2019;44(11):3764-3774. https://doi.org/10.1007/s00261-019-02042-y
Maas M, Lambregts DM, Nelemans PJ, Heijnen LA, Martens MH, Leijtens JW, Sosef M, Hulsewe KW, Hoff C, Breukink SO, Stassen L, Beets-Tan RG, Beets GL. Assessment of Clinical Complete Response After Chemoradiation for Rectal Cancer with Digital Rectal Examination, Endoscopy, and MRI: Selection for Organ-Saving Treatment. Ann Surg Oncol 2015;22(12):3873-3880. https://doi.org/10.1245/s10434-015-4687-9
Greenbaum A, Martin DR, Bocklage T, Lee JH, Ness SA, Rajput A. Tumor Heterogeneity as a Predictor of Response to Neoadjuvant Chemotherapy in Locally Advanced Rectal Cancer. Clin Colorectal Cancer 2019;18(2):102-109. https://doi.org/10.1016/j.clcc.2019.02.003
Molinari C, Marisi G, Passardi A, Matteucci L, De Maio G, Ulivi P. Heterogeneity in Colorectal Cancer: A Challenge for Personalized Medicine? Int J Mol Sci 2018;19(12). https://doi.org/10.3390/ijms19123733
Horvat N, Carlos Tavares Rocha C, Clemente Oliveira B, Petkovska I, Gollub MJ. MRI of Rectal Cancer: Tumor Staging, Imaging Techniques, and Management. Radiographics 2019;39(2):367-387. https://doi.org/10.1148/rg.2019180114
Acknowledgements
The authors would like to express their deepest gratitude to Joanne Chin, MFA, ELS, for her editorial support on this manuscript and to Natalie Gangai, MPH, Ye Choi, BS, and Lee Rodriguez, MPH, for their data support.
Funding
This study was funded in part through the NIH/NCI Cancer Center Support Grant P30 CA008748.
Author information
Authors and Affiliations
Contributions
NH contributed to conceptualization; data curation; formal analysis; investigation; methodology; project administration; resources; software; supervision; validation; visualization; roles/writing of the original draft; and writing, reviewing, & editing of the manuscript. HV contributed to conceptualization; data curation; formal analysis; investigation; methodology; project administration; resources; software; supervision; validation; visualization; roles/writing of the original draft; and writing, reviewing, & editing of the manuscript. CSRN contributed to data curation and writing, reviewing, & editing of the manuscript. DDBB contributed to formal analysis and writing, reviewing, & editing of the manuscript. FRF contributed to data curation; formal analysis; and writing, reviewing, & editing of the manuscript. JLFIII contributed to formal analysis and writing, reviewing, & editing of the manuscript. MCF contributed to formal analysis and writing, reviewing, & editing of the manuscript. RES contributed to data curation and writing, reviewing, & editing of the manuscript. VSJ contributed to formal analysis and writing, reviewing, & editing of the manuscript. GGC contributed to project administration; supervision; and writing, reviewing, & editing of the manuscript. SCN contributed to data curation; project administration; supervision; and writing, reviewing, & editing of the manuscript. IP contributed to conceptualization; data curation; formal analysis; investigation; methodology; project administration; resources; software; supervision; validation; visualization; roles/writing of the original draft; and writing, reviewing, & editing of the manuscript.
Corresponding author
Ethics declarations
Conflict of interest
The authors have no competing interests to declare that are relevant to the content of this article.
Ethics approval
This study was performed in line with the principles of the Declaration of Helsinki. Approval was granted by the Institutional Review Board at Memorial Sloan Kettering Cancer Center, USA and at the University of Sao Paulo, Brazil.
Consent
Written informed consent was waived by the Institutional Review Board at Memorial Sloan Kettering Cancer Center, USA and at the University of Sao Paulo, Brazil.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Horvat, N., Veeraraghavan, H., Nahas, C.S.R. et al. Combined artificial intelligence and radiologist model for predicting rectal cancer treatment response from magnetic resonance imaging: an external validation study. Abdom Radiol 47, 2770–2782 (2022). https://doi.org/10.1007/s00261-022-03572-8
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00261-022-03572-8