Skip to main content

Advertisement

Log in

Combined artificial intelligence and radiologist model for predicting rectal cancer treatment response from magnetic resonance imaging: an external validation study

  • Hollow Organ GI
  • Published:
Abdominal Radiology Aims and scope Submit manuscript

Abstract

Purpose

To evaluate an MRI-based radiomic texture classifier alone and combined with radiologist qualitative assessment in predicting pathological complete response (pCR) using restaging MRI with internal training and external validation.

Methods

Consecutive patients with locally advanced rectal cancer (LARC) who underwent neoadjuvant therapy followed by total mesorectal excision from March 2012 to February 2016 (Memorial Sloan Kettering Cancer Center/internal dataset, n = 114, 41% female, median age = 55) and July 2014 to October 2015 (Instituto do Câncer do Estado de São Paulo/external dataset, n = 50, 52% female, median age = 64.5) were retrospectively included. Two radiologists (R1, senior; R2, junior) independently evaluated restaging MRI, classifying patients (radiological complete response vs radiological partial response). Model A (n = 33 texture features), model B (n = 91 features including texture, shape, and edge features), and two combination models (model A + B + R1, model A + B + R2) were constructed. Pathology served as the reference standard for neoadjuvant treatment response. Comparison of the classifiers’ AUCs on the external set was done using DeLong’s test.

Results

Models A and B had similar discriminative ability (P = 0.3; Model B AUC = 83%, 95% CI 70%–97%). Combined models increased inter-reader agreement compared with radiologist-only interpretation (κ = 0.82, 95% CI 0.70–0.89 vs k = 0.25, 95% CI 0.11–0.61). The combined model slightly increased junior radiologist specificity, positive predictive value, and negative predictive values (93% vs 90%, 57% vs 50%, and 91% vs 90%, respectively).

Conclusion

We developed and externally validated a combined model using radiomics and radiologist qualitative assessment, which improved inter-reader agreement and slightly increased the diagnostic performance of the junior radiologist in predicting pCR after neoadjuvant treatment in patients with LARC.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data/code availability

The datasets used and analyzed in this study are not publicly available due to patient privacy requirements but are available upon reasonable request from the corresponding author. All R code used in the analysis will be made available through the author’s GitHub repository.

References

  1. Habr-Gama A, Perez RO, Nadalin W, Sabbaga J, Ribeiro U, Jr., Silva e Sousa AH, Jr., Campos FG, Kiss DR, Gama-Rodrigues J. Operative versus nonoperative treatment for stage 0 distal rectal cancer following chemoradiation therapy: long-term results. Ann Surg 2004;240(4):711–717; discussion 717–718. https://doi.org/10.1097/01.sla.0000141194.27992.32

  2. Patel UB, Brown G, Rutten H, West N, Sebag-Montefiore D, Glynne-Jones R, Rullier E, Peeters M, Van Cutsem E, Ricci S, Van de Velde C, Kjell P, Quirke P. Comparison of magnetic resonance imaging and histopathological response to chemoradiotherapy in locally advanced rectal cancer. Ann Surg Oncol 2012;19(9):2842-2852. https://doi.org/10.1245/s10434-012-2309-3

    Article  PubMed  Google Scholar 

  3. Sclafani F, Brown G, Cunningham D, Wotherspoon A, Mendes LST, Balyasnikova S, Evans J, Peckitt C, Begum R, Tait D, Tabernero J, Glimelius B, Rosello S, Thomas J, Oates J, Chau I. Comparison between MRI and pathology in the assessment of tumour regression grade in rectal cancer. Br J Cancer 2017;117(10):1478-1485. https://doi.org/10.1038/bjc.2017.320

    Article  PubMed  PubMed Central  Google Scholar 

  4. Siddiqui MR, Gormly KL, Bhoday J, Balyansikova S, Battersby NJ, Chand M, Rao S, Tekkis P, Abulafi AM, Brown G. Interobserver agreement of radiologists assessing the response of rectal cancers to preoperative chemoradiation using the MRI tumour regression grading (mrTRG). Clin Radiol 2016;71(9):854-862. https://doi.org/10.1016/j.crad.2016.05.005

    Article  CAS  PubMed  Google Scholar 

  5. Nahas SC, Rizkallah Nahas CS, Sparapan Marques CF, Ribeiro U, Jr., Cotti GC, Imperiale AR, Capareli FC, Chih Chen AT, Hoff PM, Cecconello I. Pathologic Complete Response in Rectal Cancer: Can We Detect It? Lessons Learned From a Proposed Randomized Trial of Watch-and-Wait Treatment of Rectal Cancer. Dis Colon Rectum 2016;59(4):255-263. https://doi.org/10.1097/DCR.0000000000000558

    Article  PubMed  Google Scholar 

  6. De Cecco CN, Ganeshan B, Ciolina M, Rengo M, Meinel FG, Musio D, De Felice F, Raffetto N, Tombolini V, Laghi A. Texture analysis as imaging biomarker of tumoral response to neoadjuvant chemoradiotherapy in rectal cancer patients studied with 3-T magnetic resonance. Invest Radiol 2015;50(4):239-245. https://doi.org/10.1097/RLI.0000000000000116

    Article  CAS  PubMed  Google Scholar 

  7. De Cecco CN, Ciolina M, Caruso D, Rengo M, Ganeshan B, Meinel FG, Musio D, De Felice F, Tombolini V, Laghi A. Performance of diffusion-weighted imaging, perfusion imaging, and texture analysis in predicting tumoral response to neoadjuvant chemoradiotherapy in rectal cancer patients studied with 3T MR: initial experience. Abdom Radiol (NY) 2016;41(9):1728-1735. https://doi.org/10.1007/s00261-016-0733-8

    Article  PubMed  Google Scholar 

  8. Meng Y, Zhang C, Zou S, Zhao X, Xu K, Zhang H, Zhou C. MRI texture analysis in predicting treatment response to neoadjuvant chemoradiotherapy in rectal cancer. Oncotarget 2018;9(15):11999-12008. https://doi.org/10.18632/oncotarget.23813

    Article  PubMed  Google Scholar 

  9. Aker M, Ganeshan B, Afaq A, Wan S, Groves AM, Arulampalam T. Magnetic Resonance Texture Analysis in Identifying Complete Pathological Response to Neoadjuvant Treatment in Locally Advanced Rectal Cancer. Dis Colon Rectum 2019;62(2):163-170. https://doi.org/10.1097/DCR.0000000000001224

    Article  PubMed  Google Scholar 

  10. Horvat N, Veeraraghavan H, Khan M, Blazic I, Zheng J, Capanu M, Sala E, Garcia-Aguilar J, Gollub MJ, Petkovska I. MR Imaging of Rectal Cancer: Radiomics Analysis to Assess Treatment Response after Neoadjuvant Therapy. Radiology 2018;287(3):833-843. https://doi.org/10.1148/radiol.2018172300

    Article  PubMed  Google Scholar 

  11. Cusumano D, Dinapoli N, Boldrini L, Chiloiro G, Gatta R, Masciocchi C, Lenkowicz J, Casa C, Damiani A, Azario L, Van Soest J, Dekker A, Lambin P, De Spirito M, Valentini V. Fractal-based radiomic approach to predict complete pathological response after chemo-radiotherapy in rectal cancer. Radiol Med 2018;123(4):286-295. https://doi.org/10.1007/s11547-017-0838-3

    Article  PubMed  Google Scholar 

  12. Cui Y, Yang X, Shi Z, Yang Z, Du X, Zhao Z, Cheng X. Radiomics analysis of multiparametric MRI for prediction of pathological complete response to neoadjuvant chemoradiotherapy in locally advanced rectal cancer. Eur Radiol 2019;29(3):1211-1220. https://doi.org/10.1007/s00330-018-5683-9

    Article  PubMed  Google Scholar 

  13. Bibault JE, Giraud P, Housset M, Durdux C, Taieb J, Berger A, Coriat R, Chaussade S, Dousset B, Nordlinger B, Burgun A. Deep Learning and Radiomics predict complete response after neo-adjuvant chemoradiation for locally advanced rectal cancer. Sci Rep 2018;8(1):12611. https://doi.org/10.1038/s41598-018-30657-6

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Liu Z, Zhang XY, Shi YJ, Wang L, Zhu HT, Tang Z, Wang S, Li XT, Tian J, Sun YS. Radiomics Analysis for Evaluation of Pathological Complete Response to Neoadjuvant Chemoradiotherapy in Locally Advanced Rectal Cancer. Clin Cancer Res 2017;23(23):7253-7262. https://doi.org/10.1158/1078-0432.CCR-17-1038

    Article  CAS  PubMed  Google Scholar 

  15. Nie K, Shi L, Chen Q, Hu X, Jabbour SK, Yue N, Niu T, Sun X. Rectal Cancer: Assessment of Neoadjuvant Chemoradiation Outcome based on Radiomics of Multiparametric MRI. Clin Cancer Res 2016;22(21):5256-5264. https://doi.org/10.1158/1078-0432.CCR-15-2997

    Article  PubMed  Google Scholar 

  16. Petkovska I, Tixier F, Ortiz EJ, Golia Pernicka JS, Paroder V, Bates DD, Horvat N, Fuqua J, Schilsky J, Gollub MJ, Garcia-Aguilar J, Veeraraghavan H. Clinical utility of radiomics at baseline rectal MRI to predict complete response of rectal cancer after chemoradiation therapy. Abdom Radiol (NY) 2020;45(11):3608-3617. https://doi.org/10.1007/s00261-020-02502-w

    Article  Google Scholar 

  17. Di Re AM, Sun Y, Sundaresan P, Hau E, Toh JWT, Gee H, Or M, Haworth A. MRI radiomics in the prediction of therapeutic response to neoadjuvant therapy for locoregionally advanced rectal cancer: a systematic review. Expert Rev Anticancer Ther 2021:1-25. https://doi.org/10.1080/14737140.2021.1860762

    Article  PubMed  Google Scholar 

  18. Um H, Tixier F, Bermudez D, Deasy JO, Young RJ, Veeraraghavan H. Impact of image preprocessing on the scanner dependence of multi-parametric MRI radiomic features and covariate shift in multi-institutional glioblastoma datasets. Phys Med Biol 2019;64(16):165011. https://doi.org/10.1088/1361-6560/ab2f44

    Article  PubMed  Google Scholar 

  19. Bulens P, Couwenberg A, Intven M, Debucquoy A, Vandecaveye V, Van Cutsem E, D'Hoore A, Wolthuis A, Mukherjee P, Gevaert O, Haustermans K. Predicting the tumor response to chemoradiotherapy for rectal cancer: Model development and external validation using MRI radiomics. Radiother Oncol 2020;142:246-252. https://doi.org/10.1016/j.radonc.2019.07.033

    Article  CAS  PubMed  Google Scholar 

  20. Dinapoli N, Barbaro B, Gatta R, Chiloiro G, Casa C, Masciocchi C, Damiani A, Boldrini L, Gambacorta MA, Dezio M, Mattiucci GC, Balducci M, van Soest J, Dekker A, Lambin P, Fiorino C, Sini C, De Cobelli F, Di Muzio N, Gumina C, Passoni P, Manfredi R, Valentini V. Magnetic Resonance, Vendor-independent, Intensity Histogram Analysis Predicting Pathologic Complete Response After Radiochemotherapy of Rectal Cancer. Int J Radiat Oncol Biol Phys 2018;102(4):765-774. https://doi.org/10.1016/j.ijrobp.2018.04.065

    Article  PubMed  Google Scholar 

  21. van Griethuysen JJM, Lambregts DMJ, Trebeschi S, Lahaye MJ, Bakers FCH, Vliegen RFA, Beets GL, Aerts H, Beets-Tan RGH. Radiomics performs comparable to morphologic assessment by expert radiologists for prediction of response to neoadjuvant chemoradiotherapy on baseline staging MRI in rectal cancer. Abdom Radiol (NY) 2020;45(3):632-643. https://doi.org/10.1007/s00261-019-02321-8

    Article  Google Scholar 

  22. Nyul LG, Udupa JK, Zhang X. New variants of a method of MRI scale standardization. IEEE Trans Med Imaging 2000;19(2):143-150. https://doi.org/10.1109/42.836373

    Article  CAS  PubMed  Google Scholar 

  23. Robitaille N, Mouiha A, Crepeault B, Valdivia F, Duchesne S, The Alzheimer's Disease Neuroimaging I. Tissue-based MRI intensity standardization: application to multicentric datasets. Int J Biomed Imaging 2012;2012:347120. https://doi.org/10.1155/2012/347120

    Article  PubMed  PubMed Central  Google Scholar 

  24. Yoo TS, Ackerman MJ, Lorensen WE, Schroeder W, Chalana V, Aylward S, Metaxas D, Whitaker R. Engineering and algorithm design for an image processing Api: a technical report on ITK--the Insight Toolkit. Stud Health Technol Inform 2002;85:586-592.

    PubMed  Google Scholar 

  25. Zwanenburg A, Vallieres M, Abdalah MA, Aerts H, Andrearczyk V, Apte A, Ashrafinia S, Bakas S, Beukinga RJ, Boellaard R, Bogowicz M, Boldrini L, Buvat I, Cook GJR, Davatzikos C, Depeursinge A, Desseroit MC, Dinapoli N, Dinh CV, Echegaray S, El Naqa I, Fedorov AY, Gatta R, Gillies RJ, Goh V, Gotz M, Guckenberger M, Ha SM, Hatt M, Isensee F, Lambin P, Leger S, Leijenaar RTH, Lenkowicz J, Lippert F, Losnegard A, Maier-Hein KH, Morin O, Muller H, Napel S, Nioche C, Orlhac F, Pati S, Pfaehler EAG, Rahmim A, Rao AUK, Scherer J, Siddique MM, Sijtsema NM, Socarras Fernandez J, Spezi E, Steenbakkers R, Tanadini-Lang S, Thorwarth D, Troost EGC, Upadhaya T, Valentini V, van Dijk LV, van Griethuysen J, van Velden FHP, Whybra P, Richter C, Lock S. The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping. Radiology 2020;295(2):328-338. https://doi.org/10.1148/radiol.2020191145

    Article  PubMed  Google Scholar 

  26. Breiman L. Random forests. Mach Learn 2001;45(1):5-32. https://doi.org/10.1023/A:1010933404324

    Article  Google Scholar 

  27. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. SMOTE: Synthetic minority over-sampling technique. J Artif Intell Res 2002;16:321-357. DOI https://doi.org/10.1613/jair.953

    Article  Google Scholar 

  28. Gitto S, Cuocolo R, Annovazzi A, Anelli V, Acquasanta M, Cincotta A, Albano D, Chianca V, Ferraresi V, Messina C, Zoccali C, Armiraglio E, Parafioriti A, Sciuto R, Luzzati A, Biagini R, Imbriaco M, Sconfienza LM. CT radiomics-based machine learning classification of atypical cartilaginous tumours and appendicular chondrosarcomas. EBioMedicine 2021;68:103407. https://doi.org/10.1016/j.ebiom.2021.103407

    Article  PubMed  PubMed Central  Google Scholar 

  29. Hu J, Zhao Y, Li M, Liu J, Wang F, Weng Q, Wang X, Cao D. Machine learning-based radiomics analysis in predicting the meningioma grade using multiparametric MRI. Eur J Radiol 2020;131:109251. https://doi.org/10.1016/j.ejrad.2020.109251

    Article  PubMed  Google Scholar 

  30. Min X, Li M, Dong D, Feng Z, Zhang P, Ke Z, You H, Han F, Ma H, Tian J, Wang L. Multi-parametric MRI-based radiomics signature for discriminating between clinically significant and insignificant prostate cancer: Cross-validation of a machine learning method. Eur J Radiol 2019;115:16-21. https://doi.org/10.1016/j.ejrad.2019.03.010

    Article  PubMed  Google Scholar 

  31. Fehr D, Veeraraghavan H, Wibmer A, Gondo T, Matsumoto K, Vargas HA, Sala E, Hricak H, Deasy JO. Automatic classification of prostate cancer Gleason scores from multiparametric magnetic resonance images. Proc Natl Acad Sci U S A 2015;112(46):E6265-6273. https://doi.org/10.1073/pnas.1505935112

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  32. Kittler J, Hatef M, Duin RPW, Matas J. On combining classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence 1998;20(3):226-239. https://doi.org/10.1109/34.667881

    Article  Google Scholar 

  33. Lin LI. A concordance correlation coefficient to evaluate reproducibility. Biometrics 1989;45(1):255-268.

    Article  CAS  Google Scholar 

  34. Kalpathy-Cramer J, Mamomov A, Zhao B, Lu L, Cherezov D, Napel S, Echegaray S, Rubin D, McNitt-Gray M, Lo P, Sieren JC, Uthoff J, Dilger SK, Driscoll B, Yeung I, Hadjiiski L, Cha K, Balagurunathan Y, Gillies R, Goldgof D. Radiomics of Lung Nodules: A Multi-Institutional Study of Robustness and Agreement of Quantitative Imaging Features. Tomography 2016;2(4):430-437. https://doi.org/10.18383/j.tom.2016.00235

    Article  PubMed  PubMed Central  Google Scholar 

  35. Zhao B, Tan Y, Tsai WY, Qi J, Xie C, Lu L, Schwartz LH. Reproducibility of radiomics for deciphering tumor phenotype with imaging. Sci Rep 2016;6:23428. https://doi.org/10.1038/srep23428

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  36. Lakhman Y, Veeraraghavan H, Chaim J, Feier D, Goldman DA, Moskowitz CS, Nougaret S, Sosa RE, Vargas HA, Soslow RA, Abu-Rustum NR, Hricak H, Sala E. Differentiation of Uterine Leiomyosarcoma from Atypical Leiomyoma: Diagnostic Accuracy of Qualitative MR Imaging Features and Feasibility of Texture Analysis. Eur Radiol 2017;27(7):2903-2915. https://doi.org/10.1007/s00330-016-4623-9

    Article  PubMed  Google Scholar 

  37. Tixier F, Um H, Bermudez D, Iyer A, Apte A, Graham MS, Nevel KS, Deasy JO, Young RJ, Veeraraghavan H. Preoperative MRI-radiomics features improve prediction of survival in glioblastoma patients over MGMT methylation status alone. Oncotarget 2019;10(6):660-672. https://doi.org/10.18632/oncotarget.26578

    Article  PubMed  PubMed Central  Google Scholar 

  38. Bhatia A, Birger M, Veeraraghavan H, Um H, Tixier F, McKenney AS, Cugliari M, Caviasco A, Bialczak A, Malani R, Flynn J, Zhang Z, Yang TJ, Santomasso BD, Shoushtari AN, Young RJ. MRI radiomic features are associated with survival in melanoma brain metastases treated with immune checkpoint inhibitors. Neuro Oncol 2019;21(12):1578-1586. https://doi.org/10.1093/neuonc/noz141

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  39. Horvat N, Bates DDB, Petkovska I. Novel imaging techniques of rectal cancer: what do radiomics and radiogenomics have to offer? A literature review. Abdom Radiol (NY) 2019;44(11):3764-3774. https://doi.org/10.1007/s00261-019-02042-y

    Article  Google Scholar 

  40. Maas M, Lambregts DM, Nelemans PJ, Heijnen LA, Martens MH, Leijtens JW, Sosef M, Hulsewe KW, Hoff C, Breukink SO, Stassen L, Beets-Tan RG, Beets GL. Assessment of Clinical Complete Response After Chemoradiation for Rectal Cancer with Digital Rectal Examination, Endoscopy, and MRI: Selection for Organ-Saving Treatment. Ann Surg Oncol 2015;22(12):3873-3880. https://doi.org/10.1245/s10434-015-4687-9

    Article  PubMed  PubMed Central  Google Scholar 

  41. Greenbaum A, Martin DR, Bocklage T, Lee JH, Ness SA, Rajput A. Tumor Heterogeneity as a Predictor of Response to Neoadjuvant Chemotherapy in Locally Advanced Rectal Cancer. Clin Colorectal Cancer 2019;18(2):102-109. https://doi.org/10.1016/j.clcc.2019.02.003

    Article  PubMed  PubMed Central  Google Scholar 

  42. Molinari C, Marisi G, Passardi A, Matteucci L, De Maio G, Ulivi P. Heterogeneity in Colorectal Cancer: A Challenge for Personalized Medicine? Int J Mol Sci 2018;19(12). https://doi.org/10.3390/ijms19123733

    Article  PubMed  PubMed Central  Google Scholar 

  43. Horvat N, Carlos Tavares Rocha C, Clemente Oliveira B, Petkovska I, Gollub MJ. MRI of Rectal Cancer: Tumor Staging, Imaging Techniques, and Management. Radiographics 2019;39(2):367-387. https://doi.org/10.1148/rg.2019180114

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

The authors would like to express their deepest gratitude to Joanne Chin, MFA, ELS, for her editorial support on this manuscript and to Natalie Gangai, MPH, Ye Choi, BS, and Lee Rodriguez, MPH, for their data support.

Funding

This study was funded in part through the NIH/NCI Cancer Center Support Grant P30 CA008748.

Author information

Authors and Affiliations

Authors

Contributions

NH contributed to conceptualization; data curation; formal analysis; investigation; methodology; project administration; resources; software; supervision; validation; visualization; roles/writing of the original draft; and writing, reviewing, & editing of the manuscript. HV contributed to conceptualization; data curation; formal analysis; investigation; methodology; project administration; resources; software; supervision; validation; visualization; roles/writing of the original draft; and writing, reviewing, & editing of the manuscript. CSRN contributed to data curation and writing, reviewing, & editing of the manuscript. DDBB contributed to formal analysis and writing, reviewing, & editing of the manuscript. FRF contributed to data curation; formal analysis; and writing, reviewing, & editing of the manuscript. JLFIII contributed to formal analysis and writing, reviewing, & editing of the manuscript. MCF contributed to formal analysis and writing, reviewing, & editing of the manuscript. RES contributed to data curation and writing, reviewing, & editing of the manuscript. VSJ contributed to formal analysis and writing, reviewing, & editing of the manuscript. GGC contributed to project administration; supervision; and writing, reviewing, & editing of the manuscript. SCN contributed to data curation; project administration; supervision; and writing, reviewing, & editing of the manuscript. IP contributed to conceptualization; data curation; formal analysis; investigation; methodology; project administration; resources; software; supervision; validation; visualization; roles/writing of the original draft; and writing, reviewing, & editing of the manuscript.

Corresponding author

Correspondence to Iva Petkovska.

Ethics declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this article.

Ethics approval

This study was performed in line with the principles of the Declaration of Helsinki. Approval was granted by the Institutional Review Board at Memorial Sloan Kettering Cancer Center, USA and at the University of Sao Paulo, Brazil.

Consent

Written informed consent was waived by the Institutional Review Board at Memorial Sloan Kettering Cancer Center, USA and at the University of Sao Paulo, Brazil.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 1040 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Horvat, N., Veeraraghavan, H., Nahas, C.S.R. et al. Combined artificial intelligence and radiologist model for predicting rectal cancer treatment response from magnetic resonance imaging: an external validation study. Abdom Radiol 47, 2770–2782 (2022). https://doi.org/10.1007/s00261-022-03572-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00261-022-03572-8

Keywords

Navigation