Skip to main content

Deep Treatment Response Assessment and Prediction of Colorectal Cancer Liver Metastases

Part of the Lecture Notes in Computer Science book series (LNCS,volume 13433)


Evaluating treatment response is essential in patients who develop colorectal liver metastases to decide the necessity for second-line treatment or the admissibility for surgery. Currently, RECIST1.1 is the most widely used criteria in this context. However, it involves time-consuming, precise manual delineation and size measurement of main liver metastases from Computed Tomography (CT) images. Moreover, an early prediction of the treatment response given a specific chemotherapy regimen and the initial CT scan would be of tremendous use to clinicians. To overcome these challenges, this paper proposes a deep learning-based treatment response assessment pipeline and its extension for prediction purposes. Based on a newly designed 3D Siamese classification network, our method assigns a response group to patients given CT scans from two consecutive follow-ups during the treatment period. Further, we extended the network to predict the treatment response given only the image acquired at first time point. The pipelines are trained on the PRODIGE20 dataset collected from a phase-II multi-center clinical trial in colorectal cancer with liver metastases and exploit an in-house dataset to integrate metastases delineations derived from a U-Net inspired network as additional information. Our approach achieves overall accuracies of 94.94% and 86.86% for treatment response assessment and early prediction respectively, suggesting that both treatment response assessment and prediction issues can be effectively solved with deep learning.


  • Treatment response
  • Siamese network
  • Colorectal cancer
  • Liver metastases
  • Longitudinal analysis

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions


  1. Kow, A.W.C.: Hepatic metastasis from colorectal cancer. J. Gastrointest. Oncol. 10(6), 1274 (2019)

    CrossRef  MathSciNet  Google Scholar 

  2. van Gestel, Y., et al.: Patterns of metachronous metastases after curative treatment of colorectal cancer. Cancer Epidemiol. 38(4), 448–454 (2014)

    CrossRef  Google Scholar 

  3. Eisenhauer, E.A., Therasse, P., Bogaerts, J., Schwartz, L.H., et al.: New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1). Eur. J. Cancer 45(2), 228–247 (2009)

    CrossRef  Google Scholar 

  4. Kuhl, C.K., et al.: Validity of RECIST version 1.1 for response assessment in metastatic cancer: a prospective, multireader study. Radiology 290(2), 349–356 (2019)

    CrossRef  Google Scholar 

  5. Dohan, A., et al.: Early evaluation using a radiomic signature of unresectable hepatic metastases to predict outcome in patients with colorectal cancer treated with FOLFIRI and bevacizumab. Gut 69(3), 531–539 (2020)

    CrossRef  Google Scholar 

  6. Topol, E.J.: High-performance medicine: the convergence of human and artificial intelligence. Nat. Med. 25(1), 44–56 (2019)

    CrossRef  Google Scholar 

  7. Esteva, A., et al.: Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7639), 115–118 (2017)

    CrossRef  Google Scholar 

  8. Gulshan, V., et al.: Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316(22), 2402–2410 (2016)

    CrossRef  Google Scholar 

  9. Lee, H., et al.: An explainable deep-learning algorithm for the detection of acute intracranial haemorrhage from small datasets. Nat. Biomed. Eng. 3(3), 173–182 (2019)

    CrossRef  Google Scholar 

  10. Maaref, A., et al.: Predicting the response to FOLFOX-based chemotherapy regimen from untreated liver metastases on baseline CT: a deep neural network approach. J. Digit. Imaging 33(4), 937–945 (2020)

    CrossRef  Google Scholar 

  11. Graffy, P.M., Liu, J., Pickhardt, P.J., Burns, J.E., Yao, J., Summers, R.M.: Deep learning-based muscle segmentation and quantification at abdominal CT: application to a longitudinal adult screening cohort for sarcopenia assessment. Br. J. Radiol. 92(1100), 20190327 (2019)

    CrossRef  Google Scholar 

  12. Jin, C., et al.: Predicting treatment response from longitudinal images using multi-task deep learning. Nat. Commun. 12(1), 1–11 (2021)

    CrossRef  Google Scholar 

  13. Cao, Y., et al.: Longitudinal assessment of COVID-19 using a deep learning-based quantitative CT pipeline: illustration of two cases. Radiol.: Cardiothorac. Imaging 2(2) (2020)

    Google Scholar 

  14. Zhu, H.-B., et al.: Deep learning-assisted magnetic resonance imaging prediction of tumor response to chemotherapy in patients with colorectal liver metastases. Int. J. Cancer 148, 1717–1730 (2021)

    CrossRef  Google Scholar 

  15. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015).

    CrossRef  Google Scholar 

  16. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)

    Google Scholar 

  17. Conze, P.-H., Brochard, S., Burdin, V., Sheehan, F.T., Pons, C.: Healthy versus pathological learning transferability in shoulder muscle MRI segmentation using deep convolutional encoder-decoders. Comput. Med. Imaging Graph. 83, 101733 (2020)

    Google Scholar 

  18. Conze, P.-H., et al.: Abdominal multi-organ segmentation with cascaded convolutional and adversarial deep networks. Artif. Intell. Med. 117, 102109 (2021)

    Google Scholar 

  19. Yan, Y., et al.: Longitudinal detection of diabetic retinopathy early severity grade changes using deep learning. In: Fu, H., Garvin, M.K., MacGillivray, T., Xu, Y., Zheng, Y. (eds.) OMIA 2021. LNCS, vol. 12970, pp. 11–20. Springer, Cham (2021).

    CrossRef  Google Scholar 

  20. Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1492–1500 (2017)

    Google Scholar 

  21. Roy, A.G., Navab, N., Wachinger, C.: Concurrent spatial and channel ‘squeeze & excitation’ in fully convolutional networks. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 421–429. Springer, Cham (2018).

    CrossRef  Google Scholar 

  22. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)

    Google Scholar 

  23. Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014)

  24. Aparicio, T., et al.: Bevacizumab+chemotherapy versus chemotherapy alone in elderly patients with untreated metastatic colorectal cancer: a randomized phase II trial-PRODIGE 20 study results. Ann. Oncol. 29(1), 133–138 (2018)

    CrossRef  Google Scholar 

  25. Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13(2), 281–305 (2012)

    MathSciNet  MATH  Google Scholar 

  26. Rücker, G., Schumacher, M.: Summary ROC curve based on a weighted Youden index for selecting an optimal cutpoint in meta-analysis of diagnostic accuracy. Stat. Med. 29(30), 3069–3078 (2010)

    CrossRef  MathSciNet  Google Scholar 

  27. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: IEEE International Conference on Computer Vision, pp. 618–626 (2017)

    Google Scholar 

Download references


This work was partially funded by Ligue contre le Cancer. The PRODIGE20 dataset was provided with the support from FFCD (Fédération Francophone de Cancérologie Digestive). The authors would like to thank all the PRODIGE20 investigators as well as A. Dohan from AP-HP for fruitful discussions.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Pierre-Henri Conze .

Editor information

Editors and Affiliations

Ethics declarations

Compliance with Ethical Standards

CT data acquisition was performed in line with the principles of the Declaration of Helsinki. Ethical approval was provided by the Ethics Committee CPP EST I DIJON n\(^\circ \)100109 in Jan, 26 2010 and registered in with number NCT01900717. Authors declare that they do not have any conflicts of interest.

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 2316 KB)

Rights and permissions

Reprints and Permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Islam, M.M. et al. (2022). Deep Treatment Response Assessment and Prediction of Colorectal Cancer Liver Metastases. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13433. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-16436-1

  • Online ISBN: 978-3-031-16437-8

  • eBook Packages: Computer ScienceComputer Science (R0)