Deep transfer learning methods for colon cancer classification in confocal laser microscopy images

  • Nils GessertEmail author
  • Marcel Bengs
  • Lukas Wittig
  • Daniel Drömann
  • Tobias Keck
  • Alexander Schlaefer
  • David B. Ellebrecht
Original Article



The gold standard for colorectal cancer metastases detection in the peritoneum is histological evaluation of a removed tissue sample. For feedback during interventions, real-time in vivo imaging with confocal laser microscopy has been proposed for differentiation of benign and malignant tissue by manual expert evaluation. Automatic image classification could improve the surgical workflow further by providing immediate feedback.


We analyze the feasibility of classifying tissue from confocal laser microscopy in the colon and peritoneum. For this purpose, we adopt both classical and state-of-the-art convolutional neural networks to directly learn from the images. As the available dataset is small, we investigate several transfer learning strategies including partial freezing variants and full fine-tuning. We address the distinction of different tissue types, as well as benign and malignant tissue.


We present a thorough analysis of transfer learning strategies for colorectal cancer with confocal laser microscopy. In the peritoneum, metastases are classified with an AUC of 97.1, and in the colon the primarius is classified with an AUC of 73.1. In general, transfer learning substantially improves performance over training from scratch. We find that the optimal transfer learning strategy differs for models and classification tasks.


We demonstrate that convolutional neural networks and transfer learning can be used to identify cancer tissue with confocal laser microscopy. We show that there is no generally optimal transfer learning strategy and model as well as task-specific engineering is required. Given the high performance for the peritoneum, even with a small dataset, application for intraoperative decision support could be feasible.


Colon cancer Confocal laser microscopy Transfer learning Convolution neural network 


Compliance with ethical standards

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving animals were in accordance with the ethical standards of the institution or practice at which the studies were conducted.

Informed consent

Informed consent was obtained from all individual participants included in the study.


  1. 1.
    Torre LA, Bray F, Siegel RL, Ferlay J, Lortet-Tieulent J, Jemal A (2015) Global cancer statistics, 2012. CA Cancer J Clin 65(2):87–108PubMedPubMedCentralGoogle Scholar
  2. 2.
    Verwaal VJ, van Ruth S, Witkamp A, Boot H, van Slooten G, Zoetmulder FA (2005) Long-term survival of peritoneal carcinomatosis of colorectal origin. Ann Surg Oncol 12(1):65–71CrossRefGoogle Scholar
  3. 3.
    Franko J, Shi Q, Goldman CD, Pockaj BA, Nelson GD, Goldberg RM, Pitot HC, Grothey A, Alberts SR, Sargent DJ (2012) Treatment of colorectal peritoneal carcinomatosis with systemic chemotherapy: a pooled analysis of north central cancer treatment group phase III trials N9741 and N9841. J Clin Oncol 30(3):263CrossRefGoogle Scholar
  4. 4.
    de Bree E, Koops W, Kröger R, van Ruth S, Witkamp AJ, Zoetmulder FA (2004) Peritoneal carcinomatosis from colorectal or appendiceal origin: correlation of preoperative CT with intraoperative findings and evaluation of interobserver agreement. J Surg Oncol 86(2):64–73CrossRefGoogle Scholar
  5. 5.
    Dromain C, Leboulleux S, Auperin A, Goere D, Malka D, Lumbroso J, Schumberger M, Sigal R, Elias D (2008) Staging of peritoneal carcinomatosis: enhanced CT vs. PET/CT. Abdom Imaging 33(1):87–93CrossRefGoogle Scholar
  6. 6.
    Low RN, Semelka RC, Worawattanakul S, Alzate GD (2000) Extrahepatic abdominal imaging in patients with malignancy: comparison of MR imaging and helical CT in 164 patients. J Magn Reson Imaging 12(2):269–277CrossRefGoogle Scholar
  7. 7.
    Iafrate F, Ciolina M, Sammartino P, Baldassari P, Rengo M, Lucchesi P, Sibio S, Accarpio F, Di Giorgio A, Laghi A (2012) Peritoneal carcinomatosis: imaging with 64-MDCT and 3T MRI with diffusion-weighted imaging. Abdom Imaging 37(4):616–627CrossRefGoogle Scholar
  8. 8.
    González-Moreno S, González-Bayón L, Ortega-Pérez G, González-Hernando C (2009) Imaging of peritoneal carcinomatosis. Cancer J 15(3):184–189CrossRefGoogle Scholar
  9. 9.
    Ishigami S, Uenosono Y, Arigami T, Yanagita S, Okumura H, Uchikado Y, Kita Y, Kurahara H, Kijima Y, Nakajo A, Maemura K, Natsugoe S (2014) Clinical utility of perioperative staging laparoscopy for advanced gastric cancer. World J Surg Oncol 12(1):350CrossRefGoogle Scholar
  10. 10.
    Ellebrecht DB, Kuempers C, Horn M, Keck T, Kleemann M (2019) Confocal laser microscopy as novel approach for real-time and in-vivo tissue examination during minimal-invasive surgery in colon cancer. Surg Endosc 33(6):1811–1817CrossRefGoogle Scholar
  11. 11.
    Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, van der Laak JA, Van Ginneken B, Sánchez CI (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88CrossRefGoogle Scholar
  12. 12.
    Goceri E, Goceri N (2017) Deep learning in medical image analysis: recent advances and future trends. In: International conferences computer graphics, visualization, computer vision and image processing, pp 305–311Google Scholar
  13. 13.
    Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7639):115CrossRefGoogle Scholar
  14. 14.
    Shen D, Wu G, Suk HI (2017) Deep learning in medical image analysis. Ann Rev Biomed Eng 19:221–248CrossRefGoogle Scholar
  15. 15.
    Bengio Y (2012) Deep learning of representations for unsupervised and transfer learning. In: Proceedings of ICML workshop on unsupervised and transfer learning, pp 17–36Google Scholar
  16. 16.
    Hoo-Chang S, Roth HR, Gao M, Lu L, Xu Z, Nogues I, Yao J, Mollura D, Summers RM (2016) Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans Med Imaging 35(5):1285CrossRefGoogle Scholar
  17. 17.
    Gessert N, Lutz M, Heyder M, Latus S, Leistner DM, Abdelwahed YS, Schlaefer A (2019) Automatic plaque detection in IVOCT pullbacks using convolutional neural networks. IEEE Trans Med Imaging 38(2):426–434CrossRefGoogle Scholar
  18. 18.
    Tajbakhsh N, Shin JY, Gurudu SR, Hurst RT, Kendall CB, Gotway MB, Liang J (2016) Convolutional neural networks for medical image analysis: full training or fine tuning? IEEE Trans Med Imaging 35(5):1299–1312CrossRefGoogle Scholar
  19. 19.
    Rajadhyaksha M, Grossman M, Esterowitz D, Webb RH, Anderson RR (1995) In vivo confocal scanning laser microscopy of human skin: melanin provides strong contrast. J Investig Dermatol 104(6):946–952CrossRefGoogle Scholar
  20. 20.
    Niederer RL, Perumal D, Sherwin T, McGhee CN (2007) Age-related differences in the normal human cornea: a laser scanning in vivo confocal microscopy study. Br J Ophthalmol 91(9):1165–1169CrossRefGoogle Scholar
  21. 21.
    Aubreville M, Knipfer C, Oetter N, Jaremenko C, Rodner E, Denzler J, Bohr C, Neumann H, Stelzle F, Maier A (2017) Automatic classification of cancerous tissue in laser endomicroscopy images of the oral cavity using deep learning. Sci Rep 7(1):11979CrossRefGoogle Scholar
  22. 22.
    Aubreville M, Stoeve M, Oetter N, Goncalves M, Knipfer C, Neumann H, Bohr C, Stelzle F, Maier A (2019) Deep learning-based detection of motion artifacts in probe-based confocal laser endomicroscopy images. Int J Comput Assist Radiol Surg 14(1):31–42CrossRefGoogle Scholar
  23. 23.
    Wiltgen M, Bloice M (2016) Automatic interpretation of melanocytic images in confocal laser scanning microscopy. In: Microscopy and analysis. InTechGoogle Scholar
  24. 24.
    Hong J, Park By, Park H (2017) Convolutional neural network classifier for distinguishing Barrett’s esophagus and neoplasia endomicroscopy images. In: 2017 39th annual international conference of the IEEE engineering in medicine and biology society (EMBC), IEEE, pp 2892–2895Google Scholar
  25. 25.
    Izadyyazdanabadi M, Belykh E, Mooney MA, Eschbacher JM, Nakaji P, Yang Y, Preul MC (2018) Prospects for theranostics in neurosurgical imaging: empowering confocal laser endomicroscopy diagnostics via deep learning. Front Oncol 8:240CrossRefGoogle Scholar
  26. 26.
    Izadyyazdanabadi M, Belykh E, Martirosyan N, Eschbacher J, Nakaji P, Yang Y, Preul MC (2017) Improving utility of brain tumor confocal laser endomicroscopy: objective value assessment and diagnostic frame detection with convolutional neural networks. In: Medical imaging 2017: computer-aided diagnosis, vol. 10134. International Society for Optics and Photonics, p 101342JGoogle Scholar
  27. 27.
    Izadyyazdanabadi M, Belykh E, Cavallo C, Zhao X, Gandhi S, Moreira LB, Eschbacher J, Nakaji P, Preul MC, Yang Y (2018) Weakly-supervised learning-based feature localization for confocal laser endomicroscopy glioma images. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 300–308Google Scholar
  28. 28.
    Izadyyazdanabadi M, Belykh E, Mooney M, Martirosyan N, Eschbacher J, Nakaji P, Preul MC, Yang Y (2018) Convolutional neural networks: ensemble modeling, fine-tuning and unsupervised semantic localization for neurosurgical CLE images. J Vis Commun Image Represent 54:10–20CrossRefGoogle Scholar
  29. 29.
    Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
  30. 30.
    Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: CVPR, pp 2818–2826Google Scholar
  31. 31.
    Huang G, Liu Z, Weinberger KQ, van der Maaten L (2016) Densely connected convolutional networks. arXiv preprint arXiv:1608.06993
  32. 32.
    Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141Google Scholar
  33. 33.
    Gessert N, Wittig L, Drömann D, Keck T, Schlaefer A, Ellebrecht DB (2019) Feasibility of colon cancer detection in confocal laser microscopy images using convolution neural networks. In: Bildverarbeitung für die Medizin 2019Google Scholar
  34. 34.
    Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: ICMLGoogle Scholar
  35. 35.
    He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: CVPR, pp 770–778Google Scholar
  36. 36.
    Nair V, Hinton GE (2010) Rectified linear units improve restricted Boltzmann machines. In: ICML, pp 807–814Google Scholar
  37. 37.
    Xie S, Girshick R, Dollár P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 5987–5995Google Scholar
  38. 38.
    Shin HC, Roth HR, Gao M, Le Lu, Xu Z, Nogues I, Yao J, Mollura D, Summers RM (2016) Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans Med Imaging 35(5):1285–1298CrossRefGoogle Scholar
  39. 39.
    Yosinski J, Clune J, Bengio Y, Lipson H (2014) How transferable are features in deep neural networks? In: Advances in neural information processing systems, pp 3320–3328Google Scholar
  40. 40.
    Herath S, Harandi M, Porikli F (2017) Going deeper into action recognition: a survey. Image Vis Comput 60:4–21CrossRefGoogle Scholar

Copyright information

© CARS 2019

Authors and Affiliations

  1. 1.Institute of Medical TechnologyHamburg University of TechnologyHamburgGermany
  2. 2.Department of PulmologyUniversity Medical Centre Schleswig-HolsteinLübeckGermany
  3. 3.Department of SurgeryUniversity Medical Centre Schleswig-HolsteinLübeckGermany

Personalised recommendations