Deep learning-based detection of motion artifacts in probe-based confocal laser endomicroscopy images

  • Marc AubrevilleEmail author
  • Maike Stoeve
  • Nicolai Oetter
  • Miguel Goncalves
  • Christian Knipfer
  • Helmut Neumann
  • Christopher Bohr
  • Florian Stelzle
  • Andreas Maier
Original Article



Probe-based confocal laser endomicroscopy (pCLE) is a subcellular in vivo imaging technique capable of producing images that enable diagnosis of malign structural modifications in epithelial tissue. Images acquired with pCLE are, however, often tainted by significant artifacts that impair diagnosis. This is especially detrimental for automated image analysis, which is why said images are often excluded from recognition pipelines.


We present an approach for the automatic detection of motion artifacts in pCLE images and apply this methodology to a data set of 15 thousand images of epithelial tissue acquired in the oral cavity and the vocal folds. The approach is based on transfer learning from intermediate endpoints within a pre-trained Inception v3 network with tailored preprocessing. For detection within the non-rectangular pCLE images, we perform pooling within the activation maps of the network and evaluate this at different network depths.


We achieved area under the ROC curve values of 0.92 with the proposed method, compared to 0.80 for the best feature-based machine learning approach. Our overall accuracy with the presented approach is 94.8%.


Over traditional machine learning approaches with state-of-the-art features, we achieved significantly improved overall performance.


Deep convolutional neural networks Confocal laser endomicroscopy Motion artifact detection 


Compliance with ethical standards

Conflict of interest

All authors declare that they do not have conflicts of interest regarding the work covered by this manuscript.

Human and animal participants

All procedures involving human participants were in accordance with the 1964 Declaration of Helsinki and its later amendments.

Informed consent

The local ethics committee approved the studies (ethics committee of the University of Erlangen-Nürnberg; reference numbers 243 12 B and 60 14 B), and all patients gave their written informed consent.


  1. 1.
    Abbaci M, Breuskin I, Casiraghi O, De Leeuw F (2014) Confocal laser endomicroscopy for non-invasive head and neck cancer imaging: a comprehensive review. Oral Oncol 50(8):711–6. CrossRefGoogle Scholar
  2. 2.
    Agaimy A, Weichert W (2016) Grading of head and neck neoplasms. Der Pathol 37(4):285–292. CrossRefGoogle Scholar
  3. 3.
    Araujo H, Dias JM (1996) An introduction to the log-polar mapping (image sampling). In: Proceedings 2nd workshop on cybernetic vision. IEEE, pp 139–144.
  4. 4.
    Aubreville M, Knipfer C, Oetter N, Jaremenko Christian, Rodner E, Denzler J, Bohr C, Neumann H, Stelzle F, Maier A (2017) Automatic classification of cancerous tissue in laserendomicroscopy images of the oral cavity using deep learning. Sci Rep 7(1):11979. CrossRefGoogle Scholar
  5. 5.
    Bier B, Mualla F, Steidl S, Bohr C, Neumann H, Maier A, Hornegger J (2015) Band-pass filter design by segmentation in frequency domain for detection of epithelial cells in endomicroscope images. In: Bildverarbeitung für die Medizin. Springer, Berlin, pp 413–418.
  6. 6.
    Chauhan SS, Dayyeh BKA, Bhat YM, Gottlieb KT, Hwang JH, Komanduri S, Konda V, Lo SK, Manfredi MA, Maple JT, Murad FM, Siddiqui UD, Banerjee S, Wallace MB (2014) Confocal laser endomicroscopy. Gastrointest Endosc 80(6):928–938. CrossRefGoogle Scholar
  7. 7.
    Cireşan DC, Giusti A, Gambardella LM, Schmidhuber J (2013) Mitosis detection in breast cancer histology images with deep neural networks. Medical Image Computing and Computer-Assisted Intervention—MICCAI 2013 16(Pt 2):411–418.
  8. 8.
    Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: IEEE computer society conference on computer vision and pattern recognition, CVPR 2005. 1:886–893.
  9. 9.
    Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 248–255.
  10. 10.
    Deniz O, García GB, Salido J, De la Torre F (2011) Face recognition using histograms of oriented gradients. Pattern Recognit Lett 32(12):1598–1603. CrossRefGoogle Scholar
  11. 11.
    Dittberner A, Rodner E, Ortmann W, Stadler J, Schmidt C, Petersen I, Stallmach A, Denzler J, Guntinas-Lichius O (2016) Automated analysis of confocal laser endomicroscopy images to detect head and neck cancer. Head Neck 38(S1):E1419–E1426. CrossRefGoogle Scholar
  12. 12.
    Gil D, Ramos-Terrades O, Minchole E, Sanchez C, de Frutos NC, Diez-Ferrer M, Ortiz RM, Rosell A (2017) Classification of confocal endomicroscopy patterns for diagnosis of lung cancer. In: Medical image computing and computer-assisted intervention—MICCAI 2017, pp. 151–159. Springer, Cham.
  13. 13.
    Goncalves M, Iro H, Dittberner A, Agaimy A, Bohr C (2017) Value of confocal laser endomicroscopy in the diagnosis of vocal cord lesions. Eur Rev Med Pharmacol Sci 21:3990–3997Google Scholar
  14. 14.
    Hong J, Park By, Park H (2017) Convolutional neural network classifier for distinguishing Barrett’s esophagus and neoplasia endomicroscopy images. In: 39th Annual International conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, pp 2892–2895.
  15. 15.
    Izadyyazdanabadi M, Belykh E, Cavallo C, Zhao X, Gandhi S, Moreira LB, Eschbacher J, Nakaji P, Preul MC, Yang Y (2018) Weakly-supervised learning-based feature localization in confocal laser endomicroscopy glioma images. arXiv preprint arXiv:1804.09428v1
  16. 16.
    Izadyyazdanabadi M, Belykh E, Martirosyan N, Eschbacher J, Nakaji P, Yang Y, Preul MC (2017)Improving utility of brain tumor confocal laser endomicroscopy - objective value assessment and diagnostic frame detection with convolutional neural networks. In: Medical Imaging 2017, vol 10134. SPIE.
  17. 17.
    Izadyyazdanabadi M, Belykh E, Mooney M, Eschbacher J, Nakaji P, Yang Y, Preul MC (2018) Prospects for theranostics in neurosurgical technology—empowering confocal laser endomicroscopy diagnostics via deep learning. arXiv preprint arXiv:1804.09873
  18. 18.
    Izadyyazdanabadi M, Belykh E, Mooney M, Martirosyan N, Eschbacher J, Nakaji P, Preul MC, Yang Y (2018) Convolutional neural networks: ensemble modeling, fine-tuning and unsupervised semantic localization for neurosurgical cle images. J Vis Commun Image Represent 54:10–20. CrossRefGoogle Scholar
  19. 19.
    Jaremenko C, Maier A, Steidl S, Hornegger J, Oetter N, Knipfer C, Stelzle F, Neumann H (2015) Classification of confocal laser endomicroscopic images of the oral cavity to distinguish pathological from healthy tissue. In: Bildverarbeitung für die Medizin 2015. Springer, Berlin, pp 479–485.
  20. 20.
    Kingma D, Ba J (2014) ADAM: a method for stochastic optimization. arXiv preprint arXiv:1412.6980
  21. 21.
    Laemmel E, Genet M, Le Goualher G, Perchant A, Le Gargasson JF, Vicaut E (2004) Fibered confocal fluorescence microscopy (Cell-viZio) facilitates extended imaging in the field of microcirculation. A comparison with intravital microscopy. J Vasc Res 41(5):400–411. CrossRefGoogle Scholar
  22. 22.
    Macé V, Ahluwalia A, Coron E, Le Rhun M, Boureille A, Bossard C, Mosnier JF, Matysiak-Budnik T, Tarnawski AS (2015) Confocal laser endomicroscopy: a new gold standard for the assessment of mucosal healing in ulcerative colitis. J Gastroenterol Hepatol 30:85–92. CrossRefGoogle Scholar
  23. 23.
    Maier AK, Schebesch F, Syben C, Würfl T, Steidl S, Choi JH, Fahrig R (2017) Precision learning: towards use of known operators in neural networks. arXiv preprint arXiv:1712.00374
  24. 24.
    Maier H, Dietz A, Gewelke U, Heller W, Weidauer H (1992) Tobacco and alcohol and the risk of head and neck cancer. Clin Investig 70(3–4):320–327. Google Scholar
  25. 25.
    Muto M, Nakane M, Katada C, Sano Y, Ohtsu A, Esumi H, Ebihara S, Yoshida S (2004) Squamous cell carcinoma in situ at oropharyngeal and hypopharyngeal mucosal sites. Cancer 101(6):1375–1381. CrossRefGoogle Scholar
  26. 26.
    Nachalon Y, Alkan U, Shvero J, Yaniv D, Shkedy Y, Limon D, Popovtzer A (2017) Assessment of laryngeal cancer in patients younger than 40 years. The Laryngoscope. Google Scholar
  27. 27.
    Neumann H, Langner C, Neurath MF, Vieth M (2012) Confocal laser endomicroscopy for diagnosis of Barrett’s Esophagus. Front Oncol 2:42. CrossRefGoogle Scholar
  28. 28.
    Neumann H, Vieth M, Atreya R, Neurath MF, Mudter J (2011) Prospective evaluation of the learning curve of confocal laser endomicroscopy in patients with IBD. Histol Histopathol 26(7):867–872Google Scholar
  29. 29.
    Oetter N, Knipfer C, Rohde M, Wilmowsky C, Maier A, Brunner K, Adler W, Neukam FW, Neumann H, Stelzle F (2016) Development and validation of a classification and scoring system for the diagnosis of oral squamous cell carcinomas through confocal laser endomicroscopy. J Transl Med 14(1):1–11. CrossRefGoogle Scholar
  30. 30.
    Oquab M, Bottou L, Laptev I, Sivic J (2015) Is object localization for free? Weakly-supervised learning with convolutional neural networks. In: Proceedings of IEEE conference on computer vision and pattern recognition. IEEE, pp 685–694.
  31. 31.
    Parikh ND, Gibson J, Nagar A, Ahmed AA, Aslanian HR (2016) Confocal laser endomicroscopy features of sessile serrated adenomas/polyps. U Eur Gastroenterol J 4(4):599–603. CrossRefGoogle Scholar
  32. 32.
    Pavlov V, Meyronet D, Meyer-Bisch V, Armoiry X, Pikul B, Dumot C, Beuriat P-A, Signorelli F, Guyotat J (2016) Intraoperative probe-based confocal laser endomicroscopy in surgery and stereotactic biopsy of low-grade and high-grade gliomas. Neurosurgery 79(4):604–612. CrossRefGoogle Scholar
  33. 33.
    Robert Koch Institut. Zentrum für Krebsregisterdaten (2017) Krebs in Deutschland für 2013/2014, 11th edn. Robert Koch Institut, BerlinGoogle Scholar
  34. 34.
    Stoeve M, Aubreville M, Oetter N, Knipfer C, Neumann H, Stelzle F, Maier A (2018) Motion Artifact Detection in Confocal Laser Endomicroscopy Images. In: Bildverarbeitung für die Medizin. Springer, Berlin, pp 328–333.
  35. 35.
    Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: The IEEE conference on computer vision and pattern recognition (CVPR).
  36. 36.
    Vo K, Jaremenko Christian, Bohr C, Neumann H, Maier A (2017) Automatic classification and pathological staging of confocal laser endomicroscopic images of the vocal cords. In: Bildverarbeitung für die Medizin 2017. Springer, Berlin, pp 312–317.
  37. 37.
    Westra WH (2015) The pathology of HPV-related head and neck cancer: implications for the diagnostic pathologist. Semin Diagn Pathol 32(1):42–53. CrossRefGoogle Scholar
  38. 38.
    Wiesner C, Jäger W, Salzer A, Biesterfeld S, Kiesslich R, Hampel C, Thüroff JW, Goetz M (2010) Confocal laser endomicroscopy for the diagnosis of urothelial bladder neoplasia: a technology of the future? BJU Int 107(3):399–403. CrossRefGoogle Scholar
  39. 39.
    Xing F, Xie Y, Su H, Liu F, Yang L (2017) Deep learning in microscopy image analysis: a survey. IEEE Trans Neural Netw Learn Syst PP(99):1–19.
  40. 40.
    Yang Y, Li T, Li W, Wu H, Fan W, Zhang W (2017) Lesion detection and grading of diabetic retinopathy via two-stages deep convolutional neural networks. In: Medical image computing and computer-assisted intervention—MICCAI 2017. Springer, Cham, pp 533–540.

Copyright information

© CARS 2018

Authors and Affiliations

  1. 1.Pattern Recognition Lab, Computer ScienceFriedrich-Alexander-Universität Erlangen-NürnbergErlangenGermany
  2. 2.Department of Oral and Maxillofacial Surgery, University Hospital ErlangenFriedrich-Alexander-Universität Erlangen-NürnbergErlangenGermany
  3. 3.Department of Otorhinolaryngology, Head and Neck Surgery, University Hospital ErlangenFriedrich-Alexander-Universität Erlangen-NürnbergErlangenGermany
  4. 4.Department of Oral and Maxillofacial Surgery, University Medical Center Hamburg-EppendorfUniversität HamburgHamburgGermany
  5. 5.Department of Otorhinolaryngology, Head and Neck Surgery, University HospitalUniversität RegensburgRegensburgGermany
  6. 6.First Department of Internal Medicine, University Hospital MainzJohannes Gutenberg-Universität MainzMainzGermany

Personalised recommendations