Skip to main content
SpringerLink
Log in

D’OraCa: Deep Learning-Based Classification of Oral Lesions with Mouth Landmark Guidance for Early Detection of Oral Cancer

  • Conference paper
  • First Online:
Book cover Medical Image Understanding and Analysis (MIUA 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12722))

Included in the following conference series:

Abstract

Oral cancer is a major health issue among low- and middle-income countries due to the late diagnosis. Automated algorithms and tools have the potential to identify oral lesions for early detection of oral cancer. In this paper, we aim to develop a novel deep learning framework named D’OraCa to classify oral lesions using photographic images. We are the first to develop a mouth landmark detection model for the oral images and incorporate it into the oral lesion classification model as a guidance to improve the classification accuracy. We evaluate the performance of five different deep convolutional neural networks and MobileNetV2 was chosen as the feature extractor for our proposed mouth landmark detection model. Quantitative and qualitative results demonstrate the effectiveness of the mouth landmark detection model in guiding the classification model to classify the oral lesions into four different referral decision classes. We train our proposed mouth landmark model on a combination of five datasets, containing 221,565 images. Then, we train and evaluate our proposed classification model with mouth landmark guidance using 2,455 oral images. The results are consistent with clinicians and the \(F_1\) score of the classification model is improved to 61.68%.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Amarasinghe, H., Johnson, N., Lalloo, R., Kumaraarachchi, M., Warnakulasuriya, S.: Derivation and validation of a risk-factor model for detection of oral potentially malignant disorders in populations with high prevalence. Br. J. Cancer 103(3), 303–309 (2010)

    Article  Google Scholar 

  2. Asthana, A., Zafeiriou, S., Cheng, S., Pantic, M.: Robust discriminative response map fitting with constrained local models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3444–3451 (2013)

    Google Scholar 

  3. Aubreville, M., et al.: Automatic classification of cancerous tissue in laserendomicroscopy images of the oral cavity using deep learning. Sci. Rep. 7(1), 1–10 (2017)

    Article  Google Scholar 

  4. Ayan, E., Ünver, H.M.: Diagnosis of pneumonia from chest x-ray images using deep learning. In: 2019 Scientific Meeting on Electrical-Electronics & Biomedical Engineering and Computer Science (EBBT), pp. 1–5. IEEE (2019)

    Google Scholar 

  5. Bao, P.T., Nguyen, H., Nhan, D.: A new approach to mouth detection using neural network. In: 2009 IITA International Conference on Control, Automation and Systems Engineering (case 2009), pp. 616–619. IEEE (2009)

    Google Scholar 

  6. Belhumeur, P.N., Jacobs, D.W., Kriegman, D.J., Kumar, N.: Localizing parts of faces using a consensus of exemplars. IEEE Trans. Pattern Anal. Mach. Intell. 35(12), 2930–2940 (2013)

    Article  Google Scholar 

  7. Bray, F., Ferlay, J., Soerjomataram, I., Siegel, R.L., Torre, L.A., Jemal, A.: Global cancer statistics 2018: globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: Cancer J. Clin. 68(6), 394–424 (2018)

    Google Scholar 

  8. Cao, X., Wei, Y., Wen, F., Sun, J.: Face alignment by explicit shape regression. Int. J. Comput. Vis. 107(2), 177–190 (2014)

    Article  MathSciNet  Google Scholar 

  9. Chandran, P., Bradley, D., Gross, M., Beeler, T.: Attention-driven cropping for very high resolution facial landmark detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5861–5870 (2020)

    Google Scholar 

  10. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)

    Google Scholar 

  11. Dong, X., Yan, Y., Ouyang, W., Yang, Y.: Style aggregated network for facial landmark detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 379–388 (2018)

    Google Scholar 

  12. Esteva, A., et al.: Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7639), 115–118 (2017)

    Google Scholar 

  13. Folmsbee, J., Liu, X., Brandwein-Weber, M., Doyle, S.: Active deep learning: Improved training efficiency of convolutional neural networks for tissue classification in oral cavity cancer. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 770–773. IEEE (2018)

    Google Scholar 

  14. Fu, Q., et al.: A deep learning algorithm for detection of oral cavity squamous cell carcinoma from photographic images: a retrospective study. EClinicalMedicine 27, 100558 (2020)

    Google Scholar 

  15. Gulshan, V., et al.: Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. Jama 316(22), 2402–2410 (2016)

    Article  Google Scholar 

  16. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  17. Howard, A., et al.: Searching for mobilenetv3. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314–1324 (2019)

    Google Scholar 

  18. Howard, A.G., et al.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)

  19. Jeyaraj, P.R., Nadar, E.R.S.: Computer-assisted medical image classification for early diagnosis of oral cancer employing deep learning algorithm. J. Cancer Res. Clin. Oncol. 145(4), 829–837 (2019)

    Article  Google Scholar 

  20. Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1867–1874 (2014)

    Google Scholar 

  21. Kowalski, M., Naruniec, J., Trzcinski, T.: Deep alignment network: a convolutional neural network for robust face alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 88–97 (2017)

    Google Scholar 

  22. Krishna, M.M.R., et al.: Automated oral cancer identification using histopathological images: a hybrid feature extraction paradigm. Micron 43(2–3), 352–364 (2012)

    Article  Google Scholar 

  23. Laukamp, K.R., et al.: Fully automated detection and segmentation of meningiomas using deep learning on routine multiparametric MRI. Eur. Radiol. 29(1), 124–132 (2019)

    Article  Google Scholar 

  24. Le, V., Brandt, J., Lin, Z., Bourdev, L., Huang, T.S.: Interactive facial feature localization. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7574, pp. 679–692. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33712-3_49

    Chapter  Google Scholar 

  25. Li, R., et al.: Deep learning based imaging data completion for improved brain disease diagnosis. In: Golland, P., Hata, N., Barillot, C., Hornegger, J., Howe, R. (eds.) MICCAI 2014. LNCS, vol. 8675, pp. 305–312. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10443-0_39

    Chapter  Google Scholar 

  26. Llewellyn, C.D., Linklater, K., Bell, J., Johnson, N.W., Warnakulasuriya, S.: An analysis of risk factors for oral cancer in young people: a case-control study. Oral Oncol. 40(3), 304–313 (2004)

    Article  Google Scholar 

  27. Lv, J., Shao, X., Xing, J., Cheng, C., Zhou, X.: A deep regression architecture with two-stage re-initialization for high performance facial landmark detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3317–3326 (2017)

    Google Scholar 

  28. Mintz, Y., Brodie, R.: Introduction to artificial intelligence in medicine. Minim. Invasive Ther. Allied Technol. 28(2), 73–81 (2019)

    Article  Google Scholar 

  29. Nagao, T., Warnakulasuriya, S.: Screening for oral cancer: future prospects, research and policy development for Asia. Oral Oncol. 105, 104632 (2020)

    Google Scholar 

  30. Pantic, M., Tomc, M., Rothkrantz, L.J.: A hybrid approach to mouth features detection. In: 2001 IEEE International Conference on Systems, Man and Cybernetics. e-Systems and e-Man for Cybernetics in Cyberspace (Cat. No. 01CH37236), vol. 2, pp. 1188–1193. IEEE (2001)

    Google Scholar 

  31. Rajpurkar, P., et al.: Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225 (2017)

  32. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2016)

    Article  Google Scholar 

  33. Rimal, J., Shrestha, A., Maharjan, I.K., Shrestha, S., Shah, P.: Risk assessment of smokeless tobacco among oral precancer and cancer patients in eastern developmental region of Nepal. Asian Pac. J. Cancer Prev.: APJCP 20(2), 411 (2019)

    Article  Google Scholar 

  34. Saba, T., Khan, M.A., Rehman, A., Marie-Sainte, S.L.: Region extraction and classification of skin cancer: A heterogeneous framework of deep CNN features fusion and reduction. J. Med. Syst. 43(9), 1–19 (2019)

    Article  Google Scholar 

  35. Sagonas, C., Antonakos, E., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: database and results. Image Vis. Comput. 47, 3–18 (2016)

    Article  Google Scholar 

  36. Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: The first facial landmark localization challenge. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 397–403 (2013)

    Google Scholar 

  37. Song, B., et al.: Automatic classification of dual-modalilty, smartphone-based oral dysplasia and malignancy images using deep learning. Biomed. Opt. Express 9(11), 5318–5329 (2018)

    Article  Google Scholar 

  38. Tzimiropoulos, G.: Project-out cascaded regression with an application to face alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3659–3667 (2015)

    Google Scholar 

  39. Uthoff, R.D., et al.: Point-of-care, smartphone-based, dual-modality, dual-view, oral cancer screening device with neural network classification for low-resource communities. PloS ONE 13(12), e0207493 (2018)

    Google Scholar 

  40. Van der Waal, I., de Bree, R., Brakenhoff, R., Coebegh, J.: Early diagnosis in primary oral cancer: is it possible? Medicina oral, patologia oral y cirugia bucal 16(3), e300–e305 (2011)

    Article  Google Scholar 

  41. Welikala, R.A., et al.: Automated detection and classification of oral lesions using deep learning for early detection of oral cancer. IEEE Access 8, 132677–132693 (2020)

    Article  Google Scholar 

  42. Welikala, R.A., et al.: Fine-tuning deep learning architectures for early detection of oral cancer. In: Bebis, G., Alekseyev, M., Cho, H., Gevertz, J., Rodriguez Martinez, M. (eds.) ISMCO 2020. LNCS, vol. 12508, pp. 25–31. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-64511-3_3

    Chapter  Google Scholar 

  43. Xiong, X., De la Torre, F.: Supervised descent method and its applications to face alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 532–539 (2013)

    Google Scholar 

  44. Xu, S., et al.: An early diagnosis of oral cancer based on three-dimensional convolutional neural networks. IEEE Access 7, 158603–158611 (2019)

    Article  Google Scholar 

  45. Yu, X., Huang, J., Zhang, S., Yan, W., Metaxas, D.N.: Pose-free facial landmark fitting via optimized part mixtures and cascaded deformable shape model. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1944–1951 (2013)

    Google Scholar 

  46. Yu, X., Zhou, F., Chandraker, M.: Deep deformation network for object landmark localization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9909, pp. 52–70. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46454-1_4

    Chapter  Google Scholar 

  47. Zhang, Z., Luo, P., Loy, C.C., Tang, X.: Facial landmark detection by deep multi-task learning. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 94–108. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10599-4_7

    Chapter  Google Scholar 

  48. Zhu, X., Ramanan, D.: Face detection, pose estimation, and landmark localization in the wild. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2879–2886. IEEE (2012)

    Google Scholar 

Download references

Acknowledgements

This work was supported by the Medical Research Council under grant MR/S013865/1.

Author information

Authors and Affiliations

  1. Centre of Image and Signal Processing, Faculty of Computer Science and Information Technology, Universiti Malaya, 50603, Kuala Lumpur, Malaysia

    Jian Han Lim, Chun Shui Tan & Chee Seng Chan

  2. Head and Neck Cancer Research Team, Cancer Research Malaysia, 47500, Subang Jaya, Malaysia

    Senthilmani Rajendran & Sok Ching Cheong

  3. Digital Information Research Centre, Faculty of Science, Engineering and Computing, Kingston University, Surrey, KT1 2EE, UK

    Roshan Alex Welikala, Paolo Remagnino & Sarah Ann Barman

  4. Department of Oral and Maxillofacial Clinical Sciences, Faculty of Dentistry, Universiti Malaya, 50603, Kuala Lumpur, Malaysia

    Thomas George Kallarakkal, Rosnah Binti Zain & Wanninayake Mudiyanselage Tilakaratne

  5. Faculty of Dentistry, MAHSA University, 42610, Bandar Saujana Putra, Jenjarom, Malaysia

    Rosnah Binti Zain

  6. Centre for Research in Oral Cancer, Department of Oral Medicine and Periodontology, Faculty of Dental Sciences, University of Peradeniya, Peradeniya, 20400, Sri Lanka

    Ruwan Duminda Jayasinghe & Wanninayake Mudiyanselage Tilakaratne

  7. Department of Oral Medicine and Radiology, BP Koirala Institute of Health Sciences, Dharan, 56700, Nepal

    Jyotsna Rimal

  8. Oral and Maxillofacial Pathology, Radiology and Medicine, New York University, New York, NY, 10010, USA

    Alexander Ross Kerr

  9. Faculty of Dentistry, Trisakti University, Kota Jakarta Barat, Jakarta, 11440, Indonesia

    Rahmi Amtha

  10. Oral Medicine and Radiology, Jagadguru Sri Shivarathreeshwara University, Mysuru, 570015, Karnataka, India

    Karthikeya Patil

  11. Institute of Dentistry, University of Aberdeen, Aberdeen, AB25 2ZR, UK

    John Gibson

Authors
  1. Jian Han Lim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chun Shui Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chee Seng Chan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Roshan Alex Welikala
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Paolo Remagnino
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Senthilmani Rajendran
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Thomas George Kallarakkal
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Rosnah Binti Zain
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Ruwan Duminda Jayasinghe
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Jyotsna Rimal
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Alexander Ross Kerr
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Rahmi Amtha
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Karthikeya Patil
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Wanninayake Mudiyanselage Tilakaratne
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. John Gibson
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Sok Ching Cheong
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Sarah Ann Barman
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Bartłomiej W. Papież

  2. Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates

    Mohammad Yaqub

  3. University of Oxford, Oxford, UK

    Jianbo Jiao

  4. University of Oxford, Oxford, UK

    Ana I. L. Namburete

  5. University of Oxford, Oxford, UK

    J. Alison Noble

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lim, J.H. et al. (2021). D’OraCa: Deep Learning-Based Classification of Oral Lesions with Mouth Landmark Guidance for Early Detection of Oral Cancer. In: Papież, B.W., Yaqub, M., Jiao, J., Namburete, A.I.L., Noble, J.A. (eds) Medical Image Understanding and Analysis. MIUA 2021. Lecture Notes in Computer Science(), vol 12722. Springer, Cham. https://doi.org/10.1007/978-3-030-80432-9_31

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-80432-9_31

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-80431-2

  • Online ISBN: 978-3-030-80432-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation