D’OraCa: Deep Learning-Based Classification of Oral Lesions with Mouth Landmark Guidance for Early Detection of Oral Cancer

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12722)


Oral cancer is a major health issue among low- and middle-income countries due to the late diagnosis. Automated algorithms and tools have the potential to identify oral lesions for early detection of oral cancer. In this paper, we aim to develop a novel deep learning framework named D’OraCa to classify oral lesions using photographic images. We are the first to develop a mouth landmark detection model for the oral images and incorporate it into the oral lesion classification model as a guidance to improve the classification accuracy. We evaluate the performance of five different deep convolutional neural networks and MobileNetV2 was chosen as the feature extractor for our proposed mouth landmark detection model. Quantitative and qualitative results demonstrate the effectiveness of the mouth landmark detection model in guiding the classification model to classify the oral lesions into four different referral decision classes. We train our proposed mouth landmark model on a combination of five datasets, containing 221,565 images. Then, we train and evaluate our proposed classification model with mouth landmark guidance using 2,455 oral images. The results are consistent with clinicians and the \(F_1\) score of the classification model is improved to 61.68%.


Deep learning Classification Oral lesions Mouth landmark 



This work was supported by the Medical Research Council under grant MR/S013865/1.


  1. 1.
    Amarasinghe, H., Johnson, N., Lalloo, R., Kumaraarachchi, M., Warnakulasuriya, S.: Derivation and validation of a risk-factor model for detection of oral potentially malignant disorders in populations with high prevalence. Br. J. Cancer 103(3), 303–309 (2010)CrossRefGoogle Scholar
  2. 2.
    Asthana, A., Zafeiriou, S., Cheng, S., Pantic, M.: Robust discriminative response map fitting with constrained local models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3444–3451 (2013)Google Scholar
  3. 3.
    Aubreville, M., et al.: Automatic classification of cancerous tissue in laserendomicroscopy images of the oral cavity using deep learning. Sci. Rep. 7(1), 1–10 (2017)CrossRefGoogle Scholar
  4. 4.
    Ayan, E., Ünver, H.M.: Diagnosis of pneumonia from chest x-ray images using deep learning. In: 2019 Scientific Meeting on Electrical-Electronics & Biomedical Engineering and Computer Science (EBBT), pp. 1–5. IEEE (2019)Google Scholar
  5. 5.
    Bao, P.T., Nguyen, H., Nhan, D.: A new approach to mouth detection using neural network. In: 2009 IITA International Conference on Control, Automation and Systems Engineering (case 2009), pp. 616–619. IEEE (2009)Google Scholar
  6. 6.
    Belhumeur, P.N., Jacobs, D.W., Kriegman, D.J., Kumar, N.: Localizing parts of faces using a consensus of exemplars. IEEE Trans. Pattern Anal. Mach. Intell. 35(12), 2930–2940 (2013)CrossRefGoogle Scholar
  7. 7.
    Bray, F., Ferlay, J., Soerjomataram, I., Siegel, R.L., Torre, L.A., Jemal, A.: Global cancer statistics 2018: globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: Cancer J. Clin. 68(6), 394–424 (2018)Google Scholar
  8. 8.
    Cao, X., Wei, Y., Wen, F., Sun, J.: Face alignment by explicit shape regression. Int. J. Comput. Vis. 107(2), 177–190 (2014)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Chandran, P., Bradley, D., Gross, M., Beeler, T.: Attention-driven cropping for very high resolution facial landmark detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5861–5870 (2020)Google Scholar
  10. 10.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)Google Scholar
  11. 11.
    Dong, X., Yan, Y., Ouyang, W., Yang, Y.: Style aggregated network for facial landmark detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 379–388 (2018)Google Scholar
  12. 12.
    Esteva, A., et al.: Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7639), 115–118 (2017)Google Scholar
  13. 13.
    Folmsbee, J., Liu, X., Brandwein-Weber, M., Doyle, S.: Active deep learning: Improved training efficiency of convolutional neural networks for tissue classification in oral cavity cancer. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 770–773. IEEE (2018)Google Scholar
  14. 14.
    Fu, Q., et al.: A deep learning algorithm for detection of oral cavity squamous cell carcinoma from photographic images: a retrospective study. EClinicalMedicine 27, 100558 (2020)Google Scholar
  15. 15.
    Gulshan, V., et al.: Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. Jama 316(22), 2402–2410 (2016)CrossRefGoogle Scholar
  16. 16.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  17. 17.
    Howard, A., et al.: Searching for mobilenetv3. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314–1324 (2019)Google Scholar
  18. 18.
    Howard, A.G., et al.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
  19. 19.
    Jeyaraj, P.R., Nadar, E.R.S.: Computer-assisted medical image classification for early diagnosis of oral cancer employing deep learning algorithm. J. Cancer Res. Clin. Oncol. 145(4), 829–837 (2019)CrossRefGoogle Scholar
  20. 20.
    Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1867–1874 (2014)Google Scholar
  21. 21.
    Kowalski, M., Naruniec, J., Trzcinski, T.: Deep alignment network: a convolutional neural network for robust face alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 88–97 (2017)Google Scholar
  22. 22.
    Krishna, M.M.R., et al.: Automated oral cancer identification using histopathological images: a hybrid feature extraction paradigm. Micron 43(2–3), 352–364 (2012)CrossRefGoogle Scholar
  23. 23.
    Laukamp, K.R., et al.: Fully automated detection and segmentation of meningiomas using deep learning on routine multiparametric MRI. Eur. Radiol. 29(1), 124–132 (2019)CrossRefGoogle Scholar
  24. 24.
    Le, V., Brandt, J., Lin, Z., Bourdev, L., Huang, T.S.: Interactive facial feature localization. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7574, pp. 679–692. Springer, Heidelberg (2012). Scholar
  25. 25.
    Li, R., et al.: Deep learning based imaging data completion for improved brain disease diagnosis. In: Golland, P., Hata, N., Barillot, C., Hornegger, J., Howe, R. (eds.) MICCAI 2014. LNCS, vol. 8675, pp. 305–312. Springer, Cham (2014). Scholar
  26. 26.
    Llewellyn, C.D., Linklater, K., Bell, J., Johnson, N.W., Warnakulasuriya, S.: An analysis of risk factors for oral cancer in young people: a case-control study. Oral Oncol. 40(3), 304–313 (2004)CrossRefGoogle Scholar
  27. 27.
    Lv, J., Shao, X., Xing, J., Cheng, C., Zhou, X.: A deep regression architecture with two-stage re-initialization for high performance facial landmark detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3317–3326 (2017)Google Scholar
  28. 28.
    Mintz, Y., Brodie, R.: Introduction to artificial intelligence in medicine. Minim. Invasive Ther. Allied Technol. 28(2), 73–81 (2019)CrossRefGoogle Scholar
  29. 29.
    Nagao, T., Warnakulasuriya, S.: Screening for oral cancer: future prospects, research and policy development for Asia. Oral Oncol. 105, 104632 (2020)Google Scholar
  30. 30.
    Pantic, M., Tomc, M., Rothkrantz, L.J.: A hybrid approach to mouth features detection. In: 2001 IEEE International Conference on Systems, Man and Cybernetics. e-Systems and e-Man for Cybernetics in Cyberspace (Cat. No. 01CH37236), vol. 2, pp. 1188–1193. IEEE (2001)Google Scholar
  31. 31.
    Rajpurkar, P., et al.: Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225 (2017)
  32. 32.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster r-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2016)CrossRefGoogle Scholar
  33. 33.
    Rimal, J., Shrestha, A., Maharjan, I.K., Shrestha, S., Shah, P.: Risk assessment of smokeless tobacco among oral precancer and cancer patients in eastern developmental region of Nepal. Asian Pac. J. Cancer Prev.: APJCP 20(2), 411 (2019)CrossRefGoogle Scholar
  34. 34.
    Saba, T., Khan, M.A., Rehman, A., Marie-Sainte, S.L.: Region extraction and classification of skin cancer: A heterogeneous framework of deep CNN features fusion and reduction. J. Med. Syst. 43(9), 1–19 (2019)CrossRefGoogle Scholar
  35. 35.
    Sagonas, C., Antonakos, E., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: database and results. Image Vis. Comput. 47, 3–18 (2016)CrossRefGoogle Scholar
  36. 36.
    Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: The first facial landmark localization challenge. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 397–403 (2013)Google Scholar
  37. 37.
    Song, B., et al.: Automatic classification of dual-modalilty, smartphone-based oral dysplasia and malignancy images using deep learning. Biomed. Opt. Express 9(11), 5318–5329 (2018)CrossRefGoogle Scholar
  38. 38.
    Tzimiropoulos, G.: Project-out cascaded regression with an application to face alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3659–3667 (2015)Google Scholar
  39. 39.
    Uthoff, R.D., et al.: Point-of-care, smartphone-based, dual-modality, dual-view, oral cancer screening device with neural network classification for low-resource communities. PloS ONE 13(12), e0207493 (2018)Google Scholar
  40. 40.
    Van der Waal, I., de Bree, R., Brakenhoff, R., Coebegh, J.: Early diagnosis in primary oral cancer: is it possible? Medicina oral, patologia oral y cirugia bucal 16(3), e300–e305 (2011)CrossRefGoogle Scholar
  41. 41.
    Welikala, R.A., et al.: Automated detection and classification of oral lesions using deep learning for early detection of oral cancer. IEEE Access 8, 132677–132693 (2020)CrossRefGoogle Scholar
  42. 42.
    Welikala, R.A., et al.: Fine-tuning deep learning architectures for early detection of oral cancer. In: Bebis, G., Alekseyev, M., Cho, H., Gevertz, J., Rodriguez Martinez, M. (eds.) ISMCO 2020. LNCS, vol. 12508, pp. 25–31. Springer, Cham (2020). Scholar
  43. 43.
    Xiong, X., De la Torre, F.: Supervised descent method and its applications to face alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 532–539 (2013)Google Scholar
  44. 44.
    Xu, S., et al.: An early diagnosis of oral cancer based on three-dimensional convolutional neural networks. IEEE Access 7, 158603–158611 (2019)CrossRefGoogle Scholar
  45. 45.
    Yu, X., Huang, J., Zhang, S., Yan, W., Metaxas, D.N.: Pose-free facial landmark fitting via optimized part mixtures and cascaded deformable shape model. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1944–1951 (2013)Google Scholar
  46. 46.
    Yu, X., Zhou, F., Chandraker, M.: Deep deformation network for object landmark localization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9909, pp. 52–70. Springer, Cham (2016). Scholar
  47. 47.
    Zhang, Z., Luo, P., Loy, C.C., Tang, X.: Facial landmark detection by deep multi-task learning. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 94–108. Springer, Cham (2014). Scholar
  48. 48.
    Zhu, X., Ramanan, D.: Face detection, pose estimation, and landmark localization in the wild. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2879–2886. IEEE (2012)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2021

Authors and Affiliations

  1. 1.Centre of Image and Signal Processing, Faculty of Computer Science and Information TechnologyUniversiti MalayaKuala LumpurMalaysia
  2. 2.Head and Neck Cancer Research Team, Cancer Research Malaysia47500Malaysia
  3. 3.Digital Information Research Centre, Faculty of Science, Engineering and ComputingKingston UniversitySurreyUK
  4. 4.Department of Oral and Maxillofacial Clinical Sciences, Faculty of DentistryUniversiti MalayaKuala LumpurMalaysia
  5. 5.Faculty of DentistryMAHSA UniversityBandar Saujana PutraMalaysia
  6. 6.Centre for Research in Oral Cancer, Department of Oral Medicine and Periodontology, Faculty of Dental SciencesUniversity of PeradeniyaPeradeniyaSri Lanka
  7. 7.Department of Oral Medicine and RadiologyBP Koirala Institute of Health SciencesDharanNepal
  8. 8.Oral and Maxillofacial Pathology, Radiology and MedicineNew York UniversityNew YorkUSA
  9. 9.Faculty of DentistryTrisakti UniversityKota Jakarta BaratIndonesia
  10. 10.Oral Medicine and RadiologyJagadguru Sri Shivarathreeshwara UniversityMysuruIndia
  11. 11.Institute of Dentistry, University of AberdeenAberdeenUK

Personalised recommendations