Skip to main content
SpringerLink
Log in

Enhancing facial geometry analysis by DeepFaceLandmark leveraging ResNet101 and transfer learning

  • Original Research
  • Published:
International Journal of Information Technology Aims and scope Submit manuscript

Abstract

The face recognition is the pivotal component in the surveillance system. To comprehensively assess facial structures, acquiring facial features becomes indispensable. This information is effectively represented by facial landmarks, which serve as specific locations of key characteristics. This manuscript proposes a novel DeepFaceLandmark algorithm by incorporating a transfer learning approach with the ResNet architecture on the Dlib iBUG 300-W dataset to detect the coordinates of 68 facial landmarks. This proposed work investigates the rationale behind encompassing DeepFaceLandmark in conjunction with facial feature extraction and face geometry analysis. The proposed work detects facial landmarks in challenging situations such as illumination, expression, head-pose, and occlusion to extract the facial features. In addition to these challenges, the proposed work also detected the facial landmarks on the face which are occluded via glass and partially visible due to the impact of light on glass. Furthermore, the author also specifically delves into elucidating the categories of face landmarks, detailing the employed algorithms, and evaluating their performance. Performance evaluation resulted in an impressive accuracy of 98.76%. The recorded training loss was 0.0003, with a validation loss of 0.0007 for the train-test split, and a minimum loss of 0.0006. Furthermore, the analysis extends its performance evaluation by conducting comparisons with other comparable state-of-the-art techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22

Similar content being viewed by others

Notes

  1. The dataset is publicly available and can be downloaded via http://dlib.net/files/data/ibug_300W_large_face_landmark_dataset.tar.gz.

References

  1. Murphy-Chutorian E, Trivedi MM (2009) Head pose estimation in computer vision: a survey. IEEE Trans Pattern Anal Mach Intell 31(4):607–626. https://doi.org/10.1109/TPAMI.2008.106

    Article  Google Scholar 

  2. Mallat K, Dugelay JL, Križaj J, Peer P, Štruc V, Dobrišek S, Chateau T (2019) A Survey on Transfer Learning. IEEE Trans Pattern Anal Mach Intell 22(1): 681–685. https://doi.org/10.1109/34.927467

  3. Prakash SR, Singh PN (2023) Background region based Face orientation prediction through HSV skin color model and K-Means clustering. Int J Inf Technol 15(3):1275–1288. https://doi.org/10.1007/s41870-023-01174-1

    Article  Google Scholar 

  4. Wu Y, Ji Q (2019) Facial landmark detection: a literature survey. Int J Comput Vision 127(2):115–142. https://doi.org/10.1007/s11263-018-1097-z

    Article  Google Scholar 

  5. Zhang H, Li Q, Sun Z, Liu Y (2018) Combining data-driven and model-driven methods for robust facial landmark detection. IEEE Trans Inf Forensics Secur 13(10):2409–2422. https://doi.org/10.1109/TIFS.2018.2800901

    Article  Google Scholar 

  6. Jindal A, Priya DR (2019) Landmark points detection in case of human facial tracking and detection. Int J Eng Adv Technol 9(2): 3769–3776. https://doi.org/10.35940/ijeat.b3367.129219

  7. Wu Y, Hassner T, Kim K, Medioni G, Natarajan P (2018) Facial landmark detection with tweaked convolutional neural networks. IEEE Trans Pattern Anal Mach Intell 40(12):3067–3074. https://doi.org/10.1109/TPAMI.2017.2787130

    Article  Google Scholar 

  8. Gondhi NK, Kour N, Effendi S, Kaushik K (2018) An efficient algorithm for facial landmark detection using haar-like features coupled with corner detection following anthropometric constraints. 2nd International Conference on Telecommunication and Networks, TEL-NET 2017, 2018-Janua, 1–6. https://doi.org/10.1109/TEL-NET.2017.8343517

  9. Wu T, Turaga P, Chellappa R (2012) Age estimation and face verification across aging using landmarks. IEEE Trans Inf Forensics Secur 7(6):1780–1788. https://doi.org/10.1109/TIFS.2012.2213812

    Article  Google Scholar 

  10. Devries T, Biswaranjan K, Taylor GW (2014) Multi-task learning of facial landmarks and expression. Proc Conf Comput Robot Vision CRV 2014:98–103. https://doi.org/10.1109/CRV.2014.21

    Article  Google Scholar 

  11. Bodini M (2019) A review of facial landmark extraction in 2D images and videos using deep learning. Big Data Cognitive Computing 3(1):1–14. https://doi.org/10.3390/bdcc3010014

    Article  MathSciNet  Google Scholar 

  12. Kim T, Mok J, Lee E (2021) Detecting facial region and landmarks at once via deep network. Sensors, 21(16). https://doi.org/10.3390/s21165360

  13. Sagonas C, Antonakos E, Tzimiropoulos G, Zafeiriou S, Pantic M (2016) 300 faces in-the-wild challenge: database and results. Image Vis Comput 47:3–18. https://doi.org/10.1016/j.imavis.2016.01.002

    Article  Google Scholar 

  14. Zane E, Yang Z, Pozzan L, Guha T, Narayanan S, Grossman RB (2019) Motion-capture patterns of voluntarily mimicked dynamic facial expressions in children and adolescents with and without ASD. J Autism Dev Disord 49(3):1062–1079. https://doi.org/10.1007/s10803-018-3811-7

    Article  Google Scholar 

  15. Xing J, Niu Z, Huang J, Hu W, Zhou X, Yan S (2018) Towards robust and accurate multi-view and partially-occluded face alignment. IEEE Trans Pattern Anal Mach Intell 40(4):987–1001. https://doi.org/10.1109/TPAMI.2017.2697958

    Article  Google Scholar 

  16. Chavan SA, Chaudhari NM, Ramteke RJ, Pawar UB (2023) Mathematical analysis behind occlusion handling in image with deep learning. Int J Inf Technol 15(7):3741–3749. https://doi.org/10.1007/s41870-023-01408-2

    Article  Google Scholar 

  17. Cootes TF, Taylor CJ, Cooper DH, Graham J (1995) Active shape models-their training and application. Comput Vis Image Understanding, 61(1): 38–59. https://doi.org/10.1006/cviu.1995.1004

  18. Cristinacce D, Cootes T (2007) Boosted regression active shape models. BMVC 2007—Proceedings of the British Machine Vision Conference 2007. https://doi.org/10.5244/C.21.79

  19. Cootes TF, Edwards GJ, Taylor C J (2001) Active Appearance and Models tracking a single deforming object we match a model which can fit and a whole class of objects. Ieee Transactions on Pattern Analysis and Machine Intelligence, Vol. 23, No. 6, June, 23(6), 681–685.

  20. Tzimiropoulos, G., & Pantic, M. (2013). Optimization problems for fast AAM fitting in-the-wild. Proceedings of the IEEE International Conference on Computer Vision, 593–600. https://doi.org/10.1109/ICCV.2013.79

  21. Mallat K, Dugelay JL (2020) Facial landmark detection on thermal data via fully annotated visible-to-thermal data synthesis. IJCB 2020 - IEEE/IAPR International Joint Conference on Biometrics. https://doi.org/10.1109/IJCB48548.2020.9304854

  22. Alabort-I-Medina J, Zafeiriou S (2014) Bayesian active appearance models. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 3438–3445. https://doi.org/10.1109/CVPR.2014.439

  23. Wang X, Fu T, Liao S, Wang S, Lei Z, Mei T (2020) Exclusivity-consistency regularized knowledge distillation for face recognition. lecture notes in computer science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 12369 LNCS, 325–342. https://doi.org/10.1007/978-3-030-58586-0_20

  24. Shin S, Lee J, Lee J, Yu Y, Lee K (2022) Teaching where to look: attention similarity knowledge distillation for low resolution face recognition. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 13672 LNCS, 631–647. https://doi.org/10.1007/978-3-031-19775-8_37

  25. Yang C, An Z, Zhou H, Cai L, Zhi X, Wu J, Zhang Q (2022) MixSKD: Self-Knowledge Distillation from Mixup for Image Recognition. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 13684 LNCS, 534–551. https://doi.org/10.1007/978-3-031-20053-3_31

  26. Liu J, Qin H, Wu Y, Guo J, Liang D, Xu K (2022) CoupleFace: Relation Matters for Face Recognition Distillation. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 13672 LNCS, 683–700. https://doi.org/10.1007/978-3-031-19775-8_40

  27. Wen Y, Zhang K, B, Z. L., & Qiao, Y. (2016) A Discriminative Feature Learning Approach. Eccv 1:499–515

    Google Scholar 

  28. Ling H, Wu J, Huang J, Chen J, Li P (2020) Attention-based convolutional neural network for deep face recognition. Multimedia Tools Appl 79(9–10):5595–5616. https://doi.org/10.1007/s11042-019-08422-2

    Article  Google Scholar 

  29. Ge H, Zhu Z, Dai Y, Wang B, Wu X (2022) Facial expression recognition based on deep learning. Comput Methods Programs Biomed 215:106621. https://doi.org/10.1016/j.cmpb.2022.106621

    Article  Google Scholar 

  30. Deng J, Guo J, Yang J, Xue N, Kotsia I, Zafeiriou S (2022) ArcFace: Additive Angular Margin Loss for Deep Face Recognition. IEEE Trans Pattern Anal Mach Intell 44(10):5962–5979. https://doi.org/10.1109/TPAMI.2021.3087709

    Article  Google Scholar 

  31. Chaurasiya R, Ganotra D (2023) Deep dilated CNN based image denoising. Int J Inf Technol 15(1):137–148. https://doi.org/10.1007/s41870-022-01125-2

    Article  Google Scholar 

  32. Chim S, Lee JG, Park HH (2019) Dilated skip convolution for facial landmark detection. Sensors (Switzerland) 19(24):1–21. https://doi.org/10.3390/s19245350

    Article  Google Scholar 

  33. Gangonda SS, Patavardhan PP, Karande KJ (2022) VGHN: variations aware geometric moments and histogram features normalization for robust uncontrolled face recognition. Int J Inf Technol 14(4):1823–1834. https://doi.org/10.1007/s41870-021-00703-0

    Article  Google Scholar 

  34. Hamid Y, Elyassami S, Gulzar Y, Balasaraswathi VR, Habuza T, Wani S (2023) An improvised CNN model for fake image detection. Int J Inf Technol 15(1):5–15. https://doi.org/10.1007/s41870-022-01130-5

    Article  Google Scholar 

  35. Shukla AK, Shukla A, Singh R (2023) Automatic attendance system based on CNN–LSTM and face recognition. Int J Inf Technol. https://doi.org/10.1007/s41870-023-01495-1

    Article  Google Scholar 

  36. Karanwal S (2023) Improved local descriptor (ILD): a novel fusion method in face recognition. Int J Inf Technol 15(4):1885–1894. https://doi.org/10.1007/s41870-023-01245-3

    Article  Google Scholar 

  37. Gokulakrishnan S, Chakrabarti P, Hung BT, Shankar SS (2023) An optimized facial recognition model for identifying criminal activities using deep learning strategy. Int J Inf Technol 15(7):3907–3921. https://doi.org/10.1007/s41870-023-01420-6

    Article  Google Scholar 

  38. Yadav R, Priyanka, Kacker P (2023) AutoMEDSys: automatic facial micro-expression detection system using random fourier features based neural network. Int J Inform Technol. https://doi.org/10.1007/s41870-023-01662-4

  39. Revina IM, Emmanuel WRS (2021) A survey on human face expression recognition techniques. J King Saud Univ—Comput Inform Sci 33(6):619–628. https://doi.org/10.1016/j.jksuci.2018.09.002

    Article  Google Scholar 

  40. Li J, Jin K, Zhou D, Kubota N, Ju Z (2020) Attention mechanism-based CNN for facial expression recognition. Neurocomputing 411:340–350. https://doi.org/10.1016/j.neucom.2020.06.014

    Article  Google Scholar 

  41. Chen J, Lv Y, Xu R, Xu C (2019) Automatic social signal analysis: Facial expression recognition using difference convolution neural network. J Parallel Distributed Comput 131:97–102. https://doi.org/10.1016/j.jpdc.2019.04.017

    Article  Google Scholar 

  42. Ji Y, Hu Y, Yang Y, Shen F, Shen HT (2019) Cross-domain facial expression recognition via an intra-category common feature and inter-category Distinction feature fusion network. Neurocomputing 333:231–239. https://doi.org/10.1016/j.neucom.2018.12.037

    Article  Google Scholar 

  43. Zhang H, Huang B, Tian G (2020) Facial expression recognition based on deep convolution long short-term memory networks of double-channel weighted mixture. Pattern Recogn Lett 131:128–134. https://doi.org/10.1016/j.patrec.2019.12.013

    Article  Google Scholar 

  44. Jain DK, Zhang Z, Huang K (2020) Multi angle optimal pattern-based deep learning for automatic facial expression recognition. Pattern Recogn Lett 139:157–165. https://doi.org/10.1016/j.patrec.2017.06.025

    Article  Google Scholar 

  45. Wang Z, Zeng F, Liu S, Zeng B (2021) OAENet: Oriented attention ensemble for accurate facial expression recognition. Pattern Recogn 112:107694. https://doi.org/10.1016/j.patcog.2020.107694

    Article  Google Scholar 

  46. Xie, W., Jia, X., Shen, L., & Yang, M. (2019). Sparse deep feature learning for facial expression recognition. Pattern Recognition, 96. https://doi.org/10.1016/j.patcog.2019.106966

  47. Zheng X, Guo Y, Huang H, Li Y, He R (2020) A survey of deep facial attribute analysis. Int J Comput Vision 128(8–9):2002–2034. https://doi.org/10.1007/s11263-020-01308-z

    Article  Google Scholar 

  48. Jaber AG, Muniyandi RC, Usman OL, Singh HKR (2022) A hybrid method of enhancing accuracy of facial recognition system using gabor filter and stacked sparse autoencoders deep neural network. Applied Sciences (Switzerland), 12(21). https://doi.org/10.3390/app122111052

  49. Jain N, Kumar S, Kumar A, Shamsolmoali P, Zareapoor M (2018) Hybrid deep neural networks for face emotion recognition. Pattern Recogn Lett 115:101–106. https://doi.org/10.1016/j.patrec.2018.04.010

    Article  Google Scholar 

  50. Shao J, Qian Y (2019) Three convolutional neural network models for facial expression recognition in the wild. Neurocomputing 355:82–92. https://doi.org/10.1016/j.neucom.2019.05.005

    Article  Google Scholar 

  51. Weiss K, Khoshgoftaar TM, Wang DD (2016) A survey of transfer learning. Journal of Big Data (Vol. 3). Springer International Publishing. https://doi.org/10.1186/s40537-016-0043-6

  52. Panigrahi S, Nanda A, Swarnkar T (2021) A Survey on transfer learning. Smart Innovation Syst Technol 194:781–789. https://doi.org/10.1007/978-981-15-5971-6_83

    Article  Google Scholar 

  53. Pan SJ, Yang Q (2010) A survey on transfer learning. IEEE Trans Knowl Data Eng 22(10):1345–1359. https://doi.org/10.1109/TKDE.2009.191

    Article  Google Scholar 

  54. Sharma NK, Rahamatkar S, Rathore AS (2023) Analysis of facial geometry to identify and extract face landmarks using transfer learning. Proceedings of the International Conference on Engineering Research and Application 2022 (Icera 2022), 2936, 020034. https://doi.org/10.1063/5.0175339

  55. Agrawal S, Sahu SP (2023) Image-based Parkinson disease detection using deep transfer learning and optimization algorithm. Int J Inf Technol. https://doi.org/10.1007/s41870-023-01601-3

    Article  Google Scholar 

  56. Mosayyebi F, Seyedarabi H, Afrouzian R (2023) Gender recognition in masked facial images using EfficientNet and transfer learning approach. Int J Inf Technol. https://doi.org/10.1007/s41870-023-01565-4

    Article  Google Scholar 

  57. Ma J, Li X, Ren Y, Yang R, Zhao Q (2021) Landmark-Based Facial Feature Construction and Action Unit Intensity Prediction. Mathematical Problems in Engineering, 2021. https://doi.org/10.1155/2021/6623239

  58. Križaj J, Peer P, Štruc V, Dobrišek S (2020) Simultaneous multi-descent regression and feature learning for facial landmarking in depth images. Neural Comput Appl 32(24):17909–17926. https://doi.org/10.1007/s00521-019-04529-7

    Article  Google Scholar 

  59. Le V, Brandt J, Lin Z, Bourdev L, Huang TS (2012) Interactive facial feature localization. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 7574 LNCS(PART 3), 679–692. https://doi.org/10.1007/978-3-642-33712-3_49

  60. Pourramezan Fard A, Mahoor MH (2022) Facial landmark points detection using knowledge distillation-based neural networks. Comput Vis Image Underst 215(2015):1–15. https://doi.org/10.1016/j.cviu.2021.103316

    Article  Google Scholar 

  61. Cootes TF, Edwards GJ, Taylor CJ (2001) Active appearance models. IEEE Trans Pattern Anal Mach Intell 23(6):681–685. https://doi.org/10.1109/34.927467

    Article  Google Scholar 

  62. Khabarlak K, Koriashkina L (2022) Fast Facial Landmark Detection and Applications: A Survey. Journal of Computer Science and Technology(Argentina), 22(1), 12–41. https://doi.org/10.24215/16666038.22.e02

  63. Sagonas C, Tzimiropoulos G, Zafeiriou S, Pantic M (2013) 300 faces in-the-wild challenge: The first facial landmark Localization Challenge. Proceedings of the IEEE International Conference on Computer Vision, 397–403. https://doi.org/10.1109/ICCVW.2013.59

  64. Köstinger M, Wohlhart P, Roth PM, Bischof H (2011) Annotated facial landmarks in the wild: A large-scale, real-world database for facial landmark localization. Proceedings of the IEEE International Conference on Computer Vision, (March 2014), 2144–2151. https://doi.org/10.1109/ICCVW.2011.6130513

  65. Zhu X, Ramanan D (2012) Face detection, pose estimation, and landmark localization in the wild. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2879–2886. https://doi.org/10.1109/CVPR.2012.6248014

  66. Martinez AM (n.d.) AR Face Database. Retrieved from https://www2.ece.ohio-state.edu/~aleix/ARdatabase.html#:~:text=This face database was created,70 men and 56 women).

  67. Messer K, Matas J, Kittler J, Luettin J, Maitre G (1999) XM2VTSDB : The Extended M2VTS Database University of Surrey 1 Introduction 2 Database Speci cation 3 The Database Acquisition System. Proceedings of the Second international conference on audio and video-based biometric person authentication, 1–6.

  68. Gross R, Matthews I, Cohn J, Kanade T, Baker S (n.d.). Multi-PIE.

  69. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-Decem, 770–778. https://doi.org/10.1109/CVPR.2016.90

  70. Zhang Q (2022) A novel ResNet101 model based on dense dilated convolution for image classification. SN Applied Sciences, 4(1). https://doi.org/10.1007/s42452-021-04897-7

  71. Asthana A, Zafeiriou S, Cheng S, Pantic M (2013) Robust Discriminative Response Map Fitting with Constrained Local Models. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition (pp. 3444–3451). https://doi.org/10.1109/CVPR.2013.442

  72. Baltrušaitis T, Robinson P, Morency LP (2014) Continuous conditional neural fields for structured regression. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 8692 LNCS(PART 4), 593–608. https://doi.org/10.1007/978-3-319-10593-2_39

  73. Tan M, Le QV (2019) EfficientNet: Rethinking model scaling for convolutional neural networks. 36th International Conference on Machine Learning, ICML 2019, 2019-June, 10691–10700.

  74. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Adam H (2017) MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. Retrieved from http://arxiv.org/abs/1704.04861

Download references

Acknowledgements

The research is supported by Technosys Security System Pvt. Ltd. Bhopal (M.P.) India, under a research collaboration. The authors would like to express gratitude to Technosys Security System Pvt. Ltd. for their support and valuable suggestions.

Funding

There is no direct and indirect finance involved in it.

Author information

Authors and Affiliations

  1. Amity School of Engineering & Technology, Amity University Chhattisgarh, Raipur, Chhattisgarh, India

    Nand Kishore Sharma

  2. Shri Vaishnav Vidyapeeth Vishwavidyalaya, Indore, Madhya Pradesh, India

    Abhishek Singh Rathore

  3. Avantika University, Ujjain, Madhya Pradesh, India

    Surendra Rahamatkar

Authors
  1. Nand Kishore Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Surendra Rahamatkar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Abhishek Singh Rathore
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nand Kishore Sharma.

Ethics declarations

Conflict of interest

The authors declare that there is no Conflict of Interest for publication of this manuscript. And the authors declare no competing interests.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sharma, N.K., Rahamatkar, S. & Rathore, A.S. Enhancing facial geometry analysis by DeepFaceLandmark leveraging ResNet101 and transfer learning. Int. j. inf. tecnol. (2024). https://doi.org/10.1007/s41870-024-01872-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s41870-024-01872-4

Keywords

Search

Navigation