Skip to main content
SpringerLink
Log in

A robust real-time facial alignment system with facial landmarks detection and rectification for multimedia applications

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Face detection often plays the first step in various visual applications. Large variants of facial deformations due to head movements and facial expression make it difficult to identify appropriate face region. In this paper, a robust real-time face alignment system, including facial landmarks detection and face rectification, is proposed. A facial landmarks detection model based on regression tree is utilized in the proposed system. In face rectification framework, 2-D geometrical analysis based on pitch, yaw and roll movements is designed to solve the misalignment problem in face detection. The experiments on the two datasets verify the performance significantly improved by the proposed method in the facial recognition task and outperform than those obtained by other alignment methods. Furthermore, the proposed method can achieve robust recognition results even if the amount of training images is not large.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

References

  1. Ahonen T, Hadid A, Pietikainen M (2006) Face description with local binary patterns: application to face recognition. IEEE Trans Pattern Anal Mach Intell 28(12):2037–2041

    Article  MATH  Google Scholar 

  2. Brunelli R, Poggio T (1993) Face recognition: features versus templates. IEEE Trans Pattern Anal Mach Intell 15(10):1042–1052

    Article  Google Scholar 

  3. Chen, Jun-Cheng, Vishal M. Patel, and Rama Chellappa. “Unconstrained face verification using deep cnn features.” In2016 IEEE winter conference on applications of computer vision (WACV), pp. 1–9. IEEE, (2016)

  4. Cootes T, Taylor C, Cooper D, Graham J (1995) Active shape models their training and application. Comput Vis Image Underst 61(01):389

    Article  Google Scholar 

  5. Cootes TF, Edwards GJ, Taylor CJ (2001) Active appearance models. IEEE Trans Pattern Anal Mach Intell 23(6):681–685

    Article  Google Scholar 

  6. Cover T, Hart P (1967) Nearest neighbor pattern classification. IEEE Trans Inf Theory 13(1)

  7. N Dalal and B Triggs, “histograms of oriented gradients for human detection,” in IEEE Conference on Computer Vision and Pattern Recognition, 2005

  8. Dapogny, A., Bailly, K., and Cord, M. (2019). DeCaFA: deep convolutional Cascade for face alignment in the wild. In proceedings of the IEEE international conference on computer vision (pp. 6893-6901).

  9. Ding C, Tao D (2015) Robust face recognition via multimodal deep face representation. IEEE Trans Multimedia 17(11):2049–2058

    Article  Google Scholar 

  10. Y Freund and R E Schapire, “A decision-theoretic generalization of on-line learning and an application to boosting,” in Proceedings of the 2nd European Conference on Computational Learning Theory, 1995.

  11. Gao W, Cao B, Shan S, Chen X, Zhou D, Zhang X, Zhao D (2008) The cas-peal large-scale chinese face database and baseline evaluations. IEEE Trans Syst Man Cybern Syst Hum 38(1):149–161

    Article  Google Scholar 

  12. J. Ho, M.H. Y. M.H. Yang, J. L. J. Lim, K.C. L. K.C. Lee, and D. Kriegman, “Clustering appearances of objects under varying illumination conditions,” Proceedings. 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 11–18, (2003).

  13. GB Huang, V Jain, and EG Learned-Miller, “Unsupervised joint alignment of complex images,” 2007 IEEE 11th International Conference on Computer Vision, pp. 1–8, (2007)

  14. GB Huang, M Ramesh, T Berg, and E Learned-Miller, “Labeled faces in the wild: A database for studying face recognition in unconstrained environments,” University of Massachusetts, Amherst, Tech Rep 07-49, (2007)

  15. Jiang, Huaizu, and Erik Learned-Miller. “Face detection with the faster R-CNN.” In 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), pp. 650–657. IEEE, (2017)

  16. V. Kazemi and J. Sullivan, “One millisecond face alignment with an ensemble of regression trees,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition, June (2014), pp. 1867–1874.

  17. C. Kotropoulos and I. Pitas, “Rule-based face detection in frontal views,” in 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 4, April (1997), pp. 2537–2540 vol.4.

  18. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1, ser. NIPS’12. USA: Curran Associates Inc., (2012), pp. 1097–1105. [Online]. Available: http://dl.acm.org/citation.cfm?id=2999134.2999257

  19. A. Lanitis, C. Taylor, and T. Cootes, “Automatic face identification system using flexible appearance models,” Image and Vision Computing, vol. 13, no. 5, pp. 393–401, (1995), 5th British Machine Vision Conference

  20. Li H, Wang P, Shen C (2010) Robust face recognition via accurate face alignment and sparse representation. In: 2010 International Conference on Digital Image Computing: Techniques and Applications, pp 262–269

    Google Scholar 

  21. H. Li, P. Wang, and C. Shen, “Robust Face Recognition via Accurate Face Alignment and Sparse Representation,” 2010 International Conference on Digital Image Computing: Techniques and Applications, pp. 262–269, (2010).

  22. Lowe DG (2004) Distinctive image features from scale-invariant key points. IJCV 60(2):91–110

    Article  Google Scholar 

  23. Lu H, Yang F (2014) Active Shape Model and Its Application to Face Alignment. Springer, Berlin, Heidelberg, pp 1–31

    Google Scholar 

  24. AM Martinez and R Benavente, “The AR face database,” CVC Technical Report 24, (1998).

  25. Murphy-Chutorian E, Trivedi MM (2009) Head pose estimation in computer vision: a survey. IEEE Trans Pattern Anal Mach Intell 31(4):607–626

    Article  Google Scholar 

  26. Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “Deepface: closing the gap to human-level performance in face verification,” in The IEEE conference on computer vision and pattern recognition (CVPR), (2014)

  27. Thomaz CE, Giraldi GA (2010) A new ranking method for principal components analysis and its application to face image analysis. Image Vis Comput 28(6):902–913 [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0262885609002613

    Article  Google Scholar 

  28. Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154

    Article  Google Scholar 

  29. Wright J, Yang AY, Ganesh A, Sastry SS, Ma Y (2009) Robust face recognition via sparse representation. IEEE Trans Pattern Anal Mach Intell 31(2):210–227

    Article  Google Scholar 

  30. Wu X, He R, Sun Z, Tan T (2018) A light cnn for deep face representation with noisy labels. IEEE Transactions on Information Forensics and Security 13(11):2884–2896

    Article  Google Scholar 

  31. Xiao, Shengtao, Jiashi Feng, Junliang Xing, Hanjiang Lai, Shuicheng Yan, and Ashraf Kassim. “Robust facial landmark detection via recurrent attentive-refinement networks.” In European conference on computer vision, pp. 57–72. Springer, Cham, (2016)

  32. Zhang K, Zhang Z, Li Z, Yu Q (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters 23(10):1499–1503

    Article  Google Scholar 

  33. Zhao Y, Tang F, Dong W, Huang F, Zhang X (2019) Joint face alignment and segmentation via deep multi-task learning. Multimed Tools Appl 78(10):13131–13148

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the Australian Research Council (ARC) under Grant DP180100670 and Grant DP180100656, in part by the U.S. Army Research Laboratory under Agreement W911NF-10-2-0022, and in part by the Taiwan Ministry of Science and Technology under Grant MOST 106-2218-E-009-027-MY3.

Author information

Authors and Affiliations

  1. Department of Computer Science, National Chiao Tung University, Hsinchu City, Taiwan

    Kuang Pen Chou, Sheng-Yao Su & Wen-Chieh Lin

  2. Centre for Artificial Intelligence, School of Computer Science, FEIT, University of Technology Sydney, Ultimo, NSW, Australia

    Mukesh Prasad, Jie Yang & Chin-Teng Lin

  3. Research Center of Precision Sensing and Control, Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Xian Tao

  4. Department of Computer Science and IT, Guru Ghasidas University, Bilaspur, India

    Amit Saxena

Authors
  1. Kuang Pen Chou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mukesh Prasad
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jie Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sheng-Yao Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xian Tao
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Amit Saxena
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Wen-Chieh Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Chin-Teng Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mukesh Prasad.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chou, K.P., Prasad, M., Yang, J. et al. A robust real-time facial alignment system with facial landmarks detection and rectification for multimedia applications. Multimed Tools Appl 80, 16635–16657 (2021). https://doi.org/10.1007/s11042-020-09216-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09216-7

Keywords

Search

Navigation