Skip to main content
SpringerLink
Log in

Face alignment combined with shape constraints and Gaussian heatmap

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Face alignment is a hot topic in the field of computer vision because it is an important intermediate step in face recognition, 3D face reconstruction, etc. Its task is to locate the key parts of the human face with landmarks. For the lack of shape constraints and weak correlation between landmarks in face alignment based on Gaussian heatmap regression, this paper presents a novel method combining shape constraints with Gaussian heatmap regression network. Our method consists of a 3D model parameter prediction network, an attention map generator, and a Gaussian heatmap regression network. The 3D model parameter prediction network fits the 3D face model and sample face. The attention map generator can generate an attention map containing shape constraint information based on the fitting results. The attention map and the original image will be used together as the input of the Gaussian heatmap regression network to achieve coarse to fine face alignment. The experimental results demonstrate that our method has a lower error on multiple mainstream datasets and is robust for some complex samples compared with the Gaussian heatmap regression method without shape constraints.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Kakadiaris IA, Toderici G, Evangelopoulos G, Passalis G, Chu D, Zhao X, Shah SK, Theoharis T (2017) 3d–2d face recognition with pose and illumination normalization. Comput Vis Image Understanding 154:137–151

    Article  Google Scholar 

  2. Ariz M, Villanueva A, Cabeza R (2019) Robust and accurate 2d-tracking-based 3d positioning method: application to head pose estimation. Comput Vis Image Understand 180:13–22

    Article  Google Scholar 

  3. Feng Y, Wu F, Shao X, Wang Y, Zhou X (2018) Joint 3d face reconstruction and dense alignment with position map regression network. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 534–551

  4. Xiong X, Dela Torre F (2013) Supervised descent method and its applications to face alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 532–539

  5. Cao X, Wei Y, Wen F, Sun J (2014) Face alignment by explicit shape regression. Int J Comput Vis 107(2):177–190

    Article  MathSciNet  Google Scholar 

  6. Ren S, Cao X, Wei Y, Sun J (2014) Face alignment at 3000 fps via regressing local binary features. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1685–1692

  7. Zhu S, Li C, Change Loy C, Tang X (2015) Face alignment by coarse-to-fine shape searching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4998–5006

  8. Sun Y, Wang X, Tang X (2013) Deep convolutional network cascade for facial point detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3476–3483

  9. Trigeorgis G, Snape P, Nicolaou MA, Antonakos E, Zafeiriou S (2016) Mnemonic descent method: A recurrent process applied for end-to-end face alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4177–4187

  10. Kumar A, Chellappa R (2018) Disentangling 3d pose in a dendritic cnn for unconstrained 2d face alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 430–439

  11. Newell A, Yang K, Deng J (2016) Stacked hourglass networks for human pose estimation. In: European Conference on Computer Vision, pp. 483–499. Springer

  12. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-assisted Intervention, pp. 234–241. Springer

  13. Wang J, Sun K, Cheng T, Jiang B, Deng C, Zhao Y, Liu D, Mu Y, Tan M, Wang X (2021) Deep high-resolution representation learning for visual recognition. IEEE Trans Pattern Anal Mach Intell 43(10):3349–3364

    Article  Google Scholar 

  14. Wu W, Qian C, Yang S, Wang Q, Cai Y, Zhou Q (2018) Look at boundary: A boundary-aware face alignment algorithm. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2129–2138

  15. Dapogny A, Bailly K, Cord M (2019) Decafa: Deep convolutional cascade for face alignment in the wild. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6893–6901

  16. Yan Y, Duffner S, Phutane P, Berthelier A, Blanc C, Garcia C, Chateau T (2021) 2d wasserstein loss for robust facial landmark detection. Pattern Recognit 116:107945

    Article  Google Scholar 

  17. Bulat A, Tzimiropoulos G (2017) How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks). In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1021–1030

  18. Liu Y, Jourabloo A, Ren W, Liu X (2017) Dense face alignment. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 1619–1628

  19. Zhu X, Liu X, Lei Z, Li SZ (2017) Face alignment in full pose range: A 3d total solution. IEEE Trans Pattern Anal Mach Intell 41(1):78–92

    Article  Google Scholar 

  20. Feng Y, Wu F, Shao X, Wang Y, Zhou X (2018) Joint 3d face reconstruction and dense alignment with position map regression network. In: Proceedings of the European Conference on Computer Vision, pp. 534–551

  21. Guo J, Zhu X, Yang Y, Yang F, Lei Z, Li SZ (2020) Towards fast, accurate and stable 3d dense face alignment. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIX 16, pp. 152–168. Springer

  22. Howard A, Sandler M, Chu G, Chen L-C, Chen B, Tan M, Wang W, Zhu Y, Pang R, Vasudevan V (2019) Searching for mobilenetv3. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314–1324

  23. Blanz V, Vetter T (2003) Face recognition based on fitting a 3d morphable model. IEEE Trans Pattern Anal Mach Intell 25(9):1063–1074

    Article  Google Scholar 

  24. Zhu X, Lei Z, Liu X, Shi H, Li SZ (2016) Face alignment across large poses: A 3d solution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 146–155

  25. Paysan P, Knothe R, Amberg B, Romdhani S, Vetter T (2009) A 3d face model for pose and illumination invariant face recognition. In: 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance, pp. 296–301. IEEE

  26. Sagonas C, Tzimiropoulos G, Zafeiriou S, Pantic M (2013) A semi-automatic methodology for facial landmark annotation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 896–903

  27. Zhu S, Li C, Loy C-C, Tang X (2016) Unconstrained face alignment via cascaded compositional learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3409–3417

  28. Zhang J, Di L, Liang J (2021) Face alignment based on fusion subspace and 3d fitting. IET Image Process 15(1):16–27

    Article  Google Scholar 

  29. Le V, Brandt J, Lin Z, Bourdev L, Huang TS (2012) Interactive facial feature localization. In: European Conference on Computer Vision, pp. 679–692. Springer

  30. Belhumeur PN, Jacobs DW, Kriegman DJ, Kumar N (2013) Localizing parts of faces using a consensus of exemplars. IEEE Trans Pattern Anal Mach Intell 35(12):2930–2940

    Article  Google Scholar 

  31. Zhu X, Ramanan D (2012) Face detection, pose estimation, and landmark localization in the wild. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2879–2886. IEEE

  32. Burgos-Artizzu XP, Perona P, Doll”á”r P (2013) Robust face landmark estimation under occlusion. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1513–1520

  33. Jourabloo A, Ye M, Liu X, Ren L (2017) Pose-invariant face alignment with a single cnn. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3200–3209

  34. Zhang Z, Luo P, Loy CC, Tang X (2015) Learning deep representation for face alignment with auxiliary attributes. IEEE Trans Pattern Anal Mach Intell 38(5):918–930

    Article  Google Scholar 

  35. Wu W, Yang S (2017) Leveraging intra and inter-dataset variations for robust face alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 150–159

  36. Valle R, Buenaposada JM, Valdes A, Baumela L (2018) A deeply-initialized coarse-to-fine ensemble of regression trees for face alignment. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 585–601

  37. Kowalski M, Naruniec J, Trzcinski T (2017) Deep alignment network: A convolutional neural network for robust face alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 88–97

  38. Zhao Y, Liu Y, Shen C, Gao Y, Xiong S (2020) Mobilefan: transferring deep hidden representation for face alignment. Pattern Recognit 100:107114

    Article  Google Scholar 

  39. Ghiasi G, Fowlkes CC (2014) Occlusion coherence: Localizing occluded faces with a hierarchical deformable part model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2385–2392

  40. Xiao S, Feng J, Xing J, Lai H, Yan S, Kassim A (2016) Robust facial landmark detection via recurrent attentive-refinement networks. In: European Conference on Computer Vision, pp. 57–72. Springer

  41. Wan J, Li J, Lai Z, Du B, Zhang L (2020) Robust face alignment by cascaded regression and de-occlusion. Neural Netw 123(C), 261–272

  42. Feng Z-H, Kittler J, Awais M, Huber P, Wu X-J (2018) Wing loss for robust facial landmark localisation with convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2235–2245

  43. Wang X, Bo L, Fuxin L (2019) Adaptive wing loss for robust face alignment via heatmap regression. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6970–6980

Download references

Acknowledgements

This research is supported by Open Project of Key Laboratory of Ministry of Public Security for Road Traffic Safety (Grant No. 2021ZDSYSKFKT04).

Author information

Authors and Affiliations

  1. School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, 214122, China

    Lan Di & Jiahui Zhang

  2. School of Computer Science and Artificial Intelligence, Changzhou University, Changzhou, 213164, China

    Jiuzhen Liang

Authors
  1. Lan Di
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiahui Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiuzhen Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lan Di.

Ethics declarations

Conflict of interest

The authors declared that we have no conflicts of interest to this work. We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Di, L., Zhang, J. & Liang, J. Face alignment combined with shape constraints and Gaussian heatmap. Int. J. Mach. Learn. & Cyber. 14, 4311–4324 (2023). https://doi.org/10.1007/s13042-023-01895-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-023-01895-6

Keywords

Search

Navigation