Skip to main content
SpringerLink
Log in

Joint face normalization and representation learning for face recognition

  • Theoretical Advances
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

Identity-independent factors, such as variations of pose, expression, illumination, etc., are the key challenges in face recognition. To avoid the effects of these factors, existing face recognition methods usually adopt two approaches: pose-invariant face feature extracting and face normalization before feature extraction. Contrary to these, we propose a single deep model jointly performing face normalization and representation learning tasks for face recognition, named normalization and reconstruction general adversarial network (NRGAN). First, the unified NRGAN model can boost the performance of the two tasks for each other. Second, NRGAN can synthesize normalized face images without the requirement of paired data, which makes our method have better generalization ability to the uncontrolled environment. Third, a factor-invariant identity disentanglement training strategy is introduced to decouple the identity feature representation from other factors without using any of these factors’ labels. Extensive experiment results on four currently popular face datasets demonstrate the effectiveness of NRGAN on both normalized face synthesis and face recognition tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability statement

The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.

References

  1. Taigman Y, Yang M, Ranzato MA, Wolf L (2014) Deepface: closing the gap to human-level performance in face verification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1701–1708

  2. Schroff F, Kalenichenko D, Philbin J (2015) FaceNet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 815–823

  3. Li J, Zhao J, Zhao F, Liu H, Sim T (2016) Robust face recognition with deep multi-view representation learning. In: The 2016 ACM

  4. Zhao J et al (2017) Dual-agent GANs for photorealistic and identity preserving profile face synthesis. In: Neural information processing systems (NIPS)

  5. Wang H, Wang Y, Zhou Z, Ji X, Gong D, Zhou J, Li Z, Liu W (2018) CosFace: large margin cosine loss for deep face recognition. Presented at the CVPR

  6. Deng J, Guo J, Zafeiriou S (2019) ArcFace: additive angular margin loss for deep face recognition. In: Presented at the 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  7. Zhao J, Xing J, Xiong L, Yan S, Feng J (2020) Recognizing profile faces by imagining frontal view. Int J Comput Vis 128(2):460–478

    Article  MathSciNet  Google Scholar 

  8. Jian Z et al (2018) Towards pose invariant face recognition in the wild. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, Salt Lake City, UT, USA, pp 2207–2216

  9. Luan T, Yin X, Liu X (2017) Disentangled representation learning GAN for pose-invariant face recognition. In: Vision computer, recognition pattern. Honolulu , USA, HI, pp 1283–1292

  10. Xiangyu Z, Lei Z, Junjie Y, Yi D, Li SZ (2015) High-fidelity pose and expression normalization for face recognition in the wild. In: IEEE conference on Computer vision and pattern recognition (CVPR), pp 787–796

  11. Sagonas C, Panagakis Y, Zafeiriou S, Pantic M (2015) Robust statistical face frontalization. In: IEEE International conference on computer vision (ICCV), Santiago, Chile, pp 3871–3879

  12. Huang R, Zhang S, Li T, He R (2017) Beyond face rotation: global and local perception GAN for photorealistic and identity preserving frontal view synthesis. In: 2017 IEEE international conference on computer vision (ICCV), Venice, Italy, pp 2458–2467

  13. Yin X, Yu X, Sohn K, Liu X, Chandraker M (2017) Towards large-pose face frontalization in the wild. In: 2017 IEEE international conference on computer vision (ICCV), Venice, Italy, pp 4010–4019

  14. Hu Y, Wu X, Yu B, He R, Sun Z (2018) Pose-guided photorealistic face rotation. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, Salt Lake City, UT, USA, pp 8398–8406

  15. Qian Y, Deng W, Hu J (2019) Unsupervised face normalization with extreme pose and expression in the wild. In: Presented at the CVPR

  16. Zhang Z et al (2021) Semi-supervised face frontalization in the wild. IEEE Trans Inf Forensics Secur 16:909–922

    Article  Google Scholar 

  17. Ahonen T, Hadid A, Pietikainen M (2006) Face description with local binary patterns: application to face recognition. IEEE Trans Pattern Anal Mach Intell 28(12):2037–2041

    Article  Google Scholar 

  18. Tan H, Yang B, Ma Z (2013) Face recognition based on the fusion of global and local HOG features of face images. IET Comput Vis 8(3):224–234

    Article  Google Scholar 

  19. Bicego M, Lagorio A, Grosso E, Tistarelli M (2006) On the use of SIFT features for face authentication. In: 2006 Conference on computer vision and pattern recognition workshop (CVPRW’06), pp 35–35

  20. Weinberger KQ, Saul LK (2009) Distance metric learning for large margin nearest neighbor classification. J Mach Learn Res 10(1):207–244

    Google Scholar 

  21. Chen D, Cao X, Wen F, Sun J (2013) Blessing of dimensionality: high-dimensional feature and its efficient compression for face verification. In: Computer vision and pattern recognition, pp 3025–3032

  22. Sun Y, Wang X, Tang X (2013) Deep learning face representation from predicting 10,000 classes. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1891–1898

  23. Sun Y, Wang X, Tang X (2014) Deep learning face representation by joint identification-verification. arXiv preprint arXiv:1406.4773

  24. Sun Y, Liang D, Wang X, Tang X (2015) DeepID3: face recognition with very deep neural networks. Comput Sci

  25. Peng X, Yu X, Sohn K, Metaxas DN, Chandraker M (2017) Reconstruction-based disentanglement for pose-invariant face recognition. In: IEEE international conference on computer vision (ICCV), pp 1632–1641

  26. Liu W, Wen Y, Yu Z, Li M, Raj B, Song L (2017) SphereFace: deep hypersphere embedding for face recognition. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 6738–6746

  27. Huang Y et al (2020) CurricularFace: adaptive curriculum learning loss for deep face recognition. In: IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 5900–5909

  28. Bao J, Chen D, Wen F, Li H, Hua G (2018) Towards open-set identity preserving face synthesis. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, Salt Lake City, UT, USA, pp 6713–6722

  29. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: International conference on neural information processing systems, pp 2672–2680

  30. Guo Y, Zhang L, Hu Y, He X, Gao J (2016) MS-Celeb-1M: a dataset and benchmark for large-scale face recognition. In: European conference on computer vision, pp 87–102

  31. Radford A, Metz L, Chintala S (2015) Unsupervised representation learning with deep convolutional generative adversarial networks. Comput Sci 11/19

  32. Gross R, Matthews I, Cohn J, Kanade T, Baker S (2008) Multi-PIE. In: 8th IEEE international conference on automatic face and gesture recognition (FG) pp 1–8

  33. Yi D, Lei Z, Liao S, Li SZ (2014) Learning face representation from scratch. Comput Sci

  34. Huang GB, Mattar M, Berg T, Learned-Miller E (2008) Labeled faces in the wild: a database for studying face recognition in unconstrained environments. In: Workshop on faces in ‘real-life’ images: detection, alignment, and recognition

  35. Klare BF, Klein B, Taborsky E, Blanton A (2015) Pushing the frontiers of unconstrained face detection and recognition: IARPA Janus Benchmark A, pp 1931–1939

  36. Sengupta S, Chen J, Castillo C, Patel VM, Chellappa R, Jacobs DW (2016) Frontal to profile face verification in the wild. In: IEEE winter conference on applications of computer vision (WACV), pp 1–9

  37. Zhu Z, Luo P, Wang X, Tang X (2013) Deep learning identity preserving face space. In: Proceedings of the ICCV, vol 1, p 2

  38. Jourabloo A, Liu X (2017) Pose-invariant face alignment via CNN-based dense 3D model fitting. Int J Comput Vis 124:187–203

    Article  MathSciNet  Google Scholar 

  39. Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multi-task cascaded convolutional networks. IEEE Signal Process Lett (SPL) 23(10):1499–1503

    Article  Google Scholar 

  40. Kingma D, Ba J (2014) Adam: a method for stochastic optimization. Comput Sci

  41. Abadi M et al (2016) TensorFlow: a system for large-scale machine learning. arXiv:1605.08695

  42. Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S (2017) GANs trained by a two time-scale update rule converge to a local nash equilibrium. ArXiv pre-print, p 6629

  43. Zhou W, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  44. Hu C, Feng Z, Wu X, Kittler J (2020) Dual encoder–decoder based generative adversarial networks for disentangled facial representation learning. IEEE Access 8:130159–130171

    Article  Google Scholar 

  45. Yu X, Shiri F, Ghanem B, Porikli F (2020) Can we see more? Joint frontalization and hallucination of unaligned tiny faces. IEEE Trans Pattern Anal Mach Intell 42(9):2148–2164

    Article  Google Scholar 

  46. Tu X et al (2022) Joint face image restoration and frontalization for recognition. IEEE Trans Circuits Syst Video Technol 32(3):1285–1298

    Article  Google Scholar 

  47. Junho Y, Heechul J, ByungIn Y, Changkyu C, Dusik P, Junmo K (2015) Rotating your face using multi-task deep neural network. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 676–684

  48. Luan X, Zheng J, Li W (2021) Learning unsupervised face normalization through frontal view reconstruction. IEEE Trans Circuits Syst Video Technol 32:5201–5212

    Article  Google Scholar 

  49. Zhu Z, Luo P, Wang X, Tang X (2014) Multi-view perceptron: a deep model for learning face identity and view representations. In: Advances in neural information processing systems

  50. Khan A, Wahab N (2016) Deep residual learning for image recognition. In: Presented at the CVPR

  51. Hassner T, Harel S, Paz E, Enbar R (2015) Effective face frontalization in unconstrained images. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), Boston, MA, USA, pp 4295–4304

  52. Zhang Z, Chen X, Wang B, Hu G, Zuo W, Hancock ER (2019) Face frontalization using an appearance-flow-based convolutional neural network. IEEE Trans Image Process 28(5):2187–2199

    Article  MathSciNet  Google Scholar 

  53. Rong C, Zhang X, Lin Y (2020) Feature-improving generative adversarial network for face frontalization. IEEE Access 4:1–11

    Google Scholar 

  54. Masi I, Rawls S, Medioni G, Natarajan P (2016) Pose-aware face recognition in the wild. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 4838–4846

  55. Luo M, Cao J, Ma X, Zhang X, He R (2021) FA-GAN: face augmentation GAN for deformation-invariant face recognition. IEEE Trans Inf Forensics Secur 16:2341–2355

    Article  Google Scholar 

  56. Chen J, Zheng J, Patel VM, Chellappa R (2016) Fisher vector encoded deep convolutional features for unconstrained face verification. In: IEEE international conference on image processing (ICIP), pp 2981–2985

  57. Leibe B, Matas J, Sebe N, Welling M (2016) A discriminative feature learning approach for deep face recognition. In: Presented at the European conference on computer vision

  58. Wang JCF, Liu W, Liu H (2018) Additive margin softmax for face verification. IEEE Signal Process Lett 25(7):926–930

    Article  Google Scholar 

  59. Zhang L et al (2022) ARFace: attention-aware and regularization for face recognition with reinforcement learning. IEEE Trans Biom Behav Ident Sci 4(1):30–42

    Article  Google Scholar 

  60. van der Maaten L, Hinton G (2008) Viualizing data using t-SNE. J Mach Learn Res 9:2579–2605

    Google Scholar 

Download references

Funding

This research is sponsored by the Natural Science Foundation of Chongqing, China (Grant No. CSTB2022NSCQ-MSX0996), the key project of science and technology research program of Chongqing Education Commission of China (No. KJZD-K202301102), and the Natural Science Foundation of Chongqing, China (No. CSTB2023NSCQ-LZX0068).

Author information

Authors and Affiliations

  1. College of Artifcial Intelligence, Chongqing University of Technology, Chongqing, China

    Yanfei Liu

  2. Key Laboratory of Industrial Internet of Things and Networked Control, Chongqing University of Posts and Telecommunications, Chongqing, 400065, China

    Junhua Chen

  3. CloudWalk Technology Co., Ltd., Chongqing, China

    Yuanqian Li, Tianshu Wu & Hao Wen

Authors
  1. Yanfei Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Junhua Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuanqian Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tianshu Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hao Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Yanfei Liu: Conceptualization, methodology, writing—original draft preparation. Junhua chen: Software, writing—review and editing. Yuanqian Li: Data curation, validation. Tianshu Wu: Investigation. Hao Wen: Writing—review and editing.

Corresponding author

Correspondence to Junhua Chen.

Ethics declarations

Conflict of interest

We declare that we have no financial and personal relationships with other people or organizations that can inappropriately influence our work, there is no professional or other personal interest of any nature or kind in any product, service, and/or company that could be construed as influencing the position presented in, or the review of the manuscript entitled “Joint Face Normalization and Representation Learning for Face Recognition”.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, Y., Chen, J., Li, Y. et al. Joint face normalization and representation learning for face recognition. Pattern Anal Applic 27, 64 (2024). https://doi.org/10.1007/s10044-024-01255-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10044-024-01255-2

Keywords

Search

Navigation