Face Localization and Enhancement



Facial landmark localization plays a critical role in facial recognition and analysis. In this chapter, we first propose a novel cascaded backbone-branches fully convolutional neural network (BB-FCN) for rapidly and accurately localizing facial landmarks in unconstrained and cluttered settings. The proposed BB-FCN generates facial landmark response maps directly from raw images without any preprocessing. It follows a coarse-to-fine cascaded pipeline, which consists of a backbone network for roughly detecting the locations of all facial landmarks and one branch network for each type of detected landmark to further refine their locations (©[2019] IEEE. Reprinted, with permission, from [1].). At the end of this chapter, we also introduce the progress in face hallucination, a fundamental problem in the face analysis field that refers to generating a high-resolution facial image from a low-resolution input image (©[2019] IEEE. Reprinted, with permission, from [2].).


  1. 1.
    L. Liu, G. Li, Y. Xie, Y. Yu, Q. Wang, L. Lin, Facial landmark machines: a backbone-branches architecture with progressive representation learning. IEEE Trans. Multimedia. Scholar
  2. 2.
    Q. Cao, L. Lin, Y. Shi, X. Liang, G. Li, Attention-aware face hallucination via deep reinforcement learning, in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, pp. 1656–1664 (2017).
  3. 3.
    P. Luo, X. Wang, X. Tang, A deep sum-product architecture for robust facial attributes analysis, in ICCV, pp. 2864–2871 (2013)Google Scholar
  4. 4.
    C. Lu, X. Tang, Surpassing human-level face verification performance on lfw with gaussianface, in AAAI (2015)Google Scholar
  5. 5.
    L. Liu, C. Xiong, H. Zhang, Z. Niu, M. Wang, S. Yan, Deep aging face verification with large gaps. TMM 18(1), 64–75 (2016)Google Scholar
  6. 6.
    Z. Zhu, P. Luo, X. Wang, X. Tang, Deep learning identity-preserving face space, in ICCV, pp. 113–120 (2013)Google Scholar
  7. 7.
    C. Ding, D. Tao, Robust face recognition via multimodal deep face representation. TMM 17(11), 2049–2058 (2015)Google Scholar
  8. 8.
    Y. Li, L. Liu, L. Lin, Q. Wang, Face recognition by coarse-to-fine landmark regression with application to atm surveillance, in CCCV (Springer, 2017), pp. 62–73Google Scholar
  9. 9.
    P. Viola, M. Jones, Rapid object detection using a boosted cascade of simple features, in CVPR, vol. 1. IEEE, pp. I–511 (2001)Google Scholar
  10. 10.
    X. Zhu, D. Ramanan, Face detection, pose estimation, and landmark localization in the wild, in CVPR (IEEE, 2012), pp. 2879–2886Google Scholar
  11. 11.
    Z. Yan, H. Zhang, R. Piramuthu, V. Jagadeesh, D. DeCoste, W. Di, Y. Yu, Hd-cnn: hierarchical deep convolutional neural networks for large scale visual recognition, in ICCV, pp. 2740–2748 (2015)Google Scholar
  12. 12.
    M. Köstinger, P. Wohlhart, P.M. Roth, H. Bischof, Annotated facial landmarks in the wild: a large-scale, real-world database for facial landmark localization, in ICCV Workshops (IEEE, 2011), pp. 2144–2151Google Scholar
  13. 13.
    Z. Zhang, P. Luo, C.C. Loy, X. Tang, Facial landmark detection by deep multi-task learning, in ECCV (Springer, 2014), pp. 94–108Google Scholar
  14. 14.
    X. Burgos-Artizzu, P. Perona, P. Dollár, Robust face landmark estimation under occlusion, in ICCV, pp. 1513–1520 (2013)Google Scholar
  15. 15.
    X. Cao, Y. Wei, F. Wen, J. Sun, Face alignment by explicit shape regression. IJCV 107(2), 177–190 (2014)MathSciNetCrossRefGoogle Scholar
  16. 16.
    X. Yu, J. Huang, S. Zhang, W. Yan, D. Metaxas, Pose-free facial landmark fitting via optimized part mixtures and cascaded deformable shape model, in ICCV, pp. 1944–1951 (2013)Google Scholar
  17. 17.
    X. Xiong, F. Torre, Supervised descent method and its applications to face alignment, in CVPR, pp. 532–539 (2013)Google Scholar
  18. 18.
    K. Zhang, Z. Zhang, Z. Li, Y. Qiao, Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process. Lett. 23(10), 1499–1503 (2016)CrossRefGoogle Scholar
  19. 19.
    S. Xiao, J. Feng, J. Xing, H. Lai, S. Yan, A. Kassim, Robust facial landmark detection via recurrent attentive-refinement networks, in ECCV (Springer, 2016), pp. 57–72Google Scholar
  20. 20.
    Z. Liu, P. Luo, X. Wang, X. Tang, Deep learning face attributes in the wild, in ICCV, pp. 3730–3738 (2015)Google Scholar
  21. 21.
    Z. Zhang, P. Luo, C.C. Loy, X. Tang, Learning deep representation for face alignment with auxiliary attributes. IEEE Trans. Pattern Anal. Mach. Intell. 38(5), 918–930 (2016)CrossRefGoogle Scholar
  22. 22.
    E. Zhou, Z. Cao, Q. Yin, Naive-deep face recognition: touching the limit of lfw benchmark or not? arXiv preprint arXiv:1501.04690 (2015)
  23. 23.
    E. Zhou, H. Fan, Z. Cao, Y. Jiang, Q. Yin, Learning face hallucination in the wild, in AAAI, pp. 3871–3877 (2015)Google Scholar
  24. 24.
    S. Zhu, S. Liu, C.C. Loy, X. Tang, Deep cascaded bi-network for face hallucination. arXiv preprint arXiv:1607.05046 (2016)
  25. 25.
    C. Liu, H.-Y. Shum, W.T. Freeman, Face hallucination: theory and practice. Int. J. Comput. Vis. 75(1), 115–134 (2007)CrossRefGoogle Scholar
  26. 26.
    C. Dong, C.C. Loy, K. He, X. Tang, Learning a deep convolutional network for image super-resolution, in ECCV, pp. 184–199 (2014)CrossRefGoogle Scholar
  27. 27.
    J. Najemnik, W.S. Geisler, Optimal eye movement strategies in visual search. Nature 434(7031), 387–391 (2005)CrossRefGoogle Scholar
  28. 28.
    Y. Sun, D. Liang, X. Wang, X. Tang, Deepid3: face recognition with very deep neural networks. arXiv preprint arXiv:1502.00873 (2015)
  29. 29.
    J.C. Caicedo, S. Lazebnik, Active object localization with deep reinforcement learning, in ICCV, pp. 2488–2496 (2015)Google Scholar
  30. 30.
    K. Gregor, I. Danihelka, A. Graves, D.J. Rezende, D. Wierstra, DRAW: a recurrent neural network for image generation, in ICLR, pp. 1462–1471 (2015)Google Scholar
  31. 31.
    D. Silver, A. Huang, C.J. Maddison, A. Guez, L. Sifre et al., Mastering the game of go with deep neural networks and tree search. Nature 529, 484–503 (2016)CrossRefGoogle Scholar
  32. 32.
    J. Kim, J.K. Lee, K.M. Lee, Accurate image super-resolution using very deep convolutional networks (2016)Google Scholar
  33. 33.
    O. Tuzel, Y. Taguchi, J.R. Hershey, Global-local face upsampling network. arXiv preprint arXiv:1603.07235 (2016)
  34. 34.
    S. Gu, W. Zuo, Q. Xie, D. Meng, X. Feng, L. Zhang, Convolutional sparse coding for image super-resolution, in ICCV, pp. 1823–1831 (2015)Google Scholar
  35. 35.
    R.J. Williams, Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8(3), 229–256 (1992)MathSciNetzbMATHGoogle Scholar
  36. 36.
    V. Mnih, N. Heess, A. Graves, K. kavukcuoglu, Recurrent models of visual attention, in NIPS, pp. 2204–2212 (2014)Google Scholar
  37. 37.
    O. Jesorsky, K.J. Kirchberg, R. Frischholz, Robust face detection using the hausdorff distance, in AVBPA, pp. 90–95 (2001)Google Scholar
  38. 38.
    G.B. Huang, M. Ramesh, T. Berg, E. Learned-Miller, Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical Report 07-49, University of Massachusetts, Amherst, October 2007Google Scholar
  39. 39.
    G.B. Huang, V. Jain, E. Learned-Miller, Unsupervised joint alignment of complex images, in ICCV (2007)Google Scholar
  40. 40.
    C.-Y. Yang, S. Liu, M.-H. Yang, Structured face hallucination, in CVPR, pp. 1099–1106 (2013)Google Scholar
  41. 41.
    X. Ma, J. Zhang, C. Qi, Hallucinating face by position-patch. Pattern Recogn. 43(6), 2224–2236 (2010)CrossRefGoogle Scholar
  42. 42.
    D.P. Kingma, J. Ba. Adam: amethod for stochastic optimization, in ICLR (2015)Google Scholar
  43. 43.
    T. Chen, L. Lin, L. Liu, X. Luo, X. Li, Disc: deep image saliency computing via progressive representation learning. TNNLS 27(6), 1135–1149 (2016)MathSciNetGoogle Scholar
  44. 44.
    L. Liu, H. Wang, G. Li, W. Ouyang, L. Lin, Crowd counting using deep recurrent spatial-aware network, in IJCAI (2018)Google Scholar
  45. 45.
    L. Liu, R. Zhang, J. Peng, G. Li, B. Du, L. Lin, Attentive crowd flow machines, in ACM MM (ACM, 2018), pp. 1553–1561Google Scholar
  46. 46.
    Y. Sun, X. Wang, X. Tang, Deep convolutional network cascade for facial point detection, in CVPR, pp. 3476–3483 (2013)Google Scholar
  47. 47.
    R. Weng, J. Lu, Y.-P. Tan, J. Zhou, Learning cascaded deep auto-encoder networks for face alignment. TMM 18(10), 2066–2078 (2016)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2020

Authors and Affiliations

  1. 1.School of Data and Computer ScienceSun Yat-sen UniversityGuangzhouChina
  2. 2.School of Information EngineeringThe Chinese University of Hong KongHong KongHong Kong
  3. 3.School of Computer ScienceHarbin Institute of TechnologyHarbinChina

Personalised recommendations