Advertisement

Advances and Trends in Video Face Alignment

  • Gang ZhangEmail author
  • Yuding Ke
  • Weikang Zhang
  • M. Hassaballah
Chapter
Part of the Studies in Computational Intelligence book series (SCI, volume 804)

Abstract

Face alignment in a video is an important research area in computer vision and can provides strong support for video face recognition, face animation, etc. It is different from face alignment in a single image where each face is regarded as an independent individual. For the latter, lack of amount of information makes the face alignment an under-determined problem although good results have been obtained by using prior information and auxiliary models. For the former, temporal and spatial relations are among faces in a video. These relations can impose constraints among multiple face images each other and help to improve alignment performance. In the chapter, definition of face alignment in a video and its significance are described. Methods for face alignment in a video are divided into three kinds: face alignment using image alignment algorithms, joint alignment of face images, and face alignment using temporal and spatial continuities. The first kind of face alignment is studied and some of surveys have described the work. The chapter will mainly focus on joint face alignment and face alignment using temporal and spatial continuities. Herein, some representative methods are described, and some factors influencing alignment performance are analyzed. Then the state-of-the-art methods are described and the future trends of face alignment in a video are discussed.

Notes

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant 61372176. It was also supported by the Liaoning Province Science and Technology Department of China under Grant 201602552.

References

  1. 1.
    Gang, Z., Jingsheng, C., Ya, S., Hassaballah, M., Lianqiang, N.: Advances in Video Face Recognition. Science Press, China (2018). ISBN 9787030538468Google Scholar
  2. 2.
    Hassaballah, M., Saleh, A.: Face recognition: Challenges, achievements and future directions. IET Comput. Vis. J. 9(4), 614–626 (2015)CrossRefGoogle Scholar
  3. 3.
    Shan, S.G., Gao, W., Chang, Y.Z., Cao, B., Chen, X.L.: Curse of mis-alignment problem in face recognition. Chin. J. Comput. 28(5), 782–791 (2005)Google Scholar
  4. 4.
    Wagner, A., Wright, J., Ganesh, A., Zhou, Z.H., Mobahi, H., Ma, Y.: Towards a practical face recognition system: robust alignment and illumination by sparse representation. IEEE Trans. Pattern Anal. Mach. Intell. 34(2), 372–386 (2012)CrossRefGoogle Scholar
  5. 5.
    Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: International Joint Conference on Artificial Intelligence, pp. 674–679. Vancouver, Can (1981)Google Scholar
  6. 6.
    Hager, G.D., Belhumeur, P.N.: Efficient region tracking with parametric models of geometry and illumination. IEEE Trans. Pattern Anal. Mach. Intell. 20(10), 1025–1039 (1998)CrossRefGoogle Scholar
  7. 7.
    Brown, L.G.: A survey of image registration techniques. ACM Comput. Surv. 24(4), 325–376 (1992)CrossRefGoogle Scholar
  8. 8.
    Zitová, B., Flusser, J.: Image registration methods: a survey. Image Vis. Comput. 21(11), 977–1000 (2003)CrossRefGoogle Scholar
  9. 9.
    Jin, X., Tan, X.Y.: Face alignment in-the-wild: a survey. Comput. Vis. Image Underst. 162, 1–22 (2017)CrossRefGoogle Scholar
  10. 10.
    Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2001)CrossRefGoogle Scholar
  11. 11.
    Cristinacce, D., Cootes, T.: Automatic feature localization with constrained local models. Pattern Recogn. 41(10), 3054–3067 (2008)CrossRefGoogle Scholar
  12. 12.
    Gao, X.B., Su, Y., Li, X.L., Tao, D.C.: A review of active appearance models. IEEE Trans. Syst. Man Cybern. Part C-Appl. Rev. 40(2), 145–158 (2010)CrossRefGoogle Scholar
  13. 13.
    Xing, J.L., Niu, Z.H., Huang, J.S.: Towards robust and accurate multi-view and partially-occluded face alignment. IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 987–1001 (2018)CrossRefGoogle Scholar
  14. 14.
    Tzimiropoulos, G., Pantic, M.: Fast algorithms for fitting active appearance models to unconstrained images. Int. J. Comput. Vis. 122(1), 17–33 (2017)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Saragih, J.M., Lucey, S., Cohn, J.E.: Deformable model fitting by regularized landmark mean-shift. Int. J. Comput. Vis. 91(2), 200–215 (2011)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Felzensawalb, P.F., Huttenlocher, D.P.: Pictorial structures for object recognition. Int. J. Comput. Vis. 61(1), 55–79 (2005)CrossRefGoogle Scholar
  17. 17.
    Zhou, F., Brandt, J., Lin, Z.: Exemplar-based graph matching for robust facial landmark localization. In: IEEE International Conference on Computer Vision, pp. 1025–1032. Sydney, Australia, 1–8 December 2013Google Scholar
  18. 18.
    Li, H.S., Huang, X.L., He, L.: Object matching using a locally affine invariant and linear programming techniques. IEEE Trans. Pattern Anal. Mach. Intell. 35(2), 411–424 (2013)CrossRefGoogle Scholar
  19. 19.
    Zhu, X.X., Ramanan, D.: Face detection, pose estimation, and landmark localization in the wild. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2879–2886. Providence, USA, 16–21 June 2012Google Scholar
  20. 20.
    Belhumeur, P.N., Jacobs, D.W., Kriegman, D.J., Kumar, N.: Localizing parts of faces using a consensus of exemplars. IEEE Trans. Pattern Anal. Mach. Intell. 35(12), 2930–2940 (2013)CrossRefGoogle Scholar
  21. 21.
    Learned-Miller, E.G.: Data driven image models through continuous joint alignment. IEEE Trans. Pattern Anal. Mach. Intell. 28(2), 236–250 (2006)CrossRefGoogle Scholar
  22. 22.
    Gross, R., Matthews, I., Baker, S.: Generic vs. person specific active appearance models. Image Vis. Comput. 23(12), 1080–1093 (2005)CrossRefGoogle Scholar
  23. 23.
    Cootes, T.F., Twining, C.J., Petrovic, V.S., Babalola, K.O., Taylor, C.J.: Computing accurate correspondences across groups of images. IEEE Trans. Pattern Anal. Mach. Intell. 32(11), 1994–2005 (2010)CrossRefGoogle Scholar
  24. 24.
    Marsland, S., Twining, C.J., Taylor, C.J.: A minimum description length objective function for groupwise non-rigid image registration. Image Vis. Comput. 26(3), 333–346 (2008)CrossRefGoogle Scholar
  25. 25.
    Sidorov, K.A., Richmond, S., Marshall, D.: Efficient groupwise non-rigid registration of textured surfaces. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2401–2408. Colorado Springs, USA, 20–25 June 2011Google Scholar
  26. 26.
    Basri, R., Jacobs, D.W.: Lambertian reflectance and linear subspaces. IEEE Trans. Pattern Anal. Mach. Intell. 25(2), 218–233 (2003)CrossRefGoogle Scholar
  27. 27.
    Zhao, C., Cham, W.K., Wang, X.G.: Joint face alignment with a generic deformable face model. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 561–568. Colorado Springs, USA, 20–25 June 2011Google Scholar
  28. 28.
    Smith, B.M., Zhang, L.: Joint face alignment with non-parametric shape models. In: European Conference on Computer Vision, pp. 43–56. Florence, Italy, 7–13 October 2012CrossRefGoogle Scholar
  29. 29.
    Irani, M., Peleg, S.: Super resolution from image sequences. In: International Conference on Pattern Recognition, pp. 115–120. Atlantic City, USA, 16–21 June 1990Google Scholar
  30. 30.
    Horn, B.K.P., Schunck, B.G.: Determining optical flow. Artif. Intell. 17, 185–203 (1981)CrossRefGoogle Scholar
  31. 31.
    Liu, C., Yuen, J., Torralba, A.: Sift flow: dense correspondence across scenes and its applications. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 978–994 (2011)CrossRefGoogle Scholar
  32. 32.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Dowson, N.D.H., Bowden, R.: Simultaneous modeling and tracking (SMAT) of feature sets. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 99–105. San Diego, USA, 20–25 June 2005Google Scholar
  34. 34.
    Sung, J., Kanade, T., Kim, D.: Pose robust face tracking by combining active appearance models and cylinder head models. Int. J. Comput. Vis. 80(2), 260–274 (2008)CrossRefGoogle Scholar
  35. 35.
    Kahraman, F., Gokmen, M., Darkner, S., Larsen, R.: An active illumination and appearance (AIA) model for face alignment. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3616–3622. Minneapolis, USA, 17–22 June 2007Google Scholar
  36. 36.
    Roh, M.C., Oguri, T., Kanade, T.: Face alignment robust to occlusion. In: IEEE International Conference on Automatic Face & Gesture Recognition, pp. 239–244. Santa Barbara, USA, 21–25 March 2011Google Scholar
  37. 37.
    Dantone, M., Gall, J., Fanelli, G., Gool, L.V.: Real-time facial feature detection using conditional regression forests. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2578–2585. Providence, USA, 16–21 June 2012Google Scholar
  38. 38.
    Cao, X., Wei, Y., Wen, F., Sun, J.: Face alignment by explicit shape regression. Int. J. Comput. Vis. 107(2), 177–190 (2014)MathSciNetCrossRefGoogle Scholar
  39. 39.
    Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1867–1874. Columbus, USA, 23–28 June 2014Google Scholar
  40. 40.
    Ren, S.Q., Cao, X.D., Wei, Y.C., Sun, J.: Face alignment at 3000 fps via regressing local binary features. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1685–1692. Columbus, USA, 23–28 June 2014Google Scholar
  41. 41.
    Lee, H.S., Kim, D.: Tensor-based AAM with continuous variation estimation: application to variation-robust face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31(6), 1102–1116 (2009)CrossRefGoogle Scholar
  42. 42.
    Shi, B.G., Bai, X., Liu, W.Y., Wang, J.D.: Face alignment with deep regression. IEEE Trans. Neural Netw. Learn. Syst. 29(1), 183–194 (2018)MathSciNetCrossRefGoogle Scholar
  43. 43.
    Lv, J.J., Shao, X.H., Xing, J.L., Cheng, C., Zhou, X.: A deep regression architecture with two-stage re-initialization for high performance facial landmark detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3691–3700. Honolulu, USA, 21–26 July 2017Google Scholar
  44. 44.
    Dibeklioglu, H., Salah, A.A., Gevers, T.: A statistical method for 2-D facial landmarking. IEEE Trans. Image Process. 21(2), 844–858 (2012)MathSciNetCrossRefGoogle Scholar
  45. 45.
    Hu, C.B., Xiao, J., Matthews, I., Baker, S., Cohn, J., Kanade, T.: Fitting a single active appearance model simultaneously to multiple images. In: British Machine Vision Conference, pp. 437–446. London, UK, 7–9 September 2004Google Scholar
  46. 46.
    Su, Y.C., Ai, H.Z., Lao, S.H.: Multi-view face alignment using 3D shape model for view estimation. In: 3rd IAPR/IEEE International Conference on Advances in Biometrics, pp. 179–188. Alghero, Italy, 2–5 June 2009CrossRefGoogle Scholar
  47. 47.
    Anderson, R., Stenger, B., Cipolla, R.: Using bounded diameter minimum spanning trees to build dense active appearance models. Int. J. Comput. Vis. 110(1), 48–57 (2014)CrossRefGoogle Scholar
  48. 48.
    Bolkart, T., Wuhrer, S.: A groupwise multilinear correspondence optimization for 3D faces. In: IEEE International Conference on Computer Vision, pp. 3604–3612. Santiago, Chile, 11–18 December 2015Google Scholar
  49. 49.
    Liu, D., Nocedal, J.: On the limited memory method for large scale optimization. Math. Prog. Ser. A B 45(1), 503–528 (1989)MathSciNetCrossRefGoogle Scholar
  50. 50.
    Matthews, I., Ishikawa, T., Baker, S.: The template update problem. IEEE Trans. Pattern Anal. Mach. Intell. 26(6), 810–815 (2004)CrossRefGoogle Scholar
  51. 51.
    Huang, C., Ding, X.Q., Fang, C.: Pose robust face tracking by combing view-based AAMs and temporal filters. Comput. Vis. Image Underst. 116(7), 777–792 (2012)CrossRefGoogle Scholar
  52. 52.
    Liu, X.M.: Video-based face model fitting using adaptive active appearance model. Image Vis. Comput. 28(7), 1162–1172 (2010)CrossRefGoogle Scholar
  53. 53.
    Zhang G., Tang S.K., Li J.Q.: Face landmark point tracking using LK pyramid optical flow. In: Tenth International Conference on Machine Vision. Vienna, Austria, 13–15 November 2018Google Scholar
  54. 54.
    Hassaballah, M., Bekhet, S., Amal A.M.R., Gang, Z.: Facial features detection and localization. In: Recent Advances in Computer Vision—Theories and Applications. Studies in Computational Intelligence Series, Springer, 2019Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Gang Zhang
    • 1
    • 2
    Email author
  • Yuding Ke
    • 2
  • Weikang Zhang
    • 2
  • M. Hassaballah
    • 3
  1. 1.School of SoftwareShenyang University of TechnologyShenyangChina
  2. 2.School of Information Science and EngineeringShenyang University of TechnologyShenyangChina
  3. 3.Faculty of Computers and Information, Computer Science DepartmentSouth Valley UniversityLuxorEgypt

Personalised recommendations