3D Morphable Face Models and Their Applications

  • Josef Kittler
  • Patrik Huber
  • Zhen-Hua Feng
  • Guosheng Hu
  • William Christmas
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9756)

Abstract

3D Morphable Face Models (3DMM) have been used in face recognition for some time now. They can be applied in their own right as a basis for 3D face recognition and analysis involving 3D face data. However their prevalent use over the last decade has been as a versatile tool in 2D face recognition to normalise pose, illumination and expression of 2D face images. A 3DMM has the generative capacity to augment the training and test databases for various 2D face processing related tasks. It can be used to expand the gallery set for pose-invariant face matching. For any 2D face image it can furnish complementary information, in terms of its 3D face shape and texture. It can also aid multiple frame fusion by providing the means of registering a set of 2D images. A key enabling technology for this versatility is 3D face model to 2D face image fitting. In this paper recent developments in 3D face modelling and model fitting will be overviewed, and their merits in the context of diverse applications illustrated on several examples, including pose and illumination invariant face recognition, and 3D face reconstruction from video.

Keywords

3D Morphable Model 3D face reconstruction Face model fitting Applications 

References

  1. 1.
    Abiantun, R., Prabhu, U., Savvides, M.: Sparse feature extraction for pose-tolerant face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 36(10), 2061–2073 (2014)CrossRefGoogle Scholar
  2. 2.
    Aldrian, O., Smith, W.A.P.: Inverse rendering of faces with a 3D morphable model. IEEE Trans. Pattern Anal. Mach. Intell. 35(5), 1080–1093 (2013)CrossRefGoogle Scholar
  3. 3.
    Asthana, A., Marks, T.K., Jones, M.J., Tieu, K.H., Rohith, M.: Fully automatic pose-invariant face recognition via 3D pose normalization. In: IEEE International Conference on Computer Vision, (ICCV), pp. 937–944. IEEE (2011)Google Scholar
  4. 4.
    Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: the 26th Annual Conference on Computer Graphics and Interactive Techniques, (SIGGRAPH), pp. 187–194. ACM Press/Addison-Wesley Publishing Co. (1999)Google Scholar
  5. 5.
    Blanz, V., Vetter, T.: Face recognition based on fitting a 3D morphable model. IEEE Trans. Pattern Anal. Mach. Intell. 25(9), 1063–1074 (2003)CrossRefGoogle Scholar
  6. 6.
    Boom, B.J., Spreeuwers, L.J., Veldhuis, R.N.J.: Subspace-based holistic registration for low-resolution facial images. EURASIP J. Adv. Signal Process. 2010, 1–14 (2010)CrossRefGoogle Scholar
  7. 7.
    Burgos-Artizzu, X., Perona, P., Dollár, P.: Robust face landmark estimation under occlusion. In: IEEE International Conference on Computer Vision, (ICCV), pp. 1513–1520 (2013)Google Scholar
  8. 8.
    Cao, C., Bradley, D., Zhou, K., Beeler, T.: Real-time high-fidelity facial performance capture. ACM Trans. Graph. 34(4), 46:1–46:9 (2015)CrossRefGoogle Scholar
  9. 9.
    Cao, X., Wei, Y., Wen, F., Sun, J.: Face alignment by explicit shape regression. Int. J. Comput. Vis. 107(2), 177–190 (2014)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Chan, C., Tahir, M.A., Kittler, J., Pietikäinen, M.: Multiscale local phase quantization for robust component-based face recognition using kernel fusion of multiple descriptors. IEEE Trans. Pattern Anal. Mach. Intell. 35(5), 1164–1177 (2013)CrossRefGoogle Scholar
  11. 11.
    Chu, B., Romdhani, S., Chen, L.: 3D-aided face recognition robust to expression and pose variations. In: IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), pp. 1907–1914 (2014)Google Scholar
  12. 12.
    Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2001)CrossRefGoogle Scholar
  13. 13.
    Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active shape models-their training and application. Comput. Vis. Image Underst. 61(1), 38–59 (1995)CrossRefGoogle Scholar
  14. 14.
    Cristinacce, D., Cootes, T.F.: Feature detection and tracking with constrained local models. In: British Machine Vision Conference, (BMVC), pp. 929–938 (2006)Google Scholar
  15. 15.
    Dollár, P., Welinder, P., Perona, P.: Cascaded pose regression. In: IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), pp. 1078–1085 (2010)Google Scholar
  16. 16.
    Egger, B., Schönborn, S., Forster, A., Vetter, T.: Pose normalization for eye gaze estimation and facial attribute description from still images. In: Jiang, X., Hornegger, J., Koch, R. (eds.) GCPR 2014. LNCS, vol. 8753, pp. 317–327. Springer, Heidelberg (2014)Google Scholar
  17. 17.
    Feng, Z.H., Hu, G., Kittler, J., Christmas, W., Wu, X.J.: Cascaded collaborative regression for robust facial landmark detection trained using a mixture of synthetic and real images with dynamic weighting. IEEE Trans. Image Process. 24(11), 3425–3440 (2015)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Feng, Z.H., Huber, P., Kittler, J., Christmas, W., Wu, X.J.: Random cascaded-regression copse for robust facial landmark detection. IEEE Signal Process. Lett. 22(1), 76–80 (2015)CrossRefGoogle Scholar
  19. 19.
    Feng, Z.H., Kittler, J., Christmas, W., Wu, X.J., Pfeiffer, S.: Automatic face annotation by multilinear AAM with missing values. In: IEEE International Conference on Pattern Recognition, (ICPR), pp. 2586–2589. IEEE (2012)Google Scholar
  20. 20.
    Garrido, P., Valgaert, L., Wu, C., Theobalt, C.: Reconstructing detailed dynamic face geometry from monocular video. ACM Trans. Graph. 32(6), 158:1–158:10 (2013)CrossRefGoogle Scholar
  21. 21.
    Gökberk, B., Akarun, L.: Comparative analysis of decision-level fusion algorithms for 3D face recognition. In: IAPR International Conference on Pattern Recognition, (ICPR), pp. 1018–1021 (2006)Google Scholar
  22. 22.
    Gross, R., Matthews, I., Cohn, J., Kanade, T., Baker, S.: Multi-pie. Image Vis. Comput. 28(5), 807–813 (2010)CrossRefGoogle Scholar
  23. 23.
    Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2004)CrossRefMATHGoogle Scholar
  24. 24.
    Hu, G., Chan, C.H., Kittler, J., Christmas, W.: Resolution-aware 3D morphable model. In: British Machine Vision Conference, (BMVC), pp. 1–10 (2012)Google Scholar
  25. 25.
    Hu, G., Chan, C., Yan, F., Christmas, W.J., Kittler, J.: Robust face recognition by an albedo based 3D morphable model. In: IEEE International Joint Conference on Biometrics, (IJCB), pp. 1–8 (2014)Google Scholar
  26. 26.
    Hu, G., Mortazavian, P., Kittler, J., Christmas, W.J.: A facial symmetry prior for improved illumination fitting of 3D morphable model. In: International Conference on Biometrics, (ICB), pp. 1–6 (2013)Google Scholar
  27. 27.
    Huber, P., Feng, Z., Christmas, W., Kittler, J., Rätsch, M.: Fitting 3D morphable models using local features. In: IEEE International Conference on Image Processing, (ICIP) (2015). http://dx.doi.org/10.1109/ICIP.2015.7350989
  28. 28.
    Huber, P., Hu, G., Tena, R., Mortazavian, P., Koppen, W.P., Christmas, W., Rätsch, M., Kittler, J.: A multiresolution 3D morphable face model and fitting framework. In: International Conference on Computer Vision Theory and Applications (VISAPP) (2016). http://dx.doi.org/10.5220/0005669500790086
  29. 29.
    Ichim, A.E., Bouaziz, S., Pauly, M.: Dynamic 3D avatar creation from hand-held video input. ACM Trans. Graph. 34(4), 45:1–45:14 (2015)CrossRefGoogle Scholar
  30. 30.
    Jeni, L., Cohn, J., Kanade, T.: Dense 3D face alignment from 2D videos in real-time. In: 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, (FG), vol. 1, pp. 1–8 (2015)Google Scholar
  31. 31.
    Jobson, D.J., Rahman, Z.U., Woodell, G.A.: Properties and performance of a center/surround retinex. IEEE Trans. Image Process. 6(3), 451–462 (1997)CrossRefGoogle Scholar
  32. 32.
    Kakadiaris, I.A., Passalis, G., Toderici, G., Murtuza, M.N., Lu, Y., Karampatziakis, N., Theoharis, T.: Three-dimensional face recognition in the presence of facial expressions: an annotated deformable model approach. IEEE Trans. Pattern Anal. Mach. Intell. 29(4), 640–649 (2007)CrossRefGoogle Scholar
  33. 33.
    Koppen, W.P., Chan, C., Christmas, W.J., Kittler, J.: An intrinsic coordinate system for 3D face registration. In: IAPR International Conference on Pattern Recognition, (ICPR), pp. 2740–2743 (2012)Google Scholar
  34. 34.
    Lanitis, A., Taylor, C.J., Cootes, T.F.: Automatic interpretation and coding of face images using flexible models. IEEE Trans. Pattern Anal. Mach. Intell. 19(7), 743–756 (1997)CrossRefGoogle Scholar
  35. 35.
    Li, A., Shan, S., Gao, W.: Coupled bias-variance tradeoff for cross-pose face recognition. IEEE Trans. Image Process. 21(1), 305–315 (2012)MathSciNetCrossRefGoogle Scholar
  36. 36.
    Li, S., Liu, X., Chai, X., Zhang, H., Lao, S., Shan, S.: Morphable displacement field based image matching for face recognition across pose. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part I. LNCS, vol. 7572, pp. 102–115. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  37. 37.
    Little, A., Hancock, P., DeBruine, L., Jones, B.: Adaptation to antifaces and the perception of correct famous identity in an average face. Frontiers Psychol. 3, 1–9 (2012)CrossRefGoogle Scholar
  38. 38.
    Mortazavian, P., Kittler, J., Christmas, W.: 3D morphable model fitting for low-resolution facial images. In: 5th IAPR International Conference on Biometrics, (ICB), pp. 132–138. IEEE (2012)Google Scholar
  39. 39.
    Niinuma, K., Han, H., Jain, A.: Automatic multi-view face recognition via 3D model based pose regularization. In: BTAS, pp. 1–8 (2013)Google Scholar
  40. 40.
    Pan, G., Han, S., Wu, Z., Wang, Y.: 3D face recognition using mapped depth images. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, (CVPRW), p. 175 (2005)Google Scholar
  41. 41.
    Paysan, P., Knothe, R., Amberg, B., Romdhani, S., Vetter, T.: A 3D face model for pose and illumination invariant face recognition. In: IEEE International Conference on Advanced Video and Signal based Surveillance (AVSS) (2009)Google Scholar
  42. 42.
    Pepik, B., Stark, M., Gehler, P., Schiele, B.: Teaching 3D geometry to deformable part models. In: IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), pp. 3362–3369 (2012)Google Scholar
  43. 43.
    Pishchulin, L., Jain, A., Andriluka, M., Thormahlen, T., Schiele, B.: Articulated people detection and pose estimation: Reshaping the future. In: IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), pp. 3178–3185 (2012)Google Scholar
  44. 44.
    Prabhu, U., Heo, J., Savvides, M.: Unconstrained pose-invariant face recognition using 3D generic elastic models. IEEE Trans. Pattern Anal. Mach. Intell. 33(10), 1952–1961 (2011)CrossRefGoogle Scholar
  45. 45.
    Ramamoorthi, R., Hanrahan, P.: A signal-processing framework for reflection. ACM Trans. Graph. 23(4), 1004–1042 (2004)CrossRefGoogle Scholar
  46. 46.
    Rätsch, M., Huber, P., Quick, P., Frank, T., Vetter, T.: Wavelet reduced support vector regression for efficient and robust head pose estimation. In: IEEE Ninth Conference on Computer and Robot Vision (CRV), pp. 260–267 (2012). http://dx.doi.org/10.1109/CRV.2012.41
  47. 47.
    Rodriguez, J.T.: 3D Face Modelling for 2D+3D Face Recognition. Ph.D. thesis, Surrey University, Guildford, UK (2007)Google Scholar
  48. 48.
    Romdhani, S., Vetter, T.: Efficient, robust and accurate fitting of a 3D morphable model. In: IEEE International Conference on Computer Vision, (ICCV), pp. 59–66. IEEE (2003)Google Scholar
  49. 49.
    Romdhani, S., Blanz, V., Vetter, T.: Face identification by fitting a 3D morphable model using linear shape and texture error functions. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002, Part IV. LNCS, vol. 2353, pp. 3–19. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  50. 50.
    Romdhani, S., Vetter, T.: Estimating 3D shape and texture using pixel intensity, edges, specular highlights, texture constraints and a prior. In: IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), vol. 2, pp. 986–993. IEEE (2005)Google Scholar
  51. 51.
    Salah, A.A., Akarun, L.: 3D facial feature localization for registration. In: Gunsel, B., Jain, A.K., Tekalp, A.M., Sankur, B. (eds.) MRCS 2006. LNCS, vol. 4105, pp. 338–345. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  52. 52.
    Sánchez-Escobedo, D., Castelán, M., Smith, W.A.P.: Statistical 3D face shape estimation from occluding contours. Comput. Vis. Image Underst. 142, 111–124 (2016)CrossRefGoogle Scholar
  53. 53.
    Schönborn, S., Forster, A., Egger, B., Vetter, T.: A Monte Carlo strategy to integrate detection and model-based face analysis. In: Weickert, J., Hein, M., Schiele, B. (eds.) GCPR 2013. LNCS, vol. 8142, pp. 101–110. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  54. 54.
    Shen, J., Zafeiriou, S., Chrysos, G.G., Kossaifi, J., Tzimiropoulos, G., Pantic, M.: The first facial landmark tracking in-the-wild challenge: benchmark and results. In: IEEE International Conference on Computer Vision Workshop, (ICCVW), pp. 1003–1011 (2015)Google Scholar
  55. 55.
    Sim, T., Baker, S., Bsat, M.: The CMU pose, illumination, and expression database. IEEE Trans. Pattern Anal. Mach. Intell. 25, 1615–1618 (2003)CrossRefGoogle Scholar
  56. 56.
    Tena, J.R., Smith, R.S., Hamouz, M., Kittler, J., Hilton, A., Illingworth, J.: 2D face pose normalisation using a 3D morphable model. In: IEEE Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 51–56, September 2007Google Scholar
  57. 57.
    Walker, M., Jiang, F., Vetter, T., Sczesny, S.: Universals and cultural differences in forming personality trait judgments from faces. Soc. Psychol. Pers. Sci. 2, 609–617 (2011)CrossRefGoogle Scholar
  58. 58.
    Walker, M., Vetter, T.: Portraits made to measure: manipulating social judgments about individuals with a statistical face model. J. Vis. 9, 1–13 (2009)CrossRefGoogle Scholar
  59. 59.
    Walker, M., Vetter, T.: Changing the personality of a face: perceived big two and big five personality factors modeled in real photographs. J. Pers. Soc. Psychol. 110, 609–624 (2015). http://dx.doi.org/10.1037/pspp0000064 CrossRefGoogle Scholar
  60. 60.
    Wang, Y., Liu, Z., Hua, G., Wen, Z., Zhang, Z., Samaras, D.: Face re-lighting from a single image under harsh lighting conditions. In: IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), pp. 1–8. IEEE (2007)Google Scholar
  61. 61.
    Xie, X., Zheng, W.S., Lai, J., Yuen, P.C., Suen, C.Y.: Normalization of face illumination based on large-and small-scale features. IEEE Trans. Image Process. 20(7), 1807–1821 (2011)MathSciNetCrossRefGoogle Scholar
  62. 62.
    Xiong, X., De la Torre, F.: Supervised descent method and its applications to face alignment. In: IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), pp. 532–539 (2013)Google Scholar
  63. 63.
    Xu, C., Wang, Y., Tan, T., Quan, L.: Automatic 3D face recognition combining global geometric features with local shape variation information. In: IAPR International Conference on Pattern Recognition, (ICPR), pp. 308–313 (2004)Google Scholar
  64. 64.
    Yan, J., Lei, Z., Yi, D., Li, S.Z.: Learn to combine multiple hypotheses for accurate face alignment. In: IEEE International Conference on Computer Vision Workshops, (ICCVW), pp. 392–396, December 2013Google Scholar
  65. 65.
    Yi, D., Lei, Z., Li, S.Z.: Towards pose robust face recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), pp. 3539–3545 (2013)Google Scholar
  66. 66.
    Zhang, L., Samaras, D.: Face recognition from a single training image under arbitrary unknown lighting using spherical harmonics. IEEE Trans. Pattern Anal. Mach. Intell. 28(3), 351–363 (2006)CrossRefGoogle Scholar
  67. 67.
    Zhang, W., Shan, S., Gao, W., Chen, X., Zhang, H.: Local gabor binary pattern histogram sequence (lgbphs): a novel non-statistical model for face representation and recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, (CVPR). vol. 1, pp. 786–791 (2005)Google Scholar
  68. 68.
    Zhang, X., Gao, Y., Leung, M.K.H.: Recognizing rotated faces from frontal and side views: An approach toward effective use of mugshot databases. IEEE Trans. Inf. Forensics Secur. 3(4), 684–697 (2008)CrossRefGoogle Scholar
  69. 69.
    Zhu, X., Yan, J., Yi, D., Lei, Z., Li, S.Z.: Discriminative 3D morphable model fitting. In: International Conference on Automatic Face and Gesture Recognition (FG) (2015)Google Scholar
  70. 70.
    Zhu, Z., Luo, P., Wang, X., Tang, X.: Deep learning identity preserving face space. In: International Conference on Computer Vision, (ICCV), vol. 1, p. 2 (2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Josef Kittler
    • 1
  • Patrik Huber
    • 1
  • Zhen-Hua Feng
    • 1
  • Guosheng Hu
    • 2
  • William Christmas
    • 1
  1. 1.Centre for Vision, Speech and Signal ProcessingUniversity of SurreyGuildfordUK
  2. 2.LEAR TeamInria Grenoble Rhone-AlpesGrenobleFrance

Personalised recommendations