A framework for automatic and perceptually valid facial expression generation

Abstract

Facial expressions are facial movements reflecting the internal emotional states of a character or in response to social communications. Realistic facial animation should consider at least two factors: believable visual effect and valid facial movements. However, most research tends to separate these two issues. In this paper, we present a framework for generating 3D facial expressions considering both the visual the dynamics effect. A facial expression mapping approach based on local geometry encoding is proposed, which encodes deformation in the 1-ring vector. This method is capable of mapping subtle facial movements without considering those shape and topological constraints. Facial expression mapping is achieved through three steps: correspondence establishment, deviation transfer and movement mapping. Deviation is transferred to the conformal face space through minimizing the error function. This function is formed by the source neutral and the deformed face model related by those transformation matrices in 1-ring neighborhood. The transformation matrix in 1-ring neighborhood is independent of the face shape and the mesh topology. After the facial expression mapping, dynamic parameters are then integrated with facial expressions for generating valid facial expressions. The dynamic parameters were generated based on psychophysical methods. The efficiency and effectiveness of the proposed methods have been tested using various face models with different shapes and topological representations.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

References

  1. 1.

    Asthana A, de la Hunty M, Dhall A, Goecke R (2012) Facial performance transfer via deformable models and parametric correspondence. IEEE Trans Vis Comput Graph 18(9):1511–1519, 460

    Article  Google Scholar 

  2. 2.

    Chin S, Kim K (2009) Emotional intensity-based facial expression cloning for low polygonal applications. IEEE Trans Syst Man Cybern-Part C: Appl Rev 39(3)

  3. 3.

    Choe B, Lee H, Ko H (2001) Performance-driven muscle-based facial animation. J Vis Comput Animat 12:67–79

    Article  MATH  Google Scholar 

  4. 4.

    Cosker D, Borkett R, Marshall D, Rosin PL (2008) Towards automatic performance driven animation between multiple types of facial model”. IET Comput Vis 2(3):129–141

    Article  Google Scholar 

  5. 5.

    Curio C, Breidt M, Kleiner M, Vuong Q, Giess M, Bulthoff H (2007) Semantic 3D motion retargeting for facial animation. In: Proceedings of the ACMAPGV 59–64

  6. 6.

    Deng Z, Noh JY (2007) Computer facial animation: a survey. Springer press, data-driven 3d facial animation 1–28

  7. 7.

    Do M, Carmo P (1976) Differential geometry of curves and surfaces. Prentice-hall

  8. 8.

    Ekman P, Friesen W (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, Palo Alto

    Google Scholar 

  9. 9.

    Ersotelos N, Dong F (2008) Building highly realistic facial animation: a survey. Vis Comput 24(1):13

    Article  Google Scholar 

  10. 10.

    Gao W, Chen Y, Wang R, Shan S, Jiang D (2003) Learning and synthesizing Mpeg-4 compatible 3d face animation from video sequence. IEEE Trans Circ Syst Video Technol 13(11):1119–1128

    Article  Google Scholar 

  11. 11.

    Hartley R, Zisserman A (2004) Multiple view geometry. In computer vision 2nd Ed. Cambridge University Press

  12. 12.

    Kalogerakis E, Simari P, Nowrouzezahrai E, Singh K (2007) Robust statistical estimation of curvature on discretized surfaces proceedings of the fifth eurographics symposium on geometry processing 13–22

  13. 13.

    Kalra P, Mangili A, Magnenatthalmann N, Thalmann D (1992) Simulation of facial muscle actions based on rational free form deformations. In Eurographics’92, 59–69

  14. 14.

    Lee I, Mahmood MT, Shim S, Choi T Optimizing image focus for 3D shape recovery through genetic algorithm, multimedian tools and applications, doi:10.1007/s11042-013-1433-9

  15. 15.

    Ma J, Cole R, Pellom B, Ward W, Wise B (2004) Accurate automatic visible speech synthesis of arbitrary 3d model based on concatenation of diviseme motion capture data. Comput Animat Virtual World 15:1–17

    Article  Google Scholar 

  16. 16.

    Meyer M, Desbrun M, Schröder P, Barr AH (2003) Discrete differential-geometry operators for triangulated 2-manifolds. In: Proc. of visualization and mathematics 35–57

  17. 17.

    Noh J, Neumann U (2001) Expression cloning. Proc Siggraph 277–288

  18. 18.

    Parke F (1972) Computer generated animation of faces. In Acm National Conferences Vol 1. Acm Press 451–457

  19. 19.

    Parke FI (1982) Parameterized models for facial animation. IEEE Comput Graph Appl 2(9):61–68

    Article  Google Scholar 

  20. 20.

    Parke FI, Waters K (1996) Computer facial animation. A K Peters, Wellesley

    Google Scholar 

  21. 21.

    Popovic Z,Witkin A (1999) Physically based motion transformation. In proceedings of siggraph 99 11–20

  22. 22.

    Saracchini RFV, Jorge S, Leitão HCG, Atkinson GA, Smith ML (2013) Robust 3D face capture using example-based photometric stereo. Comput Ind 64(9):1399–1410

    Article  Google Scholar 

  23. 23.

    Shin HJ, Lee Y (2009) Expression synthesis and transfer in parameter spaces. Comput Graph Forum 28(7):1829–1835

    Article  Google Scholar 

  24. 24.

    Sifakis E, Selle A, Robinson-Mosher A, Fedkiw R (2006) Simulating speech with a physics-based facial muscle model. Eurographics 261–270

  25. 25.

    Sorkine O, Cohen-Or D, Lipman Y, Alexa M, Ossl CR¨, Seidel H-P (2004) Laplacian surface editing. In Proc. of eurographics symposium on geometry processing 04; 179–188

  26. 26.

    Sumner R, P J (2004) Deformation transfer for triangle meshes. In: Proceedings of the Acm siggraph 04; 399–405

  27. 27.

    Sun Y, Chen X, Rosato M, Yin LJ (2010) Tracking vertex flow and model adaptation for three-dimensional sp2atiotemporal face analysis. IEEE Trans Syst Man Cybern Part A Syst Hum 40(3):461–474

    Article  Google Scholar 

  28. 28.

    Taubin G (2000) Geometric signal processing on polygonal meshes. In: Proc. of eurographics’2000: star-state of the art report

  29. 29.

    Terzopoulos D, And K (1990) Waters. physically-based facial modelling, analysis, and animation. J Vis Comput Animat 1:73–80

    Article  Google Scholar 

  30. 30.

    Vezzetti F, Marcolin F (2013) Geometry-based 3D face morphology analysis: soft-tissue landmark formalization, multimedia tools and applications, http://dx.doi.org/10.1007/s11042-012-1091-3

  31. 31.

    Weng Y, Xu W, Wu Y, Zhou K, Guo BN (2006) 2D shape deformation using nonlinear least squares optimization the visual computer. 22(9)

  32. 32.

    Williams L (1990) Performance-driven facial animation. In Siggraph’90, Acm press 235–242

  33. 33.

    Ying S, Peng J, Du S, Qiao H (2009) A scale stretch method based on icp for 3d data registration. IEEE Trans Autom Sci Eng 6(3):559–565

    Article  Google Scholar 

  34. 34.

    Yu H, Garrod O, Schyns P (2012) Perception-driven facial expression synthesis. Comput Graph 36(3):152–162

    Article  Google Scholar 

  35. 35.

    Zhang Q, Liu Z, Guo B, Terzopoulos D, Shum H (2006) Geometry-driven photorealistic facial expression synthesis’. IEEE Trans Vis Comput Graph 12(1):48–60

    Article  Google Scholar 

  36. 36.

    Zhang L, Snavely N, Curless B, Seitz SM (2004) Spacetime faces: high resolution capture for modeling and animation. ACM Trans Graph 23(3):548–558

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Hui Yu.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Yu, H., Garrod, O., Jack, R. et al. A framework for automatic and perceptually valid facial expression generation. Multimed Tools Appl 74, 9427–9447 (2015). https://doi.org/10.1007/s11042-014-2125-9

Download citation

Keywords

  • Facial expression mapping
  • Facial animation
  • FACS
  • Psychophysical
  • Perceptually valid
  • Face dynamics