Automatic 3D Facial Expression Analysis in Videos

  • Ya Chang
  • Marcelo Vieira
  • Matthew Turk
  • Luiz Velho
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3723)


We introduce a novel framework for automatic 3D facial expression analysis in videos. Preliminary results demonstrate editing facial expression with facial expression recognition. We first build a 3D expression database to learn the expression space of a human face. The real-time 3D video data were captured by a camera/projector scanning system. From this database, we extract the geometry deformation independent of pose and illumination changes. All possible facial deformations of an individual make a nonlinear manifold embedded in a high dimensional space. To combine the manifolds of different subjects that vary significantly and are usually hard to align, we transfer the facial deformations in all training videos to one standard model. Lipschitz embedding embeds the normalized deformation of the standard model in a low dimensional generalized manifold. We learn a probabilistic expression model on the generalized manifold. To edit a facial expression of a new subject in 3D videos, the system searches over this generalized manifold for optimal replacement with the ‘target’ expression, which will be blended with the deformation in the previous frames to synthesize images of the new expression with the current head pose. Experimental results show that our method works effectively.


Facial Expression Facial Expression Recognition Training Video Generalize Manifold Deformation Vector 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Blanz, V., Vetter, T.: A Morphable Model for the Synthesis of 3D Face. In: ACM SIGGRAPH, Los Angeles, CA, pp. 187–194 (1999)Google Scholar
  2. 2.
    Zhang, Y., Prakash, E.C., Sung, E.: A New Physical Model with Multi-layer Architecture for Facial Expression Animation Using Dynamic Adaptive Mesh. IEEE Transactions on Visualization and Computer Graphics 10(3), 339–352 (2004)CrossRefGoogle Scholar
  3. 3.
    Wang, Y., Huang, X., Lee, C., Zhang, S., Li, Z., Samaras, D., Metaxas, D., Elgammal, A., Huang, P.: High Resolution Acquisition, Learning and Transfer of Dynamic 3-D Facial Expressions. In: Proc. Eurographics 2004, Grenoble, France (2004)Google Scholar
  4. 4.
    Vieira, M.B., Velho, L., Sá, A., Carvalho, P.C.: A Camera-Projector System for Real-Time 3D Video. In: IEEE Int. Workshop on Projector-Camera Systems, San Diego, CA (2005)Google Scholar
  5. 5.
    Roweis, S., Saul, L.: Nonlinear Dimensionality Reduction by Locally Linear Embedding. Science 290, 2323–2326 (2000)CrossRefGoogle Scholar
  6. 6.
    Tenenbaum, J.B., Silva, V., de Langford, J.C.: A Global Geometric Framework For Nonlinear Dimensionality Reduction. Science 290, 2319–2323 (2000)CrossRefGoogle Scholar
  7. 7.
    Tenebaum, J.B., Freeman, W.T.: Separating Style and Content with Bilinear Models. Neural Computation J. 12, 1247–1283 (1999)CrossRefGoogle Scholar
  8. 8.
    Vasilescu, A.O., Terzopoulos, D.: Multilinear Subspace Analysis for Image Ensembles. In: Proc. Computer Vision and Pattern Recognition, Madison, WI (2003)Google Scholar
  9. 9.
    Elgammal, A., Lee, C.: Separating Style and Content on a Nonlinear Manifold. In: Proc. Computer Vision and Pattern Recognition, Washington (2004)Google Scholar
  10. 10.
    Sumner, R., Popovic, J.: Deformation Transfer for Triangle Meshes. In: ACM SIGGRAPH, Los Angeles, CA (2004)Google Scholar
  11. 11.
    Chang, Y., Hu, C., Turk, M.: Probabilistic Expression Analysis on Manifolds. In: Proc. Computer Vision and Pattern Recognition, Washington (2004)Google Scholar
  12. 12.
    Vacchetti, L., Lepetit, V., Fua, P.: Stable Real-time 3D Tracking Using Online and Offline Information. IEEE Trans. on Pattern Analysis and Machine Intelligence 26, 1385–1391 (2004)CrossRefGoogle Scholar
  13. 13.
    Goldenstein, S.K., Vogler, C., Metaxas, D.: Statistical Cue Integration in DAG Deformable Models. IEEE Trans. on Pattern Analysis and Machine Intelligence 25, 801–813 (2003)CrossRefGoogle Scholar
  14. 14.
    Zhang, Z., Lyons, M., Schuster, M., Akamatsu, S.: Comparison Between Geometry-based and Gabor-wavelets-based Facial Expression Recognition Using Multi-layer Perceptron: IEEE Conf. on Automatic Face and Gesture Recognition, Nara, Japan (1998)Google Scholar
  15. 15.
    Chuang, E., Deshpande, H., Bregler, C.: Facial Expression Space Learning. Pacific Graphics (2002)Google Scholar
  16. 16.
    Kanade, T., Cohn, J., Tian, Y.: Comprehensive Database for Facial Expression Analysis. In: IEEE Conf. on Automatic Face and Gesture Recognition, pp. 46–53 (2000)Google Scholar
  17. 17.
    Sim, T., Baker, S., Bsat, M.: The CMU Pose, Illumination, and Expression Database. IEEE Trans. on Pattern Analysis and Machine Intelligence 25, 1615–1618 (2003)CrossRefGoogle Scholar
  18. 18.
    Noh, J., Neumann, U.: Expression Cloning. In: ACM SIGGRAPH, Los Angeles, CA (2001)Google Scholar
  19. 19.
    Lee, J., Shin, S.Y.: A Hierarchical Approach to Interactive Motion Editing for Human-like Figures. In: ACM SIGGRAPH, Los Angeles, CA, pp. 39–48 (1999)Google Scholar
  20. 20.
    Zhang, Q., Liu, Z., Guo, B., Shum, H.: Geometry-Driven Photorealistic Facial Expression Synthesis. In: SIGGRAPH Symposium on Computer Animation (2003)Google Scholar
  21. 21.
    Pyun, H., Kim, Y., Chae, W., Kang, H.W., Shin, S.Y.: An Example-Based Approach for Facial Expression Cloning. In: Siggraph Symposium on Computer Animation (2003)Google Scholar
  22. 22.
    Sá, A., Carvalho, P.C., Velho, L.: (b,s)-BCSL: Structured Light Color Boundary Coding for 3D photography. In: Int. Fall Workshop on Vision, Modeling, and Visualization (2002). Google Scholar
  23. 23.
  24. 24.
    Sederberg, T.W., Parry, S.R.: Free-Form Deformation of Solid Geometric Models. In: ACM SIGGRAPH, Dallas, TX, pp. 151-159 (1986)Google Scholar
  25. 25.
    Bourgain, J.: On Lipschitz Embedding of Finite Metric Spaces in Hilbert Space. Israel J. Math. 52, 46–52 (1985)zbMATHCrossRefMathSciNetGoogle Scholar
  26. 26.
    Blanz, V., Scherbaum, K., Vetter, T., Seidel, H.: Exchanging Faces in Images. In: Proc. Eurographics, Grenoble, France (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Ya Chang
    • 1
  • Marcelo Vieira
    • 2
  • Matthew Turk
    • 1
  • Luiz Velho
    • 2
  1. 1.Computer Science DepartmentUniversity of CaliforniaSanta Barbara
  2. 2.Instituto de Matemática Pura e AplicadaRio de JaneiroBrazil

Personalised recommendations