Facial Action Transfer with Personalized Bilinear Regression

  • Dong Huang
  • Fernando De La Torre
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7573)


Facial Action Transfer (FAT) has recently attracted much attention in computer vision due to its diverse applications in the movie industry, computer games, and privacy protection. The goal of FAT is to “clone” the facial actions from the videos of one person (source) to another person (target). In this paper, we will assume that we have a video of the source person but only one frontal image of the target person. Most successful methods for FAT require a training set with annotated correspondence between expressions of different subjects, sometimes including many images of the target subject. However, labeling expressions is time consuming and error prone (i.e., it is difficult to capture the same intensity of the expression across people). Moreover, in many applications it is not realistic to have many labeled images of the target. This paper proposes a method to learn a personalized facial model, that can produce photo-realistic person-specific facial actions (e.g., synthesize wrinkles for smiling), from only a neutral image of the target person. More importantly, our learning method does not need an explicit correspondence of expressions across subjects. Experiments on the Cohn-Kanade and the RU-FACS databases show the effectiveness of our approach to generate video-realistic images of the target person driven by spontaneous facial actions of the source. Moreover, we illustrate applications of FAT to face de-identification.


Facial action transfer Bilinear regression 


  1. 1.
    Gross, R., Sweeney, L., De la Torre, F., Baker, S.: Model-based face de-identification. In: CVPR Workshop on Privacy Research in Vision (2006)Google Scholar
  2. 2.
    Frome, A., Cheung, G., Abdulkader, A., Zennaro, M.: B. Wu, A.B., Adam, H., Neven, H., Vincent, L.: Largescale privacy protection in google street view. In: ICCV (2009)Google Scholar
  3. 3.
    Senior, A. (ed.): Protecting Privacy in Video Surveillance. Springer (2009)Google Scholar
  4. 4.
    Saragih, J., Lucey, S., Cohn, J.: Real-time avatar animation from a single image. In: AFGR (2011)Google Scholar
  5. 5.
    Huang, D., De la Torre, F.: Bilinear Kernel Reduced Rank Regression for Facial Expression Synthesis. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part II. LNCS, vol. 6312, pp. 364–377. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  6. 6.
    Zhang, Q., Liu, Z., Guo, B., Shum, H.: Geometry-driven photorealistic facial expression synthesis. In: ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp. 177–186 (2003)Google Scholar
  7. 7.
    Liu, Z., Shan, Y., Zhang, Z.: Expressive expression mapping with ratio images. In: Ann. Conf. on Computer Graphics and Interactive Techniques (2001)Google Scholar
  8. 8.
    Noh, J., Neumann, U.: Expression cloning. SIGGRAPH, 277–288 (2001)Google Scholar
  9. 9.
    Chai, J., Xiao, J., Hodgins, J.: Vision-based control of 3d facial animation. In: Eurographics (2003)Google Scholar
  10. 10.
    Sumner, R., Popovic, J.: Deformation transfer for triangle meshes. SIGGRAPH 23, 399–405 (2004)CrossRefGoogle Scholar
  11. 11.
    Anguelov, D., Srinivasan, P., Koller, D., Thrun, S., Rodgers, J., Davis, J.: Scape: shape completion and animation of people. SIGGRAPH 24, 408–416 (2005)CrossRefGoogle Scholar
  12. 12.
    Vasilescu, M.A.O., Terzopoulos, D.: Multilinear Analysis of Image Ensembles: TensorFaces. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002, Part I. LNCS, vol. 2350, pp. 447–460. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  13. 13.
    Tenenbaum, J., Freeman, W.: Separating style and content with bilinear models. Neural Computation 12, 1247–1283 (2000)CrossRefGoogle Scholar
  14. 14.
    Wang, H., Ahuja, N.: Facial expression decomposition. In: ICCV (2003)Google Scholar
  15. 15.
    Chung, K.: Gross Anatomy (Board Review). Lippincott Williams & Wilkins, Hagerstown (2005)Google Scholar
  16. 16.
    De La Hunty, M., Asthana, A., Goecke, R.: Linear facial expression transfer with active appearance models. In: ICPR (2010)Google Scholar
  17. 17.
    Abboud, B., Davoine, F.: Appearance factorization for facial expression analysis. In: BMVC (2004)Google Scholar
  18. 18.
    Vlasic, D., Brand, M., Pfister, H., Popovic, J.: Face transfer with multiliner models. ACM Trans. Graphics 24, 426–433 (2005)CrossRefGoogle Scholar
  19. 19.
    Macedo, I., Brazil, E., Velho, L.: Expression transfer between photographs through multilinear aam’s. In: SIBGRAPI, pp. 239–246 (2006)Google Scholar
  20. 20.
    Elgammal, A., Lee, C.: Separating style and content on a nonlinear manifold. In: CVPR, pp. 478–485 (2004)Google Scholar
  21. 21.
    Wang, Y., Huang, X., Lee, C., Zhang, S., Li, Z., Samaras, D., Metaxas, D., Elgammal, A., Huang, P.: High resolution acquisition, learning and transfer of dynamic 3D facial expressions. Computer Graphic Forum 23, 677–686 (2004)CrossRefGoogle Scholar
  22. 22.
    Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: AFGR (2000)Google Scholar
  23. 23.
    Kim, M., Zhang, Z., De la Torre, F., Zhang, W.: Subspace regression: Predicting a subspace from one sample. In: Asian Conference on Computer Vision, ACCV (2010)Google Scholar
  24. 24.
    Bartlett, M., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., Movellan, J.: Automatic recognition of facial actions in spontaneous expressions. Journal of Multimedia 1, 22–35 (2006)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Dong Huang
    • 1
  • Fernando De La Torre
    • 1
  1. 1.Robotics InstituteCarnegie Mellon UniversityUSA

Personalised recommendations