Video-Based Performance Driven Facial Animation

  • Fuhao Shi
Reference work entry


Video-based performance driven facial animation is appealing as it offers the lowest cost, a simplified setup, and the potential use of legacy sources and uncontrolled videos. It is also difficult as it is ill-posed due to the loss of depth. This chapter introduces techniques in video-based facial reconstruction in three levels. Given the input video, the first level is to reconstruct 3D head poses and large-scale facial deformation at each frame. Representations of the facial deformation as well as the related 2D feature detection/tracking and 3D shape parameters optimization methods are introduced. Next, we discuss methods on recovering the fine-scale surface details such as emerging and disappearing wrinkles and folds. Finally, we briefly introduce the advanced applications based on the reconstructed facial performance, such as video editing and facial component enhancement.


Performance capture Face animation Face modeling Blendshapes Shape-from-shading Face editing 


  1. Baltrušaitis T, Robinson P, Morency LP (2012) 3D constrained local model for rigid and non-rigid facial tracking. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 2610–2617Google Scholar
  2. Basri R, Jacobs DW (2003) Lambertian reflectance and linear subspaces. IEEE Trans Pattern Anal Mach Intell 25(2):218–233CrossRefGoogle Scholar
  3. Beeler T, Bickel B, Beardsley P, Sumner B, Gross M (2010) High-quality single-shot capture of facial geometry. ACM Trans Graph 29(4):40:1–40:9CrossRefGoogle Scholar
  4. Beeler T, Hahn F, Bradley D, Bickel B, Beardsley P, Gotsman C, Sumner RW, Gross M (2011) High-quality passive facial performance capture using anchor frames. ACM Trans Graph 30(4):75:1–75:10CrossRefGoogle Scholar
  5. Bickel B, Botsch M, Angst R, Matusik W, Otaduy M, Pfister H, Gross M (2007) Multi-scale capture of facial geometry and motion. ACM Trans Graph 26(3):33:1–33:10CrossRefGoogle Scholar
  6. Blanz V, Vetter T (1999) A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing, New York, pp 187–194Google Scholar
  7. Bouaziz S, Wang Y, Pauly M (2013) Online modeling for real-time facial animation. ACM Trans Graph 32(4):40:1–40:10. Scholar
  8. Bradley D, Heidrich W, Popa T, Sheffer A (2010) High resolution passive facial performance capture. ACM Trans Graph 29(4):41:1–41:10CrossRefGoogle Scholar
  9. Cao X, Wei Y, Wen F, Sun J (2012) Face alignment by explicit shape regression. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 2887–2894Google Scholar
  10. Cao C, Weng Y, Lin S, Zhou K (2013) 3D shape regression for real-time facial animation. ACM Trans Graph 32(4):41:1–41:10. Scholar
  11. Cao C, Hou Q, Zhou K (2014a) Displaced dynamic expression regression for real-time facial tracking and animation. ACM Transactions on Graphics (TOG) 33(4):43Google Scholar
  12. Cao C, Weng Y, Zhou S, Tong Y, Zhou K (2014b) Facewarehouse: a 3D facial expression database for visual computing. IEEE Trans Vis Comput Graph 20(3):413–425CrossRefGoogle Scholar
  13. Cao C, Bradley D, Zhou K, Beeler T (2015) Real-time high-fidelity facial performance capture. ACM Transactions on Graphics (TOG) 34(4):46CrossRefGoogle Scholar
  14. Chai J, Xiao J, Hodgins J (2003) Vision-based control of 3D facial animation. In: Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on computer animation, pp 193–206Google Scholar
  15. Chen YL, Wu Ht, Shi F, Tong X, Chai J (2013) Accurate and robust 3D facial capture using a single rgbd camera. In: IEEE international conference on computer vision (ICCV), pp 3615–3622Google Scholar
  16. Garrido P, Valgaerts L, Wu C, Theobalt C (2013) Reconstructing detailed dynamic face geometry from monocular video. ACM Trans Graph 32(6):158CrossRefGoogle Scholar
  17. Garrido P, Zollhöfer M, Casas D, Valgaerts L, Varanasi K, Pérez P, Theobalt C (2016) Reconstruction of personalized 3D face rigs from monocular video. ACM Trans Graph (TOG) 35(3):28CrossRefGoogle Scholar
  18. Horn BK, Brooks MJ (1989) Shape from shading. MIT Press, Cambridge, MAzbMATHGoogle Scholar
  19. Hsieh PL, Ma C, Yu J, Li H (2015) Unconstrained real-time facial performance capture. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1675–1683Google Scholar
  20. Huang H, Chai J, Tong X, Wu HT (2011) Leveraging motion capture and 3D scanning for high-fidelity facial performance acquisition. ACM Trans Graph 30(4):74:1–74:10CrossRefGoogle Scholar
  21. Ichim AE, Bouaziz S, Pauly M (2015) Dynamic 3D avatar creation from handheld video input. ACM Trans Graph (TOG) 34(4):45CrossRefGoogle Scholar
  22. Kemelmacher-Shlizerman I, Basri R (2011) 3D face reconstruction from a single image using a single reference face shape. IEEE Trans Pattern Anal Mach Intell 33(2):394–405CrossRefGoogle Scholar
  23. Li H, Yu J, Ye Y, Bregler C (2013) Real-time facial animation with on-the-fly correctives. ACM Trans Graph 32(4):42:1–42:10. Scholar
  24. Li H, Trutoiu L, Olszewski K, Wei L, Trutna T, Hsieh PL, Nicholls A, Ma C (2015) Facial performance sensing head-mounted display. ACM Trans Graph (TOG) 34(4):47Google Scholar
  25. Liu Y, Xu F, Chai J, Tong X, Wang L, Huo Q (2015) Videoaudio driven real-time facial animation. ACM Trans Graph 34(6):182:1–182:10. Scholar
  26. Ma WC, Jones A, Chiang JY, Hawkins T, Frederiksen S, Peers P, Vukovic M, Ouhyoung M, Debevec P (2008) Facial performance synthesis using deformation-driven polynomial displacement maps. ACM Trans Graph 27(5):121:1–121:10CrossRefGoogle Scholar
  27. Matthews I, Baker S (2004) Active appearance models revisited. Int J Comp Vision 60(2):135–164CrossRefGoogle Scholar
  28. Ren S, Cao X, Wei Y, Sun J (2014) Face alignment at 3000 fps via regressing local binary features. In: IEEE conference on computer vision and pattern recognition (CVPR), IEEE, pp 1685–1692Google Scholar
  29. Saragih JM, Lucey S, Cohn JF (2011) Real-time avatar animation from a single image. In: IEEE international conference on automatic face & gesture recognition and workshops (FG 2011), IEEE, pp 117–124Google Scholar
  30. Shi F, Wu HT, Tong X, Chai J (2014) Automatic acquisition of high-fidelity facial performances using monocular videos. ACM Transactions on Graphics (TOG) 33(6):222CrossRefGoogle Scholar
  31. Sumner RW, Popović J (2004) Deformation transfer for triangle meshes. ACM Transactions on Graphics (TOG) 23(3):399–405CrossRefGoogle Scholar
  32. Suwajanakorn S, Kemelmacher-Shlizerman I, Seitz SM (2014) Total moving face reconstruction. In: European conference on computer vision, Springer, pp 796–812Google Scholar
  33. Suwajanakorn S, Seitz SM, Kemelmacher-Shlizerman I (2015) What makes tom hanks look like tom hanks. In: Proceedings of the IEEE international conference on computer vision, pp 3952–3960Google Scholar
  34. Thies J, Zollhöfer M, Stamminger M, Theobalt C, Nießner M (2016) Face2face: real-time face capture and reenactment of RGB videos. In: Proceedings of computer vision and pattern recognition (CVPR), IEEE 1Google Scholar
  35. Valgaerts L, Wu C, Bruhn A, Seidel HP, Theobalt C (2012) Lightweight binocular facial performance capture under uncontrolled lighting. ACM Trans Graph 31(6):187:1–187:11. Scholar
  36. Vlasic D, Brand M, Pfister H, Popović J (2005) Face transfer with multilinear models. ACM Trans Graph (TOG) 24:426–433CrossRefGoogle Scholar
  37. Wang C, Shi F, Xia S, Chai J (2016) Real-time 3D eye gaze animation using a single RGB camera. ACM Trans Graph (TOG) 35:1Google Scholar
  38. Weise T, Li H, Van Gool L, Pauly M (2009) Face/off: live facial puppetry. In: Symposium on computer animation, pp 7–16.
  39. Weise T, Bouaziz S, Li H, Pauly M (2011) Real-time performance-based facial animation. ACM Trans Graph 30(4):77:1–77:10CrossRefGoogle Scholar
  40. Xiong X, De la Torre F (2013) Supervised descent method and its applications to face alignment. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 532–539Google Scholar
  41. Zhang L, Snavely N, Curless B, Seitz S (2004) Spacetime faces: high resolution capture for modeling and animation. ACM Transactions on Graphics 23(3):548–558CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Texas A&M UniversityCollege StationUSA

Personalised recommendations