Skip to main content
SpringerLink
Log in

Robust3D: a robust 3D face reconstruction application

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

In the process of reconstructing a historical event such as a rock concert only from video, the reconstruction of faces and expressions of the musicians is obviously important. However, in the process of rebuilding appearance, because of the low quality of the video of the recorded concert, the result of the reconstruction may be far from the real appearance. In this paper, a robust 3D face reconstruction application is described that can be applied to a video recording. The application first uses DeblurGAN program to run anti-ambiguity calculation and removes the ambiguity in the concert video. Then, the super-resolution program is used to enlarge every frame of the concert video by four times, thus making every frame of the video clearer. Finally, the 3D faces are obtained after 3D reconstruction of the processed video frames via the 3DMM_CNN program.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Zhang R, Tsai PS, James EC et al (1999) Shape from shading: a survey. IEEE Trans Pattern Anal Mach Intell 21(8):690–706

    Article  Google Scholar 

  2. Blanz V, Vetter T (1999) A morphable model for the synthesis of 3D faces. In: Proceeding of the SIGGRAPH’99. Los Angeles, USA, ACM Press, New York

  3. Blanz V, Vetter T (2003) Face recognition based on fitting a 3D morphable model. IEEE Trans Pattern Analy Mach Intell 25(9):1063–1074

    Article  Google Scholar 

  4. Blanz V, Scherbaum K, Vetter T, Seidel H (2004) Exchanging faces in images. Comput Gr Forum 23(3):669–676

    Article  Google Scholar 

  5. Blanz V, Vetter T (1999) Morphable model for the synthesis of 3D faces. In: Proceedings of the ACM SIGGRAPH conference on computer graphics

  6. Chu B, Romdhani S, Chen L (2014) 3D-aided face recognition robust to expression and pose variations. In: Proceedings of the conference on computer vision pattern recognition

  7. Paysan P, Knothe R, Amberg B, Romhani S, Vetter T (2009) A 3D face model for pose and illumination invariant face recognition. In: International conference on advanced video and signal based surveillance

  8. Romdhani S, Vetter T (2003) Efficient, robust and accurate fitting of a 3D morphable model. In: Proceedings of the international conference on computer vision

  9. Tang H, Hu Y, Fu Y, Hasegawa-Johnson M, Huang TS (2008) Real-time conversion from a single 2d face image to a 3D text-driven emotive audio-visual avatar. In: International conference on multimedia and expo, pp 1205–1208, IEEE

  10. Yang F, Wang J, Shechtman E, Bourdev L, Metaxas D (2011) Expression flow for 3D-aware face component transfer. ACM Trans Gr 30(4):60

    Article  Google Scholar 

  11. Blanz V, Romdhani S, Vetter T (2002) Face identification across different poses and illuminations with a 3d morphable model. In: International conference on automatic face and gesture recognition, pp 192–197

  12. Blanz V, Vetter T (2003) Face recognition based on fitting a 3d morphable model. Trans Pattern Anal Mach Intell 25(9):1063–1074

    Article  Google Scholar 

  13. Chu B, Romdhani S, Chen L (2014) 3D-aided face recognition robust to expression and pose variations. In: Proceedings of the conference on computer vision and pattern recognition

  14. Hu G, Yan F, Chan C-H, Deng W, Christmas W, Kittler J, Robertson NM (2016) Face recognition using a unified 3D morphable model. In: European conference on computere vision, Springer, Berlin, pp 73–89

  15. Paysan P, Knothe R, Amberg B, Romhani S, Vetter T (2009) A 3D face model for pose and illumination invariant face recognition. In: International conference on advanced video and signal based surveillance

  16. Romdhani S, Vetter T (2003) Efficient, robust and accurate fitting of a 3D morphable model. In: Proceedings of the 9th IEEE international conference on computer vision. Nice, France, IEEE Computer Society

  17. Romdhani S, Blanz V, Vetter T (2002) Face identification by fitting a 3D morphable model using linear shape and texture error functions. In: Proceedings of the ECCV’02. London, UK, Spring, Berlin, 2002, pp 3–19

  18. Jiang Dalong H, Yuxiao YS et al (2005) Efficient 3D reconstruction for face recognition. Pattern Recognit 38(6):787–798

    Article  Google Scholar 

  19. Liang S, Shapiro LG, Kemelmacher-Shlizerman I (2016) Head reconstruction from internet photos. In: European conference on computer vision, Springer, Berlin, pp 360–374

  20. Piotraschke M, Blanz V (2016) Automated 3D face reconstruction from multiple images using quality measures. In: Proceedings of the conference computer vision pattern recognition

  21. Roth J, Tong Y, Liu X (2015) Unconstrained 3D face reconstruction. In: Proceedings of the conference on computer vision pattern recognition, June

  22. Roth J, Tong Y, Liu X (2016) Adaptive 3D face reconstruction from unconstrained photo collections. In: Proceedings of the conference on computer vision pattern recognition

  23. Suwajanakorn S, Kemelmacher-Shlizerman I, Seitz SM (2014) Total moving face reconstruction. In: European conference on computer vision, Springer, Berlin, pp 796–812

  24. Hassner T (2013) Viewing real-world faces in 3D. In: Proceedings of the IEEE international conference on computer vision, IEEE, pp 3607–3614

  25. Hassner T, Harel S, Paz E, Enbar R (2015) Effective face frontalization in unconstrained images. In: Proceedings of the conference computer vision pattern recognition

  26. Masi I, Tran A, Hassner T, Leksut JT, Medioni G (2016) Do we really need to collect millions of faces for effective face recognition? In: European conference on computer vision

  27. Taigman Y, Yang M, Ranzato M, Wolf L (2014) Deepface: closing the gap to human-level performance in face verification. In: Proceedings of the conference computer vision pattern recognition, IEEE

  28. Richardson E, Sela M, Kimmel R (2016) 3d face reconstruction by learning from synthetic data. In: International conference on 3D vision

  29. Masi I, Tran A, Hassner T, Leksut JT, Medioni G (2016) Do we really need to collect millions of faces for effective face recognition? In: European conference on computer vision

  30. Kemelmacher-Shlizerman I, Basri R (2011) 3D face reconstruction from a single image using a single reference face shape. Trans Pattern Anal Mach Intell 33(2):394–405

    Article  Google Scholar 

  31. Dovgard R, Basri R (2004) Statistical symmetric shape from shading for 3D structure recovery of faces. In: European conference on computer vision, pp 99–113

  32. Hassner T (2013) Viewing real-world faces in 3D. In: Proceedings of the international conference on computer vision, IEEE, pp 3607–3614

  33. Hassner T, Basri R (2006) Example based 3D reconstruction from single 2D images. In: Proceedings of the conference on computer on vision pattern recognition workshops, IEEE

  34. Taigman Y, Yang M, Ranzato M, Wolf L (2014) Deepface: closing the gap to human-level performance in face verification. In: Proceedings of the conference on computer vision pattern recognition, IEEE

  35. Huber P, Hu G, Tena R, Mortazavian P, Koppen W, Christmas W, Rtsch M, Kittler J (2016) A multiresolution 3D morphable face model and fitting framework. In: International conference on computer vision theory and applications

  36. Jourabloo A, Liu X (2016) Large-pose face alignment via CNN-based dense 3D model fitting. In: Proceedings of the conference on computer vision pattern recognition

  37. Saito S, Li T, Li H (2016) Real-time facial segmentation and performance capture from RGB input. In: European conference on computer vision

  38. Zhu X, Lei Z, Liu X, Shi H, Li SZ (2016) Face alignment across large poses: a 3D solution. In: Proceedings of the conference on computer vision pattern recognition

  39. Kupyn O, Budzan V, Mykhailych M, Mishkin D, Matas J (2018) Deblurgan: blind motion deblurring using conditional adversarial networks. In:Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8183–8192

  40. Boracchi G, Foi A (2012) Modeling the performance of image restoration from motion blur. IEEE Trans Image Process 21(8):3502–3517

    Article  MathSciNet  Google Scholar 

  41. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  42. Kim J, Kwon Lee J, Mu Lee K (2016) Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1646–1654

  43. Tuan Tran A, Hassner T, Masi I, Medioni G (2017) Regressing robust and discriminative 3D morphable models with a very deep neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5163–5172

  44. Piotraschke M, Blanz V (2016) Automated 3d face reconstruction from multiple images using quality measures. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3418–3427

  45. Cao Z, Hidalgo G, Simon T, Wei SE, Sheikh Y (2018) OpenPose: realtime multi-person 2D pose estimation using part affinity fields. arXiv preprint arXiv:1812.08008

  46. Shi R, Gan Y, Wang Y (2018) Evaluating scalability bottlenecks by workload extrapolation. In: 2018 IEEE 26th international symposium on modeling, analysis, and simulation of computer and telecommunication systems (MASCOTS) 2018 Sep 25, IEEE, pp 333–347

  47. Shi R, Wang Y (2016) Cheap and available state machine replication. In: 2016 {USENIX} annual technical conference ({USENIX}{ATC} 16) 2016, pp 265–279

Download references

Acknowledgements

This work was funded by the European Research Council (ERC) Advanced Grant to M. Slater, MoTIVE, ERC-2016-ADG 742989.

Author information

Authors and Affiliations

  1. Qingdao University, Qingdao, China

    Zhihan Lv

  2. University of Barcelona, Barcelona, Spain

    Zhihan Lv

Authors
  1. Zhihan Lv
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhihan Lv.

Ethics declarations

Conflict of interest

The author declares that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lv, Z. Robust3D: a robust 3D face reconstruction application. Neural Comput & Applic 32, 8893–8900 (2020). https://doi.org/10.1007/s00521-019-04380-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-019-04380-w

Keywords

Search

Navigation