Skip to main content
SpringerLink
Log in

Video error concealment through 3-D face model

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

All current state-of-the-art video error concealment schemes conceal the lost area through the reconstruction of 2-D patches. Reconstructed corrupted areas in the facial parts of head-and-shoulder video sequences, as in video conferencing applications, often suffer from objectionable artifacts. In this work, we present a novel video error concealment technique, which is assisted by Candide-3, a standard 3-D head-and-shoulder face model, for the reconstruction of corrupted facial regions with reduced artifact. The model is first adapted to facial images and then updated and tracked across frames, even in presence of lost macroblocks. The lost portions of the face are reconstructed through the projection of the adapted 3-D face model. The proposed concealment scheme has been experimented on sequences having facial areas such as Foreman, Carphone, News etc. and it outperforms some of the recently developed 2-D concealment schemes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Ahlberg J (2001) CANDIDE-3 – An updated parameterized face. Report No. LiTH-ISY-R-2326, Department of Electrical Engineering, Linköping University, Sweden

  2. Arulampalam MS, Maskell S, Gordon N, Clapp T (2002) A tutorial on particle filters for online nonlinear/non-gaussian bayesian tracking. IEEE Trans Signal Process 50(2):174–188

    Article  Google Scholar 

  3. Asheri H, Rabiee HR, Pourdamghani N, Ghanbari M (2012) Multi-directional spatial error concealment using adaptive edge thresholding. IEEE Trans Consum Electron 58(3):880–885

    Article  Google Scholar 

  4. Blanz V, Vetter T (1999) A morphable model for the synthesis of 3D faces. In: 26th annual conference on computer graphics and interactive techniques, New York, NY, USA, pp 187–194

  5. Chakraborti T, Sengupta A, Midya A, Konar A, Sengupta S (2013) 3-D model assisted facial error concealment technique using regenerative particle filter based tracking. In: IEEE international conference on multimedia and expo workshops (ICMEW), San Jose, CA, USA, pp 1–6

  6. Chen Y, Hu Y, Au OC, Li H, Chen CW (2008) Video error concealment using spatio-temporal boundary matching and partial differential equation. IEEE Trans Multimedia 10(1):2–15

    Article  Google Scholar 

  7. Cui S, Cui H, Tang K (2013) Error concealment via kalman filter for heavily corrupted videos in H.264/AVC. Signal Process Image Commun 28(5):430–440

    Article  Google Scholar 

  8. Dufaux F, Ebrahimi T (2004) Error-resilient video coding performance analysis of motion JPEG-2000 and MPEG-4. In: SPIE visual communications and image processing, vol 5308, San Jose, CA, USA, pp 596–607

  9. Feamster N, Balakrishnan H (2002) Packet loss recovery for streaming video. In: IEEE international packet video workshop. Pittsburg, Pennsylvania, USA

  10. Hartanto F, Sirisena HR (1999) Hybrid error control mechanism for video transmission in the wireless IP networks. In: 10th IEEE workshop on local and metropolitan area networks (LANMAN), Sydney, New South Wales, Australia, pp 126–132

  11. Joint Model The JVT reference software for H.264/MPEG-4 AVC. [online] Available: http://iphome.hhi.de/suehring/tml/index.htm

  12. Kalman RE (1960) A new approach to linear filtering and prediction problems. Trans ASME J Basic Eng 82(Series D):35–45

    Article  Google Scholar 

  13. Kampmann M (2002) Automatic 3-D face model adaptation for model-based coding of videophone sequences. IEEE Trans Circuits Syst Video Technol 12(3):172–182

    Article  Google Scholar 

  14. Kim W, Koo J, Jeong J (2006) Fine directional interpolation for spatial error concealment. IEEE Trans Consum Electron 52(3):1050–1056

    Article  Google Scholar 

  15. Kui WY, Hannuksela MM, Varsa V, Hourunranta A, Gabbouj M (2002) The error concealment feature in the H.26L test model. In: International conference on image processing (ICIP), vol 2, New York, USA, pp II 729–II 732

  16. Kumwilaisak W, Kuo CCJ (2011) Spatial error concealment with sequence-aligned texture modeling and adaptive directional recovery. J Vis Commun Image Represent 22(2):164–177

    Article  Google Scholar 

  17. Kung WY, Kim CS, Kuo CCJ (2006) Spatial and temporal error concealment techniques for video transmission over noisy channels. IEEE Trans Circuits Syst Video Technol 16(7):789–802

    Article  Google Scholar 

  18. Lee PJ, Kuo KT (2016) An adaptive error concealment method for depth map in 3D video coding. Int J Fuzzy Syst 18(1):62–71

    Article  MathSciNet  Google Scholar 

  19. Lie WN, Lin GH (2015) Error concealment for the transmission of H.264/AVC-compressed 3D video in color plus depth format. J Vis Commun Image Represent 32:237–245

    Article  Google Scholar 

  20. Lin TL, Chang TE, Huang GX, Chou CC, Thakur US (2014) Improved interview video error concealment on whole frame packet loss. J Vis Commun Image Represent 25(8):1811–1822

    Article  Google Scholar 

  21. Ma M, Au OC, Chan SHG, Sun MT (2010) Edge-directed error concealment. IEEE Trans Circuits Syst Video Technol 20(3):382–395

    Article  Google Scholar 

  22. Midya A, Ranjan R, Sengupta S (2014) Scene content driven FEC allocation for video streaming. Signal Process Image Commun 29(1):37–48

    Article  Google Scholar 

  23. Midya A, Sengupta S (2011) Hybrid temporal/spatial error concealment strategy robust to scene transitions. In: IEEE pacific rim conference on communications, computers and signal processing (PACRIM), Victoria, BC, Canada, pp 416–421

  24. Midya A, Sengupta S (2015) Switchable video error concealment using encoder driven scene transition detection and edge preserving SEC. Multimedia Tools and Applications 74(6):2033–2054

    Article  Google Scholar 

  25. Parke FI (1974) A parametric model of human faces. Ph.D. thesis, University of Utah, Salt Lake City, UT, USA

  26. Persson D, Eriksson T (2008) Packet video error concealment with gaussian mixture models. IEEE Trans Image Process 17(2):145–154

    Article  MathSciNet  Google Scholar 

  27. Pyun JY (2008) Error concealment aware streaming video system over packet-based mobile networks. IEEE Trans Consum Electron 54(4):1708–1713

    Article  Google Scholar 

  28. Pyun JY, Lee JS, Jeong JW, Jeong JH, Ko SJ (2003) Robust error concealment for visual communications in burst-packet-loss networks. IEEE Trans Consum Electron 49:1013–1019

    Article  Google Scholar 

  29. Turaga DS, Chen T (2002) Model-based error concealment for wireless video. IEEE Trans Circuits Syst Video Technol 12(6):483–495

    Article  Google Scholar 

  30. Video Quality Metric (VQM) [online] Available: http://www.its.bldrdoc.gov/resources/video-quality-research/guides-and-tutorials/description-of-vqm-tools.aspx http://www.its.bldrdoc.gov/resources/video-quality-research/guides-and-tutorials/description-of-vqm-tools.aspx. Accessed on April 26, 2015

  31. Wang Y, Zhu QF, Shaw L (1993) Maximally smooth image recovery in transform coding. IEEE Trans Commun 41(10):1544–1551

    Article  MATH  Google Scholar 

  32. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13 (4):600–612

    Article  Google Scholar 

  33. Wiegand T, Sullivan GJ, Bjontegaard G, Luthra A (2003) Overview of the H.264/AVC video coding standard. IEEE Trans Circuits Syst Video Technol 13 (7):560–576

    Article  Google Scholar 

  34. Yan B, Gharavi H (2010) A hybrid frame concealment algorithm for H.264/AVC. IEEE Trans Image Process 19(1):98–107

    Article  MATH  MathSciNet  Google Scholar 

  35. Yang D, Liu T, Liu SM, Chen FC (2016) An adaptive spatial-temporal error concealment scheme based on h.264/avc. In: 5Th international conference on electronics, communications and networks (CECNet), Shanghai, China, pp 79–88

  36. Yang M, Gadgil N, Comer ML, Delp EJ (2016) Adaptive error concealment for temporal-spatial multiple description video coding. Signal Process Image Commun 47:313–331

    Article  Google Scholar 

  37. Yin L, Basu A (1997) MPEG-4 Face modeling using fiducial points. In: International conference on image processing (ICIP), Washington, DC, USA, pp 109–112

  38. Zhang L (1998) Automatic adaptation of a face model using action units for semantic coding of videophone sequences. IEEE Trans Circuits Syst Video Technol 8(6):781–795

    Article  Google Scholar 

Download references

Acknowledgments

Authors would like to acknowledge Prof. Somnath Sengupta, Indian Institute of Technology Kharagpur, India for his endless support through out the entire research.

Author information

Authors and Affiliations

  1. Department of Electronics & Instrumentation Engineering, National Institute of Technology Silchar, Silchar, Assam, 788010, India

    Abhishek Midya

  2. Department of Surgery, Memorial Sloan Kettering Cancer Center, New York, NY, 10022, USA

    Jayasree Chakraborty

  3. Department of Electrical & Computer Engineering, University of Maryland, College Park, MD, 20742, USA

    Rajeev Ranjan

Authors
  1. Abhishek Midya
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jayasree Chakraborty
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rajeev Ranjan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jayasree Chakraborty.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Midya, A., Chakraborty, J. & Ranjan, R. Video error concealment through 3-D face model. Multimed Tools Appl 76, 23931–23955 (2017). https://doi.org/10.1007/s11042-016-4148-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-016-4148-x

Keywords

Search

Navigation