Advertisement

Fast and high-quality virtual view synthesis from multi-view plus depth videos

  • Li YaoEmail author
  • Yingdong Han
  • Xiaomin Li
Article
  • 6 Downloads

Abstract

Depth image based rendering (DIBR) is an effective method for virtual view synthesis from Multi-view Plus Depth(MVD) video. Synthetic images, however, often contain ghost effect and some holes of varying sizes. This paper uses color correction of reference views, and combines depth-based image fusion with direct color image fusion to decrease the ghost effect. Meanwhile, the cracks are filled using depth filtering and inverse warping. What’s more, the image depth-aided inpainting with GPU acceleration is used to fill the remaining big disocclusions. Experimental results show that our proposed method improved the quality of virtual view synthetic images and reduced the processing time sharply.

Keywords

View synthesis Artifacts removal Depth-aided inpainting GPU acceleration 

Abbreviations

DIBR

Depth image based rendering

MVD

Multi-view Plus Depth

FVV

Free Viewpoint Video

SSD

Sum of Squared Difference

PSNR

Peak-Signal to Noise Ratio

SSIM

Structural Similarity Index Measurement

CUDA

Compute Unified Device Architecture

Notes

Funding

This work is supported by natural science foundation of Jiangsu Province under Grant No.BK20181267, Industrial Prospective Project of Jiangsu Technology Department under Grant No.BE2018119.

References

  1. 1.
    Chen W, Chang Y, Lin S, Ding L, Chen L (2005) Efficient depth image based rendering with edge dependent depth filter and interpolation. In: ICME. IEEE international conference on multimedia & expo. IEEEGoogle Scholar
  2. 2.
    Criminisi A, Prez P, Toyama K (2003) Object Removal by Exemplar-Based Inpainting. In 2003 IEEE conference on computer vision and pattern recognition (CVPR). IEEE Computer Society 2:721–728Google Scholar
  3. 3.
    Daribo I, Tillier C, Pesquet-Popescu B (2007) Distance dependent depth filtering in 3D warping for 3DTV. In: IEEE workshop on multimedia signal processing. IEEEGoogle Scholar
  4. 4.
    Do L, Zinger S, With PHND (2010) Quality improving techniques for free-viewpoint DIBR. In: 3dtv conference: the true vision - capture, transmission and display of 3d video, vol 7524. IEEE Xplore, pp 1–4Google Scholar
  5. 5.
    Fehn C (2004) Depth-image-based rendering (dibr), compression, and transmission for a new approach on 3d-tv. Proc SPIE 5291:93–104CrossRefGoogle Scholar
  6. 6.
    Fezza SA, Larabi MC, Faraoun KM (2014) Feature-based color correction of multiview video for coding and rendering enhancement. IEEE Trans Circuits Syst Video Technol 24(9):1486–1498CrossRefGoogle Scholar
  7. 7.
    Fickel GP, Jung CR, Lee B (2015) Multiview image and video interpolation using weighted vector median filters. In: IEEE international conference on image processing, vol 29. IEEE, pp 5387–5391Google Scholar
  8. 8.
    Jung JI, Ho YS (2013) Color correction for multi-view images using relative luminance and chrominance mapping curves. Journal of Signal Processing Systems 72(2):107–117CrossRefGoogle Scholar
  9. 9.
    Leonard Mcmillan J (1997) An image-based approach to three-dimensional computer graphics. University of North Carolina at Chapel HillGoogle Scholar
  10. 10.
    Li S, Zhu C, Sun MT (2018) Hole filling with multiple reference views in dibr view synthesis. IEEE Trans Multimedia 20(8):1948–1959Google Scholar
  11. 11.
    Loghman M, Kim J (2015) Segmentation-based view synthesis for multi-view video plus depth. Multimed Tools Appl 74(5):1611–1625CrossRefGoogle Scholar
  12. 12.
    Luo G, Zhu Y, Li Z, Zhang L (2016) A hole filling approach based on background reconstruction for view synthesis in 3D video. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR). IEEE Computer SocietyGoogle Scholar
  13. 13.
    Marcelino S, Soares S, Faria SMMD, Assuncao P (2016) Reconstruction of lost depth data in multiview video-plus-depth communications using geometric transforms. J Vis Commun Image Represent 40:589–599CrossRefGoogle Scholar
  14. 14.
    Merkle P, Smolic A, Müller K, Wiegand T (2007) Multi-view video plus depth representation and coding. In: IEEE international conference on image processing. IEEEGoogle Scholar
  15. 15.
    Rahaman DM, Paul M (2018) Virtual view synthesis for free viewpoint video and multiview video compression using gaussian mixture modelling. IEEE Trans Image Process PP(99):1190–1201MathSciNetCrossRefGoogle Scholar
  16. 16.
    Tanimoto M (2006) Overview of free viewpoint television. Signal Process Image Commun 21(6):454–461CrossRefGoogle Scholar
  17. 17.
    Tanimoto M (2012) Ftv: free-viewpoint television. Signal Process Image Commun 27(6):555–570CrossRefGoogle Scholar
  18. 18.
    Anthony Vetro, Thomas Wiegand, Gary J. Sullivan (2011). Overview of the stereo and multiview video coding extensions of the h.264/mpeg-4 avc standard. Proceedings of the IEEE 99(4):626–642 Google Scholar
  19. 19.
    Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612CrossRefGoogle Scholar
  20. 20.
    Yao L, Han Y, Li X (2016) Virtual viewpoint synthesis using CUDA acceleration. In: ACM conference on virtual reality software and technology. ACM, pp 367–368Google Scholar
  21. 21.
    Zamarin M, Salmistraro M, Forchhammer S, Ortega A (2013) Edge-preserving intra depth coding based on context-coding and H.264/AVC. In: IEEE international conference on multimedia & expo. IEEEGoogle Scholar
  22. 22.
    Zhang L, Tam WJ, Wang D (2004) Stereoscopic image generation based on depth images. In: International conference on image processing. IEEEGoogle Scholar
  23. 23.
    Zhang L, Tam WJ, Wang D (2005) Stereoscopic image generation based on depth images. IEEE Trans Broadcast 51(2):191–199CrossRefGoogle Scholar
  24. 24.
    Zinger S, Do L, With PHND (2010) Free-viewpoint depth image based rendering. J Vis Commun Image Represent 21(5):533–541CrossRefGoogle Scholar
  25. 25.
    Zitnick CL, Kang SB, Uyttendaele M, Winder SAJ, Szeliski R (2004) High-quality video view interpolation using a layered representation. ACM Trans Graph 23(3):600–608CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.College of Computer Science and EngineeringSoutheast UniversityNanjingChina
  2. 2.Key Laboratory of Computer Network and Information Integration ( Southeast University )Ministry of EducationNanjingPeople’s Republic of China

Personalised recommendations