Skip to main content
SpringerLink
Log in

Fast and high-quality virtual view synthesis from multi-view plus depth videos

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Depth image based rendering (DIBR) is an effective method for virtual view synthesis from Multi-view Plus Depth(MVD) video. Synthetic images, however, often contain ghost effect and some holes of varying sizes. This paper uses color correction of reference views, and combines depth-based image fusion with direct color image fusion to decrease the ghost effect. Meanwhile, the cracks are filled using depth filtering and inverse warping. What’s more, the image depth-aided inpainting with GPU acceleration is used to fill the remaining big disocclusions. Experimental results show that our proposed method improved the quality of virtual view synthetic images and reduced the processing time sharply.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Abbreviations

DIBR:

Depth image based rendering

MVD:

Multi-view Plus Depth

FVV:

Free Viewpoint Video

SSD:

Sum of Squared Difference

PSNR:

Peak-Signal to Noise Ratio

SSIM:

Structural Similarity Index Measurement

CUDA:

Compute Unified Device Architecture

References

  1. Chen W, Chang Y, Lin S, Ding L, Chen L (2005) Efficient depth image based rendering with edge dependent depth filter and interpolation. In: ICME. IEEE international conference on multimedia & expo. IEEE

  2. Criminisi A, Prez P, Toyama K (2003) Object Removal by Exemplar-Based Inpainting. In 2003 IEEE conference on computer vision and pattern recognition (CVPR). IEEE Computer Society 2:721–728

  3. Daribo I, Tillier C, Pesquet-Popescu B (2007) Distance dependent depth filtering in 3D warping for 3DTV. In: IEEE workshop on multimedia signal processing. IEEE

  4. Do L, Zinger S, With PHND (2010) Quality improving techniques for free-viewpoint DIBR. In: 3dtv conference: the true vision - capture, transmission and display of 3d video, vol 7524. IEEE Xplore, pp 1–4

  5. Fehn C (2004) Depth-image-based rendering (dibr), compression, and transmission for a new approach on 3d-tv. Proc SPIE 5291:93–104

    Article  Google Scholar 

  6. Fezza SA, Larabi MC, Faraoun KM (2014) Feature-based color correction of multiview video for coding and rendering enhancement. IEEE Trans Circuits Syst Video Technol 24(9):1486–1498

    Article  Google Scholar 

  7. Fickel GP, Jung CR, Lee B (2015) Multiview image and video interpolation using weighted vector median filters. In: IEEE international conference on image processing, vol 29. IEEE, pp 5387–5391

  8. Jung JI, Ho YS (2013) Color correction for multi-view images using relative luminance and chrominance mapping curves. Journal of Signal Processing Systems 72(2):107–117

    Article  Google Scholar 

  9. Leonard Mcmillan J (1997) An image-based approach to three-dimensional computer graphics. University of North Carolina at Chapel Hill

  10. Li S, Zhu C, Sun MT (2018) Hole filling with multiple reference views in dibr view synthesis. IEEE Trans Multimedia 20(8):1948–1959

  11. Loghman M, Kim J (2015) Segmentation-based view synthesis for multi-view video plus depth. Multimed Tools Appl 74(5):1611–1625

    Article  Google Scholar 

  12. Luo G, Zhu Y, Li Z, Zhang L (2016) A hole filling approach based on background reconstruction for view synthesis in 3D video. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR). IEEE Computer Society

  13. Marcelino S, Soares S, Faria SMMD, Assuncao P (2016) Reconstruction of lost depth data in multiview video-plus-depth communications using geometric transforms. J Vis Commun Image Represent 40:589–599

    Article  Google Scholar 

  14. Merkle P, Smolic A, Müller K, Wiegand T (2007) Multi-view video plus depth representation and coding. In: IEEE international conference on image processing. IEEE

  15. Rahaman DM, Paul M (2018) Virtual view synthesis for free viewpoint video and multiview video compression using gaussian mixture modelling. IEEE Trans Image Process PP(99):1190–1201

    Article  MathSciNet  MATH  Google Scholar 

  16. Tanimoto M (2006) Overview of free viewpoint television. Signal Process Image Commun 21(6):454–461

    Article  Google Scholar 

  17. Tanimoto M (2012) Ftv: free-viewpoint television. Signal Process Image Commun 27(6):555–570

    Article  Google Scholar 

  18. Anthony Vetro, Thomas Wiegand, Gary J. Sullivan (2011). Overview of the stereo and multiview video coding extensions of the h.264/mpeg-4 avc standard. Proceedings of the IEEE 99(4):626–642

  19. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  20. Yao L, Han Y, Li X (2016) Virtual viewpoint synthesis using CUDA acceleration. In: ACM conference on virtual reality software and technology. ACM, pp 367–368

  21. Zamarin M, Salmistraro M, Forchhammer S, Ortega A (2013) Edge-preserving intra depth coding based on context-coding and H.264/AVC. In: IEEE international conference on multimedia & expo. IEEE

  22. Zhang L, Tam WJ, Wang D (2004) Stereoscopic image generation based on depth images. In: International conference on image processing. IEEE

  23. Zhang L, Tam WJ, Wang D (2005) Stereoscopic image generation based on depth images. IEEE Trans Broadcast 51(2):191–199

    Article  Google Scholar 

  24. Zinger S, Do L, With PHND (2010) Free-viewpoint depth image based rendering. J Vis Commun Image Represent 21(5):533–541

    Article  Google Scholar 

  25. Zitnick CL, Kang SB, Uyttendaele M, Winder SAJ, Szeliski R (2004) High-quality video view interpolation using a layered representation. ACM Trans Graph 23(3):600–608

    Article  Google Scholar 

Download references

Funding

This work is supported by natural science foundation of Jiangsu Province under Grant No.BK20181267, Industrial Prospective Project of Jiangsu Technology Department under Grant No.BE2018119.

Author information

Authors and Affiliations

  1. College of Computer Science and Engineering, Southeast University, Nanjing, China

    Li Yao, Yingdong Han & Xiaomin Li

  2. Key Laboratory of Computer Network and Information Integration ( Southeast University ), Ministry of Education, Nanjing, 211189, People’s Republic of China

    Li Yao

Authors
  1. Li Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yingdong Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaomin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Li Yao.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yao, L., Han, Y. & Li, X. Fast and high-quality virtual view synthesis from multi-view plus depth videos. Multimed Tools Appl 78, 19325–19340 (2019). https://doi.org/10.1007/s11042-019-7236-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-7236-x

Keywords

Search

Navigation