Skip to main content
SpringerLink
Log in

High-quality virtual view synthesis in 3DTV and FTV

  • 3DR Express
  • Published:
3D Research

Abstract

Autostereoscopic 3DTV is becoming an exciting media that enable us to view a 3D scene from more than one viewpoint. Meanwhile, considered as the ultimate autostereoscopic 3DTV, Free-viewpoint TV (FTV) can provide arbitrary views by freely synthesizing and changing viewpoints. Essentially, either 3DTV or FTV is based on virtual view synthesis using captured views along with corresponding depth information. In this paper, we study how virtual views can be reliably generated from multiple captured videos for 3D display. One key challenge is that the required depth information may contain depth errors, leading to uncomfortable artifacts in the synthesized view. We review the recent progress in virtual view synthesis methods where depth reliability is considered to handle synthesis artifacts and improve the quality of the virtual view. Not only for intermediate virtual view, have we also presented high-quality close-up view synthesis methods for wider navigation in 3DTV and FTV.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. J. Bernardo, A. Smith (1994) Bayesian Theory, John Wiley & Sons, Chichester.

    Book  MATH  Google Scholar 

  2. M. Bertalmio, A.L. Bertozzi, G. Sapiro (2001) Navierstokes, fluid dynamics, and image and video inpainting, IEEE Conference on Computer Vision and Pattern Recognition, 1: 355–362

    Google Scholar 

  3. Y. Boykov, O. Veksler, R. Zabih (2001) Fast approximate energy minimization via graph cuts, Pattern Analysis and Machine intelligence, IEEE Transactions, 23(11):1222–1239.

    Article  Google Scholar 

  4. J.X. Chai, X. Tong, S.C. Chan, H.Y. Shum (2000) Plenoptic sampling, SIGGRAPH’ 00: Proceedings of the 27 th annual conference on Computer graphics and interactive techniques, 307–318.

  5. G. Chantas, N. Galatsanos, A. Likas (2006) Bayesian restoration using a new nonstationary edge-preserving image prior, Image Processing, IEEE Transactions, 15(10):2987–2997.

    Article  MathSciNet  Google Scholar 

  6. S. Chan, H.Y. Shum, K.T. Ng (2007) Image-based rendering and synthesis, Signal Processing Magazine, IEEE 24 (6):22–33.

    Google Scholar 

  7. S.E. Chen, L. Williams (1993) View interpolation for image synthesis, Proceedings of the 20th annual conference on Computer graphics and interactive techniques, SIGGRAPH'93, 279–288.

  8. A. Criminisi, A. Blake (2004) The SPS algorithm: Patching figural continuity and transparency by split-patch search, IEEE Conference on Computer Vision and Pattern Recognition, 1: 342–349.

    Google Scholar 

  9. P.E. Debevec, C.J. Taylor, J. Malik (1996) Modeling and rendering architecture from photographs: a hybrid geometry and image based approach, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, SIGGRAPH’ 96, 11–20.

  10. C. Fehn (2004) Depth-image-based rendering (DIBR), compression and transmission for a new approach on 3DTV, Proceedings of SPIE Stereoscopic Displays and Virtual Reality Systems XI, 93–104.

  11. A. Fitzgibbon, Y. Wexler, A. Zisserman (2005) Image-based rendering using image-based priors, International Journal of Computer Vision 63 (2):141–151.

    Google Scholar 

  12. T. Fujii (1996) Ray space coding for 3D visual communication, Picture Coding Symposium (PCS), 447–451.

  13. N. Fukushima, T. Yendo, T. Fujii, M. Tanimoto (2007) Free viewpoint image generation synchronized with free listening-point audio for 3-D real space navigation, 3DTV Conference, 1–4.

  14. H. Furihata, T. Yendo, M.P. Tehrani, T. Fujii, M. Tanimoto (2010) Novel view synthesis with residual error feedback for FTV, Stereoscopic Displays and Applications XXI 7524 (1):75240K.

    Google Scholar 

  15. R.C. Gonzalez, R.E. Woods (2001) Digital Image Processing, 2nd edn, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.

    Google Scholar 

  16. R. Hartley, A. Zisserman (2004) Multiple View Geometry in Computer Vision, 2nd edn, Cambridge University Press, Cambridge.

    Book  MATH  Google Scholar 

  17. S.W. Hasinoff, S.B. Kang, R. Szeliski (2006) Boundary matting for view synthesis, Computer Vision and Image Understanding 103 (1):22–32.

    Google Scholar 

  18. Coding of moving pictures and audio, M15377, ISO/IEC JTC1/SC29/WG11, Depth Estimation Reference Software (DERS).

  19. Coding of moving pictures and audio, N9783, ISO/IEC JTC1/SC29/WG11.

  20. L. Jiang, J. He, N. Zhang, T. Huang (2010) An overview of 3D video representation and coding, 3D Research 1:43–47.

    Article  Google Scholar 

  21. P. Kauff, N. Atzpadin, C. Fehn, M. Müller, O. Schreer, A. Smolic, R. Tanger (2007) Depth map creation and imagebased rendering for advanced 3DTV services providing interoperability and scalability, Signal Processing: Image Communication 22 (2):217–234.

    Google Scholar 

  22. J. Konrad, M. Halle (2007) 3-D displays and signal processing, Signal Processing Magazine, IEEE 24 (6):97–111.

    Google Scholar 

  23. A. Kubota, A. Smolic, M. Magnor, M. Tanimoto, T. Chen, C. Zhang (2007) Multiview imaging and 3DTV, Signal Processing Magazine, IEEE 24 (6):10–21.

    Google Scholar 

  24. C. Lee, Y.S. Ho (2008) Boundary filtering on synthesized views of 3D video, FGCNS’ 08: Proceedings of the 2008 Second International Conference on Future Generation Communication and Networking Symposia, 15–18.

  25. C. Lee, Y.S. Ho (2009) Implementation of Boundary Noise Removal for View Synthesis, ISO/IEC JTC1/SC29/WG11, coding of moving pictures and audio, M16064.

  26. M. Levoy, P. Hanrahan (1996) Light field rendering, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, SIGGRAPH’ 96, 31–42.

  27. S.Z. Li (2001) Markov Random Field Modeling in Image Analysis, Springer-Verlag, Tokyo.

    MATH  Google Scholar 

  28. W.R. Mark, L. McMillan, G. Bishop (1997) Post-rendering 3D warping, Proceedings of the 1997 symposium on Interactive 3D graphics, I3D’ 97, 7–16.

  29. L. McMillan, G. Bishop (1995) Plenoptic modeling: an image-based rendering system, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, SIGGRAPH'95, 39–46.

  30. D. Min, D. Kim, S. Yun, K. Sohn (2009) 2D/3D freeview video generation for 3DTV system, Signal Processing: Image Communication 24(1–2):31–48.

    Article  Google Scholar 

  31. Y. Mori, N. Fukushima, T. Yendo, T. Fujii, M. Tanimoto (2009) View generation with 3D warping using depth information for FTV, Signal Processing: Image Communication 24(1–2):65–72.

    Article  Google Scholar 

  32. Y. Morvan, D. Farin, P. De With (2008) System architecture for free-viewpoint video and 3D-TV, Consumer Electronics, IEEE Transactions, 54(2):925–932.

    Article  Google Scholar 

  33. U. Mudenagudi, A. Gupta, L. Goel, A. Kushal, P. Kalra, S. Banerjee (2007) Super resolution of images of 3D scenes, ACCV 2007, 85–95.

  34. K. Müller, A. Smolic, K. Dix, P. Kauff, T. Wiegand (2008) Reliability-based generation and view synthesis in layered depth video, International Workshop on Multimedia Signal Processing, 34–39.

  35. T. Naemura, H. Harashima (2000) Ray-based approach to integrated 3D visual communication, Proceedings Vol. CR76, Three-Dimensional Video and Display: Devices and Systems, 282–305.

  36. L. Onural (2010) Signal processing and 3DTV [in the spotlight], Signal Processing Magazine, IEEE 27(5):144–142.

    Article  Google Scholar 

  37. D. Scharstein, R. Szeliski (2002) A taxonomy and evaluation of dense two-frame stereo correspondence algorithms, International Journal of Computer Vision 47(1–3):7–42.

    Article  MATH  Google Scholar 

  38. S. Shimizu, H. Kimata, Y. Ohtani (2009) Real-time freeviewpoint viewer from multiview video plus depth representation coded by H.264/AVC MVC extension, IEEE 3DTV-CON, 1–4.

  39. Y. Shishikui, Y. Fujita, K. Kubota (2009) Super Hi-Vision - the star of the show! EBU Technical Review 09-SE:4–16.

  40. H.Y. Shum, S.C. Chan, S.B. Kang (2006) Image-Based Rendering, Springer-Verlag, New York.

    Google Scholar 

  41. H. Shum, S.B. Kang (2000) A review of image-based rendering techniques, VCIP'00, 2–13.

  42. A. Smolic, K. Müller, K. Dix, P. Merkle, P. Kauff, T. Wiegand (2008) Intermediate view interpolation based on multiview video plus depth for advanced 3D video systems, Proceedings of the International Conference on Image Processing, 2448–2451.

  43. A. Smolic, K. Müller, N. Stefanoski, J. Ostermann, A. Gotchev, G. Akar, G. Triantafyllidis, A. Koz (2007) Coding algorithms for 3DTV — a survey, Circuits and Systems for Video Technology, IEEE Transactions, 17(11):1606–1621.

    Article  Google Scholar 

  44. M. Tanimoto, T. Fujii, M.P. Tehrani, M. Wildeboer, L. Yang (2010) Reliability Based View Synthesis for FTV, ISO/IEC JTC1/SC29/WG11, coding of moving pictures and audio, M17767.

  45. M. Tanimoto, M.P. Tehrani, T. Fujii, T. Yendo (2011) Free-viewpoint TV, Signal Processing Magazine, IEEE 28(1):67–76.

    Article  Google Scholar 

  46. M. Tanimoto (2006) Overview of free viewpoint television, Signal Processing: Image Communication 21(6):454–461.

    Article  Google Scholar 

  47. C. Theobalt, N. Ahmed, G. Ziegler, H. Seidel (2007) Highquality reconstruction from multiview video streams, Signal Processing Magazine, IEEE 24(6):45–57.

    Article  Google Scholar 

  48. D.N. Wood, D.I. Azuma, K. Aldinger, B. Curless, T. Duchamp, D.H. Salesin, W. Stuetzle (2000) Surface light fields for 3D photography, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, SIGGRAPH’ 00, 287–296.

  49. L. Yang, M. Wildeboer, T. Yendo, M.P. Tehrani, T. Fujii, M. Tanimoto (2010) Reducing bitrates of compressed video with enhanced view synthesis for FTV, IEEE Picture Coding Symposium (PCS), 2010, 5–8.

  50. L. Yang, T. Yendo, M.P. Tehrani, T. Fujii, M. Tanimoto (2010) Probabilistic reliability based view synthesis for FTV, Image Processing (ICIP), 2010 17th IEEE International Conference, 1785–1788.

  51. L. Yang, T. Yendo, M.P. Tehrani, T. Fujii, M. Tanimoto (2010) Artifact reduction using reliability reasoning for image generation of FTV, Journal of Visual Communication and Image Representation 21(5–6):542–560.

    Article  Google Scholar 

  52. L. Yang, T. Yendo, M.P. Tehrani, T. Fujii, M. Tanimoto (2010) Reliable view synthesis with automatic error compensation for FTV, Forum on Information Technology (FIT2010), 3:35–38, RI-007.

    Google Scholar 

  53. L. Yang, T. Yendo, M.P. Tehrani, T. Fujii, M. Tanimoto (2010) View synthesis using probabilistic reliability reasoning for FTV, The Journal of The Institute of Image Information and Television Engineers 64(11):1671–1677.

    Article  Google Scholar 

  54. L. Yang, T. Yendo, M.P. Tehrani, T. Fujii, M. Tanimoto (2010) Error supression in view synthesis using reliability reasoning for FTV, IEEE 3DTV-CON, 1–4.

  55. C. Zhang, T. Chen (2004) A survey on image-based rendering-representation, sampling and compression, Signal Processing: Image Communication 19(1):1–28.

    Article  MATH  Google Scholar 

  56. Y. Zhao, C. Zhu, Z. Chen, D. Tian, L. Yu (2011) Boundary artifact reduction in view synthesis of 3D video: From perspective of texture-depth alignment, Broadcasting, IEEE Transactions, 57(2):510–522.

    Article  Google Scholar 

  57. C.L. Zitnick, S.B. Kang, M. Uyttendaele, S. Winder, R. Szeliski (2004) High-quality video view interpolation using a layered representation, SIGGRAPH'04, 600–608.

Download references

Author information

Authors and Affiliations

  1. Graduate School of Engineering, Nagoya University, Nagoya, Japan

    Lu Yang, Mehrdad Panahpour Tehrani, Toshiaki Fujii & Masayuki Tanimoto

Authors
  1. Lu Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mehrdad Panahpour Tehrani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Toshiaki Fujii
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Masayuki Tanimoto
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lu Yang.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Cite this article

Yang, L., Tehrani, M.P., Fujii, T. et al. High-quality virtual view synthesis in 3DTV and FTV. 3D Res 2, 5 (2011). https://doi.org/10.1007/3DRes.04(2011)5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/3DRes.04(2011)5

Keywords

Search

Navigation