Advertisement

The Visual Computer

, Volume 32, Issue 5, pp 579–589 | Cite as

An enhanced depth map based rendering method with directional depth filter and image inpainting

  • Wei LiuEmail author
  • Dehua Zhang
  • Mingyue Cui
  • Jianwei Ding
Original Article

Abstract

Depth image-based rendering (DIBR), which is used to render virtual views with a color image and the corresponding depth map, is one of the key techniques in the 2D to 3D video conversion process. In this paper, a novel method is proposed to partially solve two puzzles of DIBR, i.e. visual image generation and hole filling. The method combines two different approaches for synthesizing new views from an existing view and a corresponding depth map. Disoccluded parts of the synthesized image are first classified as either smooth or highly structured. At structured regions, inpainting is used to preserve the background structure. In other regions, an improved directional depth smoothing is used to avoid disocclusion. Thus, more details and straight line structures in the generated virtual image are preserved. The key contributions include an enhanced adaptive directional filter and a directional hole inpainting algorithm. Experiments show that the disocclusion is removed and the geometric distortion is reduced efficiently. The proposed method can generate more visually satisfactory results.

Keywords

Depth image-based rendering Directional depth filter  Directional hole inpainting Stereoscopic image generation 

Notes

Acknowledgments

This work was supported in part by Ministry of Science and Technology of China under National 973 Basic Research Program (Grants No. 2013CB228206 and No. 2011CB302505) and National Natural Science Foundation of China (Grant No. U1404614 ).

References

  1. 1.
    Zhang, L., Vazquez, C., Knorr, S.: 3D-TV content creation: automatic 2D-to-3D video conversion. IEEE Trans. Broadcast. 57(2), 372–383 (2011)CrossRefGoogle Scholar
  2. 2.
    Liu, W., Wu, Y., Guo, F., et al.: An efficient method for 2D to 3D conversion based on structure from motion. Vis. Comput. (2013). doi: 10.1007/s00371-013-0904-3
  3. 3.
    Knorr, S., Smolic, A., Sikora, T.: From 2D-to stereo-to multi-view video. In: Proceedings of 3DTV, pp. 1–4 (2007)Google Scholar
  4. 4.
    Rotem, E., Wolowelsky, K., Pelz, D.: Automatic video to stereoscopic video conversion. In: Proceedings of SPIE Conference on Stereoscopic Displays and Virtual Reality Systems, vol. 5664, pp. 198–206 (2005)Google Scholar
  5. 5.
    Zhang, G., Hua, W., Qin, X., et al.: Stereoscopic video synthesis from a monocular video. IEEE Trans. Vis. Comput. Graph. 13(4), 686–696 (2007)CrossRefGoogle Scholar
  6. 6.
    Fehn, C.: Depth-image-based rendering (DIBR), compression and transmission for a new approach on 3D-TV. In: Proceedings of SPIE Conference on Stereoscopic Displays and Virtual Reality Systems vol. 5291, pp. 93–104 (2004)Google Scholar
  7. 7.
    Zhang, L., Tam, W.J.: Stereoscopic image generation based on depth images for 3D TV. IEEE Trans. Broadcast. 51(2), 191–199 (2005)CrossRefGoogle Scholar
  8. 8.
    Wang, L., Huang, X., Xi, M., et al.: An asymmetric edge adaptive filter for depth generation and hole filling in 3DTV. IEEE Trans. Broadcast. 56(3), 425–431 (2010)CrossRefGoogle Scholar
  9. 9.
    Lee, S., Ho, Y.: Discontinuity-adaptive depth map filtering for 3D view generation. In: Proceedings of the 2nd International Conference on Immersive Telecommunications, pp. 1–6 (2009)Google Scholar
  10. 10.
    Fehn, C.: A 3D-TV approach using depth-image-based-rendering (DIBR). In: Proceedings of Visualization, Imaging, and Image Processing, pp. 482–487 (2003)Google Scholar
  11. 11.
    Hartley, R., Zisserman, A.: Multiple View Geometry. Cambridge University Press, Cambridge (2003)zbMATHGoogle Scholar
  12. 12.
    Criminisi, A., Perez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13(9), 1200–1212 (2004)CrossRefGoogle Scholar
  13. 13.
    Gabriel, P.: Texture synthesis with grouplets. Pattern Anal. Mach. Intell. 32(4), 733–746 (2010)CrossRefGoogle Scholar
  14. 14.
    Choi, J., Choe, Y., Kim, Y.: Sparsity based depth estimation and hole-filling algorithm for 2D to 3D video conversion. In: Proceedings of International Conference on Signals and Electronic Systems, pp. 1–4 (2012)Google Scholar
  15. 15.
    Shen, J., Jin, X., Zhou, C., et al.: Gradient based image completion by solving the Poisson equation. Comput. Graph. 31(1), 119–126 (2007)CrossRefGoogle Scholar
  16. 16.
    Hwang, J., Lee, K., Kim, J., et al.: A novel hole filling method using image segmentation-based image in-painting. In: Proceedings of IEEE International Conference on Consumer Electronics (ICCE), pp. 470–471 (2013)Google Scholar
  17. 17.
    Reel, S., Cheung, G., Wong, P., et al.: Joint texture-depth pixel inpainting of disocclusion holes in virtual view synthesis. In: Proceedings of Signal and Information Processing Association Annual Summit and Conference (APSIPA), pp. 1–7 (2013)Google Scholar
  18. 18.
    Tam, W., Alain, G., Zhang, L., et al.: Smoothing depth maps for improved stereoscopic image quality. In: Proceedings of SPIE Conference on Three-Dimensional TV, Video, and Display III, vol. 5599, pp. 162–172 (2004)Google Scholar
  19. 19.
    Daribo, I., Tillier, C., Pesquet-Popescu, B.: Distance dependent depth filtering in 3D warping for 3DTV. In: Proceedings of the IEEE Workshop on Multimedia Signal Processing (MMSP), pp. 312–315 (2007)Google Scholar
  20. 20.
    Chen, W., Chang, Y., Lin, S., et al.: Efficient depth image based rendering with edge dependent depth filter and interpolation. In: Proceedings of IEEE International Conference on Multimedia and Expo, pp. 1314–1317 (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  • Wei Liu
    • 1
    • 2
    Email author
  • Dehua Zhang
    • 3
  • Mingyue Cui
    • 1
  • Jianwei Ding
    • 4
  1. 1.Nanyang Normal UniversityNanyangChina
  2. 2.Center for Internet of ThingsInstitute of Microelectronics of Chinese Academy of SciencesBeijingChina
  3. 3.Tsinghua UniversityBeijingChina
  4. 4.People’s Public Security University of ChinaBeijingChina

Personalised recommendations