Abstract
A new algorithm is proposed for restoring disocclusion regions in depth-image-based rendering (DIBR) warped images. Current solutions include layered depth image (LDI), pre-filtering methods, and post-processing methods. The LDI is complicated, and pre-filtering of depth images causes noticeable geometrical distortions in cases of large baseline warping. This paper presents a depth-aided inpainting method which inherits merits from Criminisi’s inpainting algorithm. The proposed method features incorporation of a depth cue into texture estimation. The algorithm efficiently handles depth ambiguity by penalizing larger Lagrange multipliers of filling points closer to the warping position compared with the surrounding existing points. We perform morphological operations on depth images to accelerate the algorithm convergence, and adopt a luma-first strategy to adapt to various color sampling formats. Experiments on test multi-view sequence showed that our method has superiority in depth differentiation and geometrical loyalty in the restoration of warped images. Also, peak signal-to-noise ratio (PSNR) statistics on non-hole regions and whole image comparisons both compare favorably to those obtained by state of the art techniques.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Akar, G.B., Tekalp, A.M., Fehn, C., Civanlar, M.R., 2007. Transport methods in 3DTV: a survey. IEEE Trans. Circ. Syst. Video Technol., 17(11):1622–1630. [doi:10.1109/TCSVT.2007.905365]
Chen, W.Y., Chang, Y.L., Lin, S.F., Ding, L.F., Chen, L.G., 2005. Efficient Depth Image Based Rendering with Edge Dependent Depth Filter and Interpolation. IEEE Int. Conf. on Multimedia and Expo, p.1314–1317. [doi:10.1109/ICME.2005.1521671]
Criminisi, A., Perez, P., Toyama, K., 2004. Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process., 13(9):1200–1212. [doi:10.1109/TIP.2004.833105]
Daribo, I., Tillier, C., Pesquet-Popescu, B., 2007. Distance Dependent Depth Filtering in 3D Warping for 3DTV. IEEE 9th Workshop on Multimedia Signal Processing, p.312–315. [doi:10.1109/MMSP.2007.4412880]
Fehn, C., 2004. Depth-Image-Based Rendering (DIBR), Compression, and Transmission for a New Approach on 3D-TV. SPIE, 5291:93–104. [doi:10.1117/12.524762]
Fehn, C., Cooke, E., Schreer, O., Kauff, P., 2002. 3D analysis and image-based rendering for immersive TV applications. Signal Process. Image Commun., 17(9):705–715. [doi:10.1016/S0923-5965(02)00079-6]
Fehn, C., de la Barre, R., Pastoor, S., 2006. Interactive 3-DTV: concepts and key technologies. Proc. IEEE, 94(3):524–538. [doi:10.1109/JPROC.2006.870688]
Flierl, M., Mavlankar, A., Girod, B., 2007. Motion and disparity compensated coding, for multiview video. IEEE Trans. Circ. Syst. Video Technol., 17(11):1474–1484. [doi:10.1109/TCSVT.2007.903780]
Gonzalez, R.C., Woods, R.E., 2002. Digital Image Processing (2nd Ed.). Prentice Hall, Upper Saddle River, New Jersey, USA.
Gonzalez, R.C., Woods, R.E., Eddins, S.L., 2005. Digital Image Processing Using Matlab. Prentice Hall, Upper Saddle River, New Jersey, USA.
Hartley, R.I., Zisserman, A., 2000. Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge, UK.
Kauff, P., Atzpadin, N., Fehn, C., Muller, M., Schreer, O., Smolic, A., Tanger, R., 2007. Depth map creation and image-based rendering for advanced 3DTV services providing interoperability and scalability. Signal Process. Image Commun., 22(2):217–234. [doi:10.1016/j.image.2006.11.013]
Lee, J., Kim, C., 2007. Removing Foreground Objects by Using Depth Information from Multi-view Images. SPIE, 6696:669627–669629. [doi:10.1117/12.735824]
McMillan, L., 1997. An Image-based Approach to Three-dimensional Computer Graphics. PhD Thesis, University of North Carolina at Chapel Hill, USA.
Merkle, P., Smolic, A., Muller, K., Wiegand, T., 2007. Efficient prediction structures for multiview video coding. IEEE Trans. Circ. Syst. Video Technol., 17(11):1461–1473. [doi:10.1109/TCSVT.2007.903665]
Shade, J., Gortler, S., He, L.W., Szeliski, R., 1998. Layered Depth Images. 25th Annual Conf. on Computer Graphics and Interactive Techniques, p.231–242. [doi:10.1145/280814.280882]
Smolic, A., Mueller, K., Stefanoski, N., Ostermann, J., Gotchev, A., Akar, G.B., Triantafyllidis, G., Koz, A., 2007. Coding algorithms for 3DTV: a survey. IEEE Trans. Circ. Syst. Video Technol., 17(11):1606–1621. [doi:10.1109/TCSVT.2007.909972]
Tauber, Z., Li, Z.N., Drew, M.S., 2007. Review and preview: disocclusion by inpainting for image-based rendering. IEEE Trans. Syst. Man Cybern. Part C-Appl. Rev., 37(4):527–540. [doi:10.1109/TSMCC.2006.886967]
Vázquez, C., Tam, W.J., Speranza, F., 2006. Stereoscopic Imaging: Filling Disoccluded Areas in Depth Image-based Rendering. SPIE, 6392:63920D–12. [doi:10.1117/12.685047]
Wang, L., Jin, H., Yang, R., Gong, M., 2008. Stereoscopic Inpainting: Joint Color and Depth Completion from Stereo Images. IEEE Conf. on Computer Vision and Pattern Recognition, p.1–8. [doi:10.1109/CVPR.2008.4587704]
Wang, W., Huo, L., Zeng, W., Huang, Q., Gao, W., 2007. Depth Image Segmentation for Improved Virtual View Image Quality in 3-DTV. Int. Symp. on Intelligent Signal Processing and Communication Systems, p.300–303. [doi:10.1109/ISPACS.2007.4445883]
Xu, G., Zhang, Z., 1996. Epipolar Geometry in Stereo Motion and Object Recognition. Kluver Academic Publishers, Dordrecht, The Netherlands.
Yoon, S.U., Ho, Y.S., 2007. Multiple color and depth video coding using a hierarchical representation. IEEE Trans. Circ. Syst. Video Technol., 17(11):1450–1460. [doi:10.1109/TCSVT.2007.905363]
Zhang, L., Tam, W.J., 2005. Stereoscopic image generation based on depth images for 3D TV. IEEE Trans. Broadcast., 51(2):191–199. [doi:10.1109/TBC.2005.846190]
Zitnick, C.L., Kang, S.B., Uyttendaele, M., Winder, S., Szeliski, R., 2004. High-quality video view interpolation using a layered representation. ACM Trans. Graph., 23(3):600–608. [doi:10.1145/1015706.1015766]
Author information
Authors and Affiliations
Corresponding author
Additional information
Project supported by the National Natural Science Foundation of China (No. 60802013) and the Natural Science Foundation of Zhejiang Province, China (No. Y106574)
Rights and permissions
About this article
Cite this article
Luo, K., Li, Dx., Feng, Ym. et al. Depth-aided inpainting for disocclusion restoration of multi-view images using depth-image-based rendering. J. Zhejiang Univ. Sci. A 10, 1738–1749 (2009). https://doi.org/10.1631/jzus.A0820806
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/jzus.A0820806
Key words
- Depth-aided inpainting
- Disocclusion restoration
- Depth-image-based rendering (DIBR)
- Image warping
- Stereoscopic image
- Multi-view image
- 3D-TV