Abstract
The depth information of the scene indicates the distance between the object and the camera, and depth extraction is a key technology in 3D video system. The emergence of Kinect makes the high resolution depth map capturing possible. However, the depth map captured by Kinect can not be directly used due to the existing holes and noises, which needs to be repaired. We propose a texture combined inpainting algorithm in this paper. Firstly, the foreground is segmented combined with the color characteristics of the texture image to repair the foreground of the depth map. Secondly, region growing is used to determine the match region of the hole in the depth map, and to accurately position the match region according to the texture information. Then the match region is weighted to fill the hole. Finally, a Gaussian filter is used to remove the noise in the depth map. Experimental results show that the proposed method can effectively repair the holes existing in the original depth map and get an accurate and smooth depth map, which can be used to render a virtual image with good quality.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
M. Tanimoto, M. P. Tehrani, T. Fujii, et al.. FTV for 3-D Spatial Communication. Proceedings of the IEEE, 100(2012)4, 905–917.
P. Merkle, A. Smolic, K. Muller, et al.. Compressing time-varying visual content. Signal Processing Magazine, 24(2007)6, 58–65.
Sung-Yeol Kim, Eun-Kyung Lee, and Yo-Sung Ho. Generation of ROI enhanced depth maps using stereoscopic cameras and a depth camera. IEEE Transactions on Broadcasting, 54(2008)4, 732–740.
Zhang Qiuwen, An Ping, Zhang Yan, et al.. An improved depth map estimation for coding and view synthesis. 18th IEEE International Conference on Image Processing (ICIP), Brussels, September 2011, 2101–2104.
V. Gandhi, J. Cech, and R. Horaud. High-resolution depth maps based on TOF-stereo fusion. IEEE International Conference on, Robotics and Automation (ICRA), St. Paul, MN, USA, May 2012, 4742–4749.
K. R. Vijayanagar, M. Loghman, and K. Joohee. Refinement of depth maps generated by low-cost depth sensors. International SoC Design Conference (ISOCC), Jeju, Korea, November 2012, 355–358.
S. Milani and G. Calvagno. Joint denoising and interpolation of depth maps for MS Kinect sensors. IEEE International Conference on, Acoustics, Speech and Signal Processing (ICASSP), Kyoto, Japan, March 2012, 797–800.
S. Matyunin, D. Vatolin, Y. Berdnikov, et al.. Temporal filtering for depth maps generated by Kinect depth camera. 3DTV Conference: The True Vision-Capture, Transmission and Display of 3D Video (3DTV-CON), Antalya, Turkey, May 2011, 1–4.
Xu Kang, Zhou Jun, and Wang Zhen. A method of hole-filling for the depth map generated by Kinect with moving objects detection. IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB), Seoul, Korea, June 2012, 1–5.
M. Koppel, P. Ndjiki-Nya, D. Doshkov, et al.. Temporally consistent handling of disocclusions with texture synthesis for depth-image-based rendering. 17th IEEE International Conference on Image Processing (ICIP), Hong Kong, China, September 2010, 1809–1812.
P. Ndjiki-Nya, M. Koppel, D. Doshkov, et al.. Depth image-based rendering with advanced texture synthesis for 3-D video. IEEE Transactions on Multimedia, 13(2011)3, 453–465.
Kwan-Jung Oh, Sehoon Yea, and Yo-Sung Ho. Hole filling method using depth based in-painting for view synthesis in free viewpoint television and 3-D video. Picture Coding Symposium, Chicgo, IL, USA, May 2009, 1–4.
J. Vanne, E. Aho, T. D. Hamalainen, et al.. A high-performance sum of absolute difference implementation for motion estimation. IEEE Transactions on Circuits and Systems for Video Technology, 16(2006)7, 876–883.
Li Wenguo. Multi-threshold color image segmentation based on region growing. IEEE International Conference on Intelligent Computing and Intelligent Systems (ICIS), St. Louis, MO, USA, October 2010, 239–243.
J. C. Tilton, Y. Tarabalka, P. M. Montesano, et al.. Best merge region-growing segmentation with integrated nonadjacent region object aggregation. IEEE Transactions on Geoscience and Remote Sensing, 50 (2012)11, 4454–4467.
Liu Junyi, Gong Xiaojin, and Liu Jilin. Guided inpainting and filtering for Kinect depth maps. 21st International Conference on Pattern Recognition (ICPR), Tsukuba Science City, Japan, November 2012, 2055–2058.
Miao Dan, Fu Jing-jing, Lu Yan, et al.. Texture-assisted Kinect depth inpainting. IEEE International Symposium on Circuits and Systems (ISCAS), Seoul, Korea, May 2012, 604–607.
Wang Kui, An Ping, Zhang Yan, et al.. Real-time depth extraction and multi-view rendering algorithm based on Kincet. Journal of Optoelectronics·Laser, 23 (2012)10, 1949–1956.
Author information
Authors and Affiliations
Corresponding author
Additional information
Supported by the Key Project of National Natural Science Foundation of China (Nos. 60832003 and 61172096), major Project of Shanghai Science and Technology Committee (No. 10510500500), and the Major Innovation Project of Shanghai Municipal Education Commission.
About this article
Cite this article
Zhao, B., An, P., Liu, C. et al. Inpainting algorithm for Kinect depth map based on foreground segmentation. J. Electron.(China) 31, 41–49 (2014). https://doi.org/10.1007/s11767-013-3133-z
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11767-013-3133-z