Abstract
In this paper, we present a novel real-time algorithm to refine depth maps generated by low-cost commercial depth sensors like the Microsoft Kinect. The Kinect sensor falls under the category of RGB-D sensors that can generate a high resolution depth map and color image of a scene. They are relatively inexpensive and are commercially available off-the-shelf. However, owing to their low complexity, there are several artifacts that one encounters in the depth map like holes, mis-alignment between the depth map and color image and lack of sharp object boundaries in the depth map. This is a potential problem in applications that require the color image to be projected in 3-D using the depth map. Such applications depend heavily on the depth map and thus the quality of the depth map is of vital importance. In this paper, a novel multi-resolution anisotropic diffusion based algorithm is presented that accepts a Kinect generated depth map and color image and computes a dense depth map in which the holes have been filled and the edges of the objects are sharpened and aligned with the objects in the color image. The proposed algorithm also ensures that regions in the depth map where the depth is properly estimated are not filtered and ensures that the depth values in the final depth map are the same values that existed in the original depth map. Experimental results are provided to demonstrate the improvement in the quality of the depth map and also execution time results are provided to prove that the proposed method can be executed in real-time.
Similar content being viewed by others
References
Fehn C (2003) A 3D-TV system based on video plus depth information. In: Proceedings conference record of the thirty-seventh asilomar conference on signals, systems and computers, vol 2, pp 1529–1533
Tian C, Lai P-L, Lopez P, Gomila C (2009) View synthesis techniques for 3D video. In: Proceedings of the SPIE, vol 7443, no 2, pp 74430T–74430T-11
Scharstein D, Szeliski R, Zabih R (2001) A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. In: Proceedings of the IEEE workshop on stereo and multi-baseline vision (SMBV 2001), pp 131–140
Zhang Z (2012) Microsoft Kinect sensor and its effect. In: IEEE multimedia, vol 19, no 2, pp 4–10
Tomasi C, Manduchi R (1998) Bilateral filtering for gray and color images. In: Proceedings of the sixth international conference on computer vision, pp 839–846
Kopf J, Cohen MF, Lichinski D, Uyttendaele M (2007) Joint bilateral upsampling. In: Proceedings of ACM transactions graph (SIGGRAPH ’07), vol 26, no 3, article 96
Matyunin S, Vatolin D, Berdnikov Y, Smirnov M (2011) Temporal filtering for depth maps generated by Kinect depth camera. In: Proceedings of 3DTV conference: the true vision - capture, transactions and display of 3D video (3DTV-CON ’11), pp 1–4
Berdnikov Y, Vatolin D (2011) Real-time depth map occlusion filling and scene background restoration for projected-pattern-based depth cameras. In: Proceedings of the 21st international conference on computer graphics and vision (GraphiCon ’11)
Camplani M, Salgado L (2012) Efficient spatio temporal hole filling strategy for Kinect depth maps. In: Proceedings of IS & T/SPIE international conference on 3D image process and applications, vol 8290, pp 82900E1–10
Miao C, Fu J, Lu Y, Li S, Chen CW (2012) Texture-assisted Kinect depth inpainting. In: Proceedings of the IEEE international symposium on circuits and systems (ISCAS ’12), pp 604–607
Xu K, Zhou J, Wang Z (2012) A method of hole-filling for the depth map generated by Kinect with moving objects detection. In: Proceedings of the IEEE international symposium on broadband multimedia systems and broadcasting (BMSB ’12), pp 1–5
Fu J, Wang S., Yan L, Shipeng Z, Zeng W (2012) Kinect-like depth denoising. In: Proceedinds of the IEEE international symposium on circuits and systems (ISCAS ’12), pp 512–515
Milani S, Calvagno G (2011) Joint denoising and interpolation of depth maps for MS Kinect sensors. In: Proceedings of the IEEE international conference on acoustics, speech and signal processing (ICASSP ’12), pp 797–800
Richardt C, Stoll C, Dodgson NA, Seidel H-P, Theobalt C (2012) Coherent spatiotemporal filtering, upsampling and rendering of RGBZ videos. In: Proceedings of eurgraphics
Park J, Kim H, Tai Y-W, Brown MS, Kweon I (2011) High quality depth map upsampling for 3D-TOF cameras. In: Proceedings of the internationl conference on computer vision (ICCV ’11), pp 1623–1630
Li Y, Sun L (2010) A novel upsampling scheme for depth map compression in 3DTV system. In: Proceedings of picture coding symposium (PCS ’10), pp 186–189
Ekmekcioglu E, Mrak M, Worral S, Kondoz A (2009) Utilisation of edge adaptive upsampling in compression of depth map videos for enhanced free-viewpoint rendering. In: Proceedings of international conference on image processing (ICIP ’09), pp 733–736
Ekmekcioglu E, Worral S, Kondoz A (2008) Bit-rate adaptive downsampling for the coding of multi-view video with depth information. In: Proceedings of the 3DTV conference: the true vision - capture, transmission and display of 3D video (3DTV-CON ’08), pp 137–140
Schwarz S, Olsson R, Sjostrom M, Tourancheau S (2012) Adaptive depth filtering for HEVC 3D video coding. In: Proceedings of picture coding symposium (PCS ’12), pp 49–52
Perona P, Malik J (1990) Scale-space and edge detection using anisotropic diffusion. In: IEEE transactions on pattern analysis and machine intelligence, vol 12, pp 629–639
Canny J (1986) A computational approach to edge detection. In: IEEE transactions pattern analysis and machine intelligence, vol 6, pp 679–698
Author information
Authors and Affiliations
Corresponding author
Additional information
This work was supported by the Technology Development Program for Commercializing System Semiconductor funded By the Ministry of Trade, industry & Energy (MOTIE, Korea). (No. 10041126, Title: International Collaborative R&BD Project for System Semiconductor)
Rights and permissions
About this article
Cite this article
Vijayanagar, K.R., Loghman, M. & Kim, J. Real-Time Refinement of Kinect Depth Maps using Multi-Resolution Anisotropic Diffusion. Mobile Netw Appl 19, 414–425 (2014). https://doi.org/10.1007/s11036-013-0458-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11036-013-0458-7