Multimedia Tools and Applications

, Volume 76, Issue 3, pp 4211–4226 | Cite as

Texture edge-guided depth recovery for structured light-based depth sensor

  • Huiping Deng
  • Jin Wu
  • Lei Zhu
  • Zengqiang Yan
  • Li Yu
Article

Abstract

The emergence of depth sensor facilitates the real-time and low-cost depth capture. However, the quality of its depth map is still inadequate for further applications due to holes, noises and artifacts existing within its depth information. In this paper, we propose an iterative depth boundary refinement framework to recover Kinect depth map. We extract depth edges and detect the incorrect regions, and then re-fill the incorrect regions until the depth edges are consistent with color edges. In the incorrect region detection procedure, we propose a RGB-D data edge detection method inspired by the recently developed deep learning. In the depth in-painting procedure, we propose a priority-determined fill order in which the high confidence pixels and strong edges are assigned to high priority. The actual depth values are computed by using a weighted cost filter, in which color, spatial similarity measures and Gaussian error model are considered. Experimental results demonstrate that the proposed method provides sharp and clear edges for the Kinect depth, and depth edges are aligned with the color edges.

Keywords

3D video RGB-D data Depth map recovery Edge detection Depth edge alignment 

References

  1. 1.
    Bo LL, Ren X, Fox D (2011) A large-scale hierarchical multiview RGB-D object dataset. In Proc Int Conf Robot Autom, 1817–1824Google Scholar
  2. 2.
    Canny J (1986) A computational approach to edge detection. PAMI 8(6):679–698CrossRefGoogle Scholar
  3. 3.
    Caspi D, Kyriati N, Shamir J (1998) Range imaging with adaptive color structured light. IEEE Trans Pattern Anal Mach Intell 20(5):470–480CrossRefGoogle Scholar
  4. 4.
    Chen L, Lin H, Li S (2012) Depth image enhancement for kinect using region growing and bilateral filter. In Proc ICPR, 3070–3073Google Scholar
  5. 5.
    Dollár P, Zitnick C (2013) Structured forests for fast edge detection. ICCV, SydneyCrossRefGoogle Scholar
  6. 6.
    Dollár P, Zitnick C (2015) Fast edge detection using structured forests, PAMIGoogle Scholar
  7. 7.
    Gao Y, Tang J, Hong R, Yan S, Dai Q, Zhang N, Chua T-S (2012) Camera constraint-free view-based 3D object retrieval. IEEE Trans Image Process 21(4):2269–2281MathSciNetCrossRefGoogle Scholar
  8. 8.
    Gao Y, Wang M, Ji R, Wu X, Dai Q (2014) 3D object retrieval with hausdorff distance learning. IEEE Trans Ind Electron 61(4):2088–2098CrossRefGoogle Scholar
  9. 9.
    Gao Y, Wang M, Tao D, Ji R, Dai Q (2012) 3D object retrieval and recognition with hypergraph analysis. IEEE Trans Image Process 21(9):4290–303MathSciNetCrossRefGoogle Scholar
  10. 10.
    Hu J, Hu R, Wang Z, Gong Y, Duan M (2013) Color image guided locality regularized representation for Kinect depth holes filling, Visual Communications and Image Processing (VCIP), 1–6Google Scholar
  11. 11.
    Koninckx TP, Gool LV (2006) Real-time range acquisition by adaptive structured light. IEEE Trans Pattern Anal Mach Intell 28(3):432–445CrossRefGoogle Scholar
  12. 12.
    Liu M-Y, Tuzel O, Taguchi Y (2013) Joint geodesic upsampling of depth images. In Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), 169–176Google Scholar
  13. 13.
    Maimone A, Fuchs H (2011) Encumbrance-free telepresence system with real-time 3D capture and display using commodity depth cameras. In Proc 10th IEEE Int Symp Mixed Augmented Reality (ISMAR), 137–146Google Scholar
  14. 14.
    Martin D, Fowlkes C, Tal D, Malik J (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCVGoogle Scholar
  15. 15.
    Martin D, Fowlkes C, Malik J (2004) Learning to detect natural image boundaries using local brightness, color, and texture cues. PAMI 26(5):530–549CrossRefGoogle Scholar
  16. 16.
    Miao D, Fu J, Lu Y, Li S, Chen C (2012) Texture-assisted kinect depth inpainting. In Proc ISCAS, 604–607Google Scholar
  17. 17.
    Milani S, Calvagno G (2012) Joint denoising and interpolation of depth maps for MS Kinect sensors. In Proc ICASSP, 797–800Google Scholar
  18. 18.
    Min D, Lu J, Do MN (2012) Depth video enhancement based on weighted mode filtering. IEEE Trans Image Process 21(3):1176–1190MathSciNetCrossRefGoogle Scholar
  19. 19.
    Nowozin S, Lampert CH (2011) Structured learning and prediction in computer vision. Foundations Trends Comput Graphics Vision 6:185–365CrossRefMATHGoogle Scholar
  20. 20.
    Shen J, Cheung S-C S (2013) Layer depth denoising and completion for structured-light RGB-D cameras. In Proc IEEE Con- ference Computer Vision Pattern Recognition, IEEE, 1187–1194Google Scholar
  21. 21.
    Silberman N, Fergus R (2011) Indoor scene segmentation using a structured light sensor. In ICCV Workshop 3D Representation RecognitionGoogle Scholar
  22. 22.
    Xiang S, Yu L, Yang Y, Liu Q, Zhou J (2015) Interfered depth map recovery with texture guidance for multiple structured light depth cameras. Signal Process Image Commun 31(2015):34–46CrossRefGoogle Scholar
  23. 23.
    Xu Y, Jin X, Dai Q (2014) Spatial-temporal depth de-noising for Kinect based on texture edge-assisted depth classification. In Proc 19th International Conference on Digital Signal Processing (DSP), 327–332Google Scholar
  24. 24.
    Yang Q, Tan K, Culbertson B, Apostolopoulos J (2010) Fusion of active and passive sensors for fast 3D capture. In Proc IEEE Int Workshop Multimedia Signal Process (MMSP), 69–74Google Scholar
  25. 25.
    Yang Q, Yang R, Davis J, Nistér D (2007) Spatial-depth super resolution for range images. In Proc IEEE Comput Vis Pattern Recognit (CVPR), 1–8Google Scholar
  26. 26.
    Yang J, Ye X, Li K, Hou C, Wang Y (2014) Color-guided depth recovery from RGB-D data using an adaptive autoregressive model. IEEE Trans Image Process 23(8):3443–3458MathSciNetCrossRefGoogle Scholar
  27. 27.
    Yao Y, Fu Y (2014) Contour model based hand-gesture recognition using Kinect sensor. IEEE Trans Circuits Syst Video Technol 24(11):1935–1944CrossRefGoogle Scholar
  28. 28.
    Ziou D, Tabbone S et al (1998) Edge detection techniques-an overview. Pattern Recognition Image Analysis 8:537–559Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Huiping Deng
    • 1
  • Jin Wu
    • 1
  • Lei Zhu
    • 1
  • Zengqiang Yan
    • 2
  • Li Yu
    • 2
  1. 1.School of Inf. Sci. and EngWuhan University of Science and TechnologyWuhanChina
  2. 2.School of Elec. Inf. and CommHuazhong University of Science and TechnologyWuhanChina

Personalised recommendations