Abstract
This paper proposes an image-guided depth super-resolution framework to improve the quality of depth map captured by low-cost depth sensors, like the Microsoft Kinect. First, a contour-guided fast marching method is proposed to preprocess the raw depth map for recovering the missing data. Then, by using the non-local total generalized variation (NL-TGV) regularization, a convex optimization model is constructed to up-sample the preprocessed depth map to a high-resolution one. To preserve the sharpness of depth discontinuities, the color image and its multi-level segmentation information are utilized to assign the weights within the NL-TGV through a novel weight combining scheme. The texture energy from color image and local structure coherence around neighbor pixels in low-resolution depth map are applied to adjust the combination weights for further suppressing texture-transfer. Quantitative and qualitative evaluations of the proposed method on the Middlebury datasets and real-sensor datasets show the promising results in quality.
Similar content being viewed by others
References
Arbelaez P, Maire M, Fowlkes C, Malik J (2011) Contour detection and hierarchical image segmentation. IEEE Trans Pattern Anal Mach Intell 33(5):898–916
Buades A, Coll B, Morel J-M (2011) Non-local means denoising. Image Processing On Line 1:6
Chambolle A, Pock T (2011) A first-order primal-dual algorithm for convex problems with applications to imaging. J Math Imaging Vis 40(1):120–145
Chan D, Buisman H, Theobalt C, Thrun S (2008) A noise-aware filter for real-time depth upsampling Proc. ECCV workshop multi-camera multi-modal sensor fus. Algorithms Appl.
Cui Y, Schuon S, Chan D, Thrun S, Theobalt C (2010) 3d shape scanning with a time-of-flight camera Proc. Comput. Vis. Pattern Recognit (CVPR), pp 1173–1180
Diebel J, Thrun S (2005) An application of markov random fields to range sensing Advances in neural information processing systems (NIPS), vol 5, pp 291–298
Dollár P, Zitnick CL (2015) Fast edge detection using structured forests. IEEE Trans Pattern Anal Mach Intell 37(8):1558–1570
Ferstl D, Reinbacher C, Ranftl R, Rüther M, Bischof H (2013) Image guided depth upsampling using anisotropic total generalized variation Proc Int. Conf. Comput. Vis. (ICCV), pp 993–1000
Ferstl D, Ruther M, Bischof H (2015) Variational depth superresolution using example-based edge representations Proc Int. Conf. Comput. Vis. (ICCV), pp 513–521
He K, Sun J, Tang X (2013) Guided image filtering. IEEE Trans Pattern Anal Mach Intell 35(6):1397–1409
Hui T-W, Loy CC, Tang X (2016) Depth map super-resolution by deep multi-scale guidance Proc Eur. Conf. Comput. Vis. (ECCV). Springer, pp 353–369
Kiechle M, Hawe S, Kleinsteuber M (2013) A joint intensity and depth co-sparse analysis model for depth map super-resolution Proc Int. Conf. Comput. Vis. (ICCV), pp 1545–1552
Kopf J, Cohen MF, Lischinski D, Uyttendaele M (2007) Joint bilateral upsampling. ACM Trans Graph 26(3):96
Kwon H, Tai Y-W, Lin S (2015) Data-driven depth map refinement via multi-scale sparse representation Proc. Comput. Vis. Pattern Recognit (CVPR), pp 159–167
Li Y, Min D, Do MN, Lu J (2016) Fast guided global interpolation for depth and motion Proc Eur. Conf. Comput. Vis. (ECCV). Springer, pp 717–733
Liu M-Y, Tuzel O, Taguchi Y (2013) Joint geodesic upsampling of depth images Proc. Comput. Vis. Pattern Recognit (CVPR), pp 169–176
Liu W, Jia S, Li P, Chen X, Yang J, Wu Q (2015) An mrf-based depth upsampling: upsample the depth map with its own property. IEEE Signal Process Lett 22(10):1708–1712
Lu J, Shi K, Min D, Lin L, Do MN (2012) Cross-based local multipoint filtering Proc. Comput. Vis. Pattern Recognit (CVPR), pp 430–437
Min D, Lu J, Do MN (2012) Depth video enhancement based on weighted mode filtering. IEEE Trans Image Process 21(3):1176–1190
Min D, Choi S, Lu J, Ham B, Sohn K, Do MN (2014) Fast global image smoothing based on weighted least squares. IEEE Trans Image Process 23(12):5638–5653
Nair R, Lenzen F, Meister S, Schäfer H, Garbe C, Kondermann D (2012) High accuracy tof and stereo sensor fusion at interactive rates Proc Eur. Conf. Comput. Vis. (ECCV), pp 1–11
Nair R, Ruhl K, Lenzen F, Meister S, Schäfer H, Garbe CS, Eisemann M, Magnor M, Kondermann D (2013) A survey on time-of-flight stereo fusion Time-of-Flight and depth imaging. Sensors, Algorithms, and Applications. Springer, pp 105–127
Parikh N, Boyd SP (2014) Proximal algorithms. Found Trends Optim 1 (3):127–239
Park J, Kim H, Tai Y-W, Brown MS, Kweon I (2011) High quality depth map upsampling for 3d-tof cameras Proc Int. Conf. Comput. Vis. (ICCV). IEEE, pp 1623–1630
Park J, Kim H, Tai Y-W, Brown MS, Kweon IS (2014) High-quality depth map upsampling and completion for rgb-d cameras. IEEE Trans Image Process 23(12):5559–5572
Pock T, Chambolle A (2011) Diagonal preconditioning for first order primal-dual algorithms in convex optimization Proc Int. Conf. Comput. Vis. (ICCV), pp 1762–1769
Ranftl R, Bredies K, Pock T (2014) Non-local total generalized variation for optical flow estimation Proc Eur. Conf. Comput. Vis. (ECCV), pp 439–454
Riegler G, Rüther M, Bischof H (2016) Atgv-net: accurate depth super-resolution Proc Eur. Conf. Comput. Vis. (ECCV). Springer, pp 268–284
Scharstein D, Pal C (2007) Learning conditional random fields for stereo Proc. Comput. Vis. Pattern Recognit (CVPR), pp 1–8
Schuon S, Theobalt C, Davis J, Thrun S (2008) High-quality scanning using time-of-flight depth superresolution Proc. Comput. Vis. Pattern Recognit Workshop. (CVPRW), pp 1–7
Schuon S, Theobalt C, Davis J, Thrun S (2009) Lidarboost: Depth superresolution for tof 3d shape scanning Proc. Comput. Vis. Pattern Recognit. (CVPR), pp 343–350
Tan X, Sun C, Pham TD (2014) Multipoint filtering with local polynomial approximation and range guidance Proc. Comput. Vis. Pattern Recognit (CVPR), pp 2941–2948
Telea A (2004) An image inpainting technique based on the fast marching method. J Graphics Tools 9(1):23–34
Wang L, Jin H, Yang R, Gong M (2008) Stereoscopic inpainting: joint color and depth completion from stereo images Proc. Comput. Vis. Pattern Recognit (CVPR), pp 1–8
Xiaofeng R, Bo L (2012) Discriminatively trained sparse code gradients for contour detection Advances in neural information processing systems (NIPS), pp 584–592
Yang Q, Yang R, Davis J, Nistér D (2007) Spatial-depth super resolution for range images Proc. Comput. Vis. Pattern Recognit (CVPR), pp 1–8
Yang Q, Ahuja N, Yang R, Tan K-H, Davis J, Culbertson B, Apostolopoulos J, Wang G (2013) Fusion of median and bilateral filtering for range image upsampling. IEEE Trans Image Process 22(12):4841–4852
Yang J, Ye X, Li K, Hou C, Wang Y (2014) Color-guided depth recovery from rgb-d data using an adaptive autoregressive model. IEEE Trans Image Process 23 (8):3443–3458
Yu L-F, Yeung S-K, Tai Y-W, Lin S (2013) Shading-based shape refinement of rgb-d images Proc. Comput. Vis. Pattern Recognit (CVPR), pp 1415–1422
Zhang S, Wang C, Chan S-C (2015) A new high resolution depth map estimation system using stereo vision and kinect depth sensing. J Signal Process Sys 79(1):19–31
Zhu J, Wang L, Yang R, Davis JE, Pan Z (2011) Reliability fusion of time-of-flight depth and stereo geometry for high quality depth maps. IEEE Trans Pattern Anal Mach Intell 33(7):1400–1414
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Zhang, HT., Yu, J. & Wang, ZF. Probability contour guided depth map inpainting and superresolution using non-local total generalized variation. Multimed Tools Appl 77, 9003–9020 (2018). https://doi.org/10.1007/s11042-017-4791-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-017-4791-x