Abstract
Commodity-level scan cameras generally capture RGB-D image with depth missing or incorrect depth points if the surface of the object is transparent, bright, or black. These incorrect depth points are generated randomly and limit the downstream applications of raw RGB-D images. In this paper, we propose a coarse-to-fine method to detect and eliminate the incorrect depth points via RGB semantics. In our flowchart, deep learning-based networks are applied to predict the potential regions with incorrect depth points and the normals of the point cloud. Then we develop a three-step elimination method to remove the incorrect depth points in the regions. Experimental results show that our method leads to great improvements for downstream applications of RGB-D images, especially in depth completion application.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
C., D.H., Kannala, J., Heikkila, J.: Joint depth and color camera calibrationwith distortion correction. IEEE Trans. Pattern Anal. Mach. Intell. 34(10), 2058–2064 (2012).https://doi.org/10.1109/tpami.2012.125
Chan, D., Buisman, H., Theobalt, C., Thrun, S.: A noise-aware filter for real-time depth upsampling (2008)
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 833–851. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_49
Diebel, J., Thrun, S.: An application of Markov random fields to range sensing. In: Weiss, Y., Schölkopf, B., Platt, J.C. (eds.) Advances in Neural Information Processing Systems, vol. 18, pp. 291–298. MIT Press (2006)
Huang, Z., Fan, J., Cheng, S., Yi, S., Wang, X., Li, H.: HMS-net: hierarchical multi-scale sparsity-invariant network for sparse depth completion. IEEE Trans. Image Process. 29, 3429–3441 (2020). https://doi.org/10.1109/tpami.2012.125
Jaritz, M., Charette, R.D., Wirbel, E., Perrotton, X., Nashashibi, F.: Sparse and dense data with CNNs: depth completion and semantic segmentation. In: 2018 International Conference on 3D Vision (3DV). IEEE, September 2018. https://doi.org/10.1109/3dv.2018.00017
Kopf, J., Cohen, M.F., Lischinski, D., Uyttendaele, M.: Joint bilateral upsampling. In: ACM SIGGRAPH 2007 Papers on - SIGGRAPH 2007. ACM Press (2007). https://doi.org/10.1145/1275808.1276497
Lachat, E., Macher, H., Landes, T., Grussenmeyer, P.: Assessment and calibration of a RGB-d camera (Kinect v2 sensor) towards a potential use for close-range 3D modeling. Remote Sens. 7(10), 13070–13097 (2015). https://doi.org/10.3390/rs71013070
Liu, J., Gong, X.: Guided depth enhancement via anisotropic diffusion. In: Huet, B., Ngo, C.-W., Tang, J., Zhou, Z.-H., Hauptmann, A.G., Yan, S. (eds.) PCM 2013. LNCS, vol. 8294, pp. 408–417. Springer, Cham (2013). https://doi.org/10.1007/978-3-319-03731-8_38
Pagliari, D., Pinto, L.: Calibration of Kinect for XBOX one and comparison between the two generations of microsoft sensors. Sensors 15(11), 27569–27589 (2015). https://doi.org/10.3390/Pagliari_2015
Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., Toyama, K.: Digital photography with flash and no-flash image pairs. In: ACM SIGGRAPH 2004 Papers on - SIGGRAPH 2004. ACM Press (2004). https://doi.org/10.1145/1186562.1015777
Schall, O., Belyaev, A., Seidel, H.P.: Feature-preserving non-local denoising of static and time-varying range data. In: Proceedings of the 2007 ACM Symposium on Solid and Physical Modeling - SPM 2007. ACM Press (2007). https://doi.org/10.1145/1236246.1236277
Shan, D.R., Ke, Y.L.: Surface reconstruction from unorganized points based on 2D delaunay neighbors. In: Sui, W. (ed.) Second International Conference on Image and Graphics. SPIE, July 2002. https://doi.org/10.1117/12.477093
Uhrig, J., Schneider, N., Schneider, L., Franke, U., Brox, T., Geiger, A.: Sparsity invariant CNNs. In: 2017 International Conference on 3D Vision (3DV). IEEE, October 2017. https://doi.org/10.1109/3dv.2017.00012
Wang, X., Zhang, P., Zhang, Y., Ma, L., Kwong, S., Jiang, J.: Deep intensity guidance based compression artifacts reduction for depth map. J. Vis. Commun. Image Represent. 57, 234–242 (2018). https://doi.org/10.1016/j.jvcir.2018.11.008
Yan, S., et al.: DDRNet: depth map denoising and refinement for consumer depth cameras using cascaded CNNs. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11214, pp. 155–171. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01249-6_10
Zhang, Y., Funkhouser, T.: Deep depth completion of a single RGB-D image. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, June 2018. https://doi.org/10.1109/cvpr.2018.00026
Zhang, Y., et al.: Physically-based rendering for indoor scene understanding using convolutional neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, July 2017. https://doi.org/10.1109/cvpr.2017.537
Zhou, B., et al.: Semantic understanding of scenes through the ADE20k dataset. Int. J. Comput. Vis. 127(3), 302–321, December 2018. https://doi.org/10.1007/s11263-018-1140-0
Acknowledgement
This work was supported by the Nature Science Fund of Guangdong Province under Grant 2019A1515011793 and NSFC (No.61972160, 51978271, 61962021, 61876065).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Xian, C., Qian, K., Luo, G., Li, G., Lv, J. (2020). Elimination of Incorrect Depth Points for Depth Completion. In: Magnenat-Thalmann, N., et al. Advances in Computer Graphics. CGI 2020. Lecture Notes in Computer Science(), vol 12221. Springer, Cham. https://doi.org/10.1007/978-3-030-61864-3_21
Download citation
DOI: https://doi.org/10.1007/978-3-030-61864-3_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-61863-6
Online ISBN: 978-3-030-61864-3
eBook Packages: Computer ScienceComputer Science (R0)