Skip to main content
SpringerLink
Log in

Manifold-defect depth-map restoration for very low-cost S3D videos

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper, the proposed algorithm provides a fluent and efficient method for repairing very-low quality depth maps of considerable manifold defects for low-cost stereoscopic 3D (S3D) photographing that such a depth map can be easily yielded by 1st-generation Kinect (Kinect-v1). The corresponding framework cascades two repairing portions named discriminative non-segmentation hole filling and edge rectification-by-deforming. The former can discriminatively fill a variety of depth-invalid holes with no need of practically making attribute-discrimination and target segmentation for depth holes. The main portions of the latter contain edge-shifting-rectification and texture-edge guided dual processing for tailoring possible twisted depth edges. Since the ingredients of proposed algorithm are compactly concatenated according to intimate context, most troublesome defects in Kinect-v1 depths can be tackled. Particularly, the proposed algorithm can obtain the restoring coherency for successive depth maps. A series of experimental results under various photographing scenarios can demonstrate that a single Kinect-v1 device can fast tell on the development of very low-cost S3D imaging tool by the proposed algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Bhavsar AV, Rajagopalan AN (2012) Range map superresolution-inpainting and reconstruction from sparse data. Comp. Vis. Image Understand 116:572–591

    Article  Google Scholar 

  2. Blais F (2004) Review of 20 years of range sensor development. Journal of Electronic Imaging 13(1):231–243

    Article  Google Scholar 

  3. Chang N, Tsai TH, Hsu BH, Chen YC, Chang TS (2010) Algorithm and architecture of disparity estimation with mini-census adaptive support weight. IEEE Trans Circuits Syst Video Technol 20(6):792–805

    Article  Google Scholar 

  4. Chen L, Lin H, Li S (2012) Depth image enhancement for Kinect using region growing and bilateral filter. 21st Int’l Conf Patt Recog (ICPR 2012) 3070–3073

  5. Chiu WC, Blanke U, Fritz M (2010) Improving the Kinect by cross-modal stereo. 22nd Bri Mach Vis Conf (BMVC 2011) 116.1–116.10

  6. Ferstl D, Ranftl R, Ruther M, Bischof H (2013) Multi-modality depth map fusion using primal-dual Optimization. IEEE Int’l Conf Comput Photo 1–8

  7. Fu J, Wang S, Liu Y, Li S, Zeng W (2012) Kinect-like depth denoising. IEEE Int’l Symp Circ Syst (ISCAS) 512–515

  8. He K, Sun J, Tang X (2013) Guided image filtering. IEEE Trans Pattern Anal Mach Intell 35:1397–1409

    Article  Google Scholar 

  9. Ho YS, Lee EK (2010) Generation of multi-view video using a fusion camera system for 3D displays. IEEE Trans on Consumer Electronics 56(4):2797–2805

    Article  Google Scholar 

  10. Jiao J, Wang R, Wang W, Li D, Gao W (2017) Color image-guided boundary-inconsistent region refinement for stereo matching. IEEE Trans on Circuits Syst Video Technol 27:1155–1159

    Article  Google Scholar 

  11. Lai P, Tian D, Lopez P (2010) Depth map processing with iterative joint multilateral filtering. Pic Coding Symp (PCS) 9–12

  12. Lee PJ, Effendi (2011) Nongeometric distortion smoothing approach for depth map preprocessing. IEEE Trans on Multimedia 13:246–254

    Article  Google Scholar 

  13. Liu Y, Nie L, Liu L, Rosenblum DS (2016) From action to activity: sensor-based activity recognition. Neurocomputing 181:108–115

    Article  Google Scholar 

  14. Liu Y, Zhang L, Nie L, Yan Y, and Rosenblum DS (2016) Fortune teller: predicting your career path. Proc AAAI'16 Proc Thirtieth AAAI Conf Artif Intel 201–207

  15. Liu W, Chen X, Yang J, Wu Q (2017) Robust color guided depth map restoration. IEEE Trans. on Image Process. 26:315–327

    Article  MathSciNet  Google Scholar 

  16. Matyunin S, Vatolin D, Berdnikov Y, Smirnov M (2011) Temporal filtering for depth maps generated by Kinect depth camera. 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video 1–4

  17. Miao D, Fu J, Lu Y, Li S, Chen CW (2012) Texture-assisted Kinect depth inpainting. IEEE int’l Symp Circ Syst (ISCAS’12) 604–607

  18. Riegler G, Ranftl R, Rüther M, Pock T, Bischof, H (2015) Depth restoration via joint training of a global regression model and CNNs. Br Mach Vis Conf (BMVC) 58

  19. Sheng L, Ngan KN (2013) Depth enhancement based on hybrid geometric hole filling strategy. Proc 20th IEEE Int Conf Image Proc 2173–2176

  20. Sheng L, Ngan KN, Lim CL, Li S (2015) Online temporally consistent indoor depth video enhancement via static structure. IEEE Trans on Image Process 24:2197–2211

    Article  MathSciNet  Google Scholar 

  21. Song X, Huang H, Zhong F, Ma X, Qin X, (2016) Edge-guided depth map enhancement. 23rd Int Conf Patt Recog (ICPR) 2758–2763

  22. Sun J, Zhang NN, Shum HY (2003) Stereo matching using belief propagation. IEEE Trans on Pattern Anal Mach Intell 25:787–800

    Article  Google Scholar 

  23. Suzuki T, Ikenaga T (2012) SIFT-based low complexity keypoint extraction and its real-time hardware implementation for full-HD video. In 2012 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) 1–6

  24. Wang Y, Zhong F, Peng Q, Qin X (2014) Depth map enhancement based on color and depth consistency. Vis Comput 30:1157–1168

    Article  Google Scholar 

  25. Wang C, Chan SC, Zhu ZY, Zhang L, Shum HY (2018) Superpixel-based color–depth restoration and dynamic environment modeling for Kinect-assisted image-based rendering systems. Vis Comput 34(1):67–81

    Article  Google Scholar 

  26. Yang J, Ye X, Li K, Hou C, Wang Y (2014) Color-guided depth recovery from RGB-D data using an adaptive autoregressive model. IEEE Trans Image Process 23(8):3443–3458

    Article  MathSciNet  Google Scholar 

  27. Yoon KJ, Kweon IS (2005) Locally adaptive support-weight approach for visual correspondence search. IEEE Conf Computer Vision and Pattern Recognition 28(4):650–656

    Google Scholar 

  28. Zhu J, Wang L, Yang R, Davis JE, Pan Z (2011) Reliability fusion of time-of-flight depth and stereo geometry for high quality depth maps. IEEE Trans on Pattern Anal 33(7):1400–1414

    Article  Google Scholar 

Download references

Acknowledgements

The foundation of investigation was supported by the Ministry of Science and Technology Foundation of Taiwan, ROC with plan numbered 107-2221-E-415-016-MY2.

Author information

Authors and Affiliations

  1. Department of Computer Science and Information Engineering, National Chia-Yi University, Chiayi City, Taiwan, Republic of China

    Din-Yuen Chan & Ji-Rong Wu

Authors
  1. Din-Yuen Chan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ji-Rong Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Din-Yuen Chan.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chan, DY., Wu, JR. Manifold-defect depth-map restoration for very low-cost S3D videos. Multimed Tools Appl 79, 8863–8886 (2020). https://doi.org/10.1007/s11042-018-6804-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6804-9

Keywords

Search

Navigation