Advertisement

Journal of Signal Processing Systems

, Volume 75, Issue 1, pp 23–37 | Cite as

A High Quality Depth Map Upsampling Method Robust to Misalignment of Depth and Color Boundaries

  • Jaekwang Kim
  • Jaeho Lee
  • Seung-Ryong Han
  • Dowan Kim
  • Jongsul Min
  • Changick KimEmail author
Article

Abstract

In recent years, fusion camera systems that consist of color cameras and Time-of-Flight (TOF) depth sensors have been popularly used due to its depth sensing capability at real-time frame rates. However, captured depth maps are limited in low resolution compared to the corresponding color images due to physical limitation of the TOF depth sensor. Most approaches to enhancing the resolution of captured depth maps depend on the implicit assumption that when neighboring pixels in the color image have similar values, they are also similar in depth. Although many algorithms have been proposed, they still yield erroneous results, especially when region boundaries in the depth map and the color image are not aligned. We therefore propose a novel kernel regression framework to generate the high quality depth map. Our proposed filter is based on the vector pointing similar pixels that represents the unit vector toward similar neighbors in the local region. The vectors are used to detect misaligned regions between color edges and depth edges. Unlike conventional kernel regression methods, our method properly handles misaligned regions by introducing the numerical analysis of the local structure into the kernel regression framework. Experimental comparisons with other data fusion techniques prove the superiority of the proposed algorithm.

Keywords

Depth map upsampling Fusion camera system TOF depth sensor Trilateral filter 

Notes

Acknowledgement

This research was supported by a grant from the R&D Program (Industrial Strategic Technology Development) funded by the Ministry of Knowledge Economy (MKE), Republic of Korea. Also, The authors are deeply thankful to all interested persons of MKE and KEIT (Korea Evaluation Institute of Industrial Technology).

References

  1. 1.
    Boehme, M., Haker, M., Martinetz, T., & Barth, E. (2008). A facial feature tracker for human-computer interaction based on 3D ToF cameras. Journal of Intelligent Systems Technologies and Applications, 5(3–4), 264–273.CrossRefGoogle Scholar
  2. 2.
    Kollorz, E., Penne, J., Hornegger, J., & Barke, A. (2008). Gesture recognition with a time-of-flight camera. Journal of Intelligent Systems Technologies and Applications, 5(3–4), 334–343.CrossRefGoogle Scholar
  3. 3.
    Inoue, H., Tachikawa, T., & Inaba, M. (1992). Robot vision system with a correlation chip for real-time tracking, optical flow and depth map generation. Proceedings of the IEEE International Conference on Robotics and Automation, 5, 1621–1626.Google Scholar
  4. 4.
    Hussmann, S., & Liepert, T. (2009). Three-dimensional TOF robot vision system. IEEE Transactions on Instrumentation and Measurement, 58(1), 141–146.CrossRefGoogle Scholar
  5. 5.
    Kim, Y.M., Theobalt, C., Diebel, J., Kosecka, J., Micusik, B., & Thrun, S. (2009). Multi-view image and ToF sensor fusion for dense 3D reconstruction. Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCV Workshops). pp. 1542-1549.Google Scholar
  6. 6.
    Jang, I. Y., & Lee, K. H. (2010). Depth video based human model reconstruction resolving self-occlusion. IEEE Transactions on Consumer Electronics, 56(3), 1933–1941.CrossRefGoogle Scholar
  7. 7.
    Curless, B. (1997). New methods for surface reconstruction from range image. Ph.D. dissertation, Technical Report CSLTR-97-733, Stanford University.Google Scholar
  8. 8.
    Cheng, X., Zhang, H., & Xie, R. (2010). Study on 3D laser scanning modeling method for large-scale history building. Proceedings of the International Conference on Computer Application and System Modeling (ICCASM), 7, 573–577.Google Scholar
  9. 9.
    Nagahara, D., & Takahashi, S. (2010). Mobile robot control based on information of the scanning laser range sensor. Proceedings of the 11th IEEE International Workshop on Advanced Motion Control. pp. 258–261.Google Scholar
  10. 10.
    Smith, P. W., Nandhakumar, N., & Chien, C. H. (1997). Object motion and structure recovery for robotic vision using scanning laser range sensors. IEEE Transactions on Robotics and Automation, 13(1), 74–80.CrossRefGoogle Scholar
  11. 11.
    Yoon, K. J., & Kweon, I. S. (2006). Adaptive support-weight approach for correspondence search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4), 650–656.CrossRefGoogle Scholar
  12. 12.
    Boykov, Y., Veksler, O., & Zabih, R. (1998). A variable window approach to early vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(12), 1283–1294.CrossRefGoogle Scholar
  13. 13.
    Ohta, Y., & Kanade, T. (1985). Stereo by inter- and inter-scanline search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 7(2), 139–154.CrossRefGoogle Scholar
  14. 14.
    Diebel, J., & Thrun, S. (2005). An application of markov random fields to range sensing. Advances in Neural Information Processing Systems, 18, 291–298.Google Scholar
  15. 15.
    Zhu, J., Wang, L., Yang, R., & Davis, J. (2008). Fusion of time-of-flight depth and stereo for high accuracy depth maps. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition. pp. 1–8.Google Scholar
  16. 16.
    Zhu, J., Wang, L., Gao, J., & Yang, R. (2010). Spatial-temporal fusion for high accuracy depth maps using dynamic MRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(5), 899–909.CrossRefGoogle Scholar
  17. 17.
    Kopf, J., Cohen, M., Lischinski, D., & Uyttendaele, M. (2007). Joint bilateral upsampling. ACM Transactions on Graphics, 26, 3.Google Scholar
  18. 18.
    Chan, D., Buisman, H., Theobalt, C., & Thrun, S. (2008). A noise-aware filter for real-time depth upsampling. Proceedings of ECCV Workshop on Multicamera and Multimodal Sensor Fusion Algorithms and Applications. pp. 1–12.Google Scholar
  19. 19.
    Yang, Q. X., Yang, R. G., Davis, J. E., & Nister, D. (2007). Spatial-depth super resolution for range images. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition. pp. 1–8.Google Scholar
  20. 20.
    Riemens, A. K., Gangwal, O. P., Barenbrug, B., & Berretty, R-P. M. (2009). Multi-step joint bilateral depth upsampling. Proceedings of the SPIE (Visual Communications and Image signal Processing). 7257.Google Scholar
  21. 21.
    Garcia, F., Mirbach, B., Ottersten, B., Grandidier, F., & Cuesta, A. (2010). Pixel weighted average strategy for depth sensor data fusion. Proceedings of the IEEE International Conference on Image Processing, pp. 2805–2808.Google Scholar
  22. 22.
    Takeda, H., Farsiu, S., & Milanfar, P. (2007). Kernel regression for image processing and reconstruction. IEEE Transactions on Image Processing, 16(2), 349–366.CrossRefMathSciNetGoogle Scholar
  23. 23.
    Eisemann, E., & Durand, F. (2004). Flash photography enhancement via intrinsic relighting. ACM Transactions on Graphics, 23(3), 673–678.CrossRefGoogle Scholar
  24. 24.
    Yu, H., Zhao, L., & Wang, H. (2009). Image denoising using trivariate shrinkage filter in the wavelet domain and joint bilateral filter in the spatial domain. IEEE Transactions on Image Processing, 18(10), 2364–2369.CrossRefMathSciNetGoogle Scholar
  25. 25.
    Park, J., Kim, H., Tai, Y. W., Brown, M. S., & Kwon, I. (2011) High Quality Depth Map Upsampling for 3D-ToF Cameras. Proceedings of the IEEE International Conference on Computer Vision. pp. 1623–1630.Google Scholar
  26. 26.
    Kim, D., & Yoon, K. (2012). High-quality Depth Map Up-sampling Robust to Edge Noise of Range Sensors. Proceedings of the IEEE International Conference on Image Processing. pp. 553–556.Google Scholar
  27. 27.
    Tomasi, E. C., & Manduchi, R. (1998). Bilateral filtering for gray and color images. Proceedings of the IEEE International Conference on Compute Vision. pp. 836–846.Google Scholar
  28. 28.
    Zhang, B., & Allebach, J. P. (2008). Adaptive bilateral filter for sharpness enhancement and noise removal. IEEE Transactions on Image Processing, 17(5), 664–678.CrossRefMathSciNetGoogle Scholar
  29. 29.
    Lie, W. N., Chen, C. Y., & Chen, W. C. (2011). 2D to 3D video conversion with Key-frame depth propagation and trilateral filtering. Electronics Letters, 47(5), 319–321.CrossRefMathSciNetGoogle Scholar
  30. 30.
    Varekamp, C., & Barenbrug, B. (2007). Improved depth propagation for 2D to 3D video conversion using key-frames. Proceedings of the 4th European Conference on Visual Media production (IETCVMP). pp. 1–7.Google Scholar
  31. 31.
    Rafael, C. G., & Richard, E. W. (2002). Digital Image Processing. Prentice Hall. 2nd edition.Google Scholar
  32. 32.
    Porikli, F. (2008). Constant Time O(1) Bilateral Filtering. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, pp. 1–8.Google Scholar
  33. 33.
    Paris, S., & Durand, F. (2006) A fast approximation of the bilateral filter using a signal processing approach. International Journal of Computer Vision, 81(1), 24–52.Google Scholar
  34. 34.
    Yang, Q., Tan, K. H., & Ahuja, N. (2009). Real Time O(1) Bilateral Filtering. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition. pp. 557–564.Google Scholar
  35. 35.
    Viola, P., & Joens, M. (2001). Robust real-time face detection. Proceedings of the IEEE International Conference on Computer Vision, pp. 747–750.Google Scholar
  36. 36.
    Crow, F. (1984). Summed-area tables for texture mapping. Proceedings of the Siggraph, 18(no. 3), 207–212.CrossRefGoogle Scholar
  37. 37.
    Deriche, R. (1992). Recursively implementing the Gaussian and its derivatives. Proceeding IEEE International Conference on Image Processing. pp. 263–267.Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Jaekwang Kim
    • 1
  • Jaeho Lee
    • 2
  • Seung-Ryong Han
    • 3
  • Dowan Kim
    • 3
  • Jongsul Min
    • 3
  • Changick Kim
    • 2
    Email author
  1. 1.R&D DivisionHyundai MotorsGyeonggi-doKorea
  2. 2.EE413, IT Convergence Center (N1)Korea Advanced Institute of Science and Technology (KAIST)DaejeonKorea
  3. 3.DMC R&D CenterSamsung ElectronicsGyeonggi-doKorea

Personalised recommendations