Skip to main content
SpringerLink
Log in

Depth Reconstruction of Translucent Objects from a Single Time-of-Flight Camera Using Deep Residual Networks

  • Conference paper
  • First Online:
Computer Vision – ACCV 2018 (ACCV 2018)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 11365))

Included in the following conference series:

Abstract

We propose a novel approach to recovering translucent objects from a single time-of-flight (ToF) depth camera using deep residual networks. When recording translucent objects using the ToF depth camera, their depth values are severely contaminated due to complex light interactions with surrounding environment. While existing methods suggested new capture systems or developed the depth distortion models, their solutions were less practical because of strict assumptions or heavy computational complexity. In this paper, we adopt deep residual networks for modeling the ToF depth distortion caused by translucency. To fully utilize both the local and semantic information of objects, multi-scale patches are used to predict the depth value. Based on the quantitative and qualitative evaluation on our benchmark database, we show the effectiveness and robustness of the proposed algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Belhedi, A., Bartoli, A., Bourgeois, S., Gay-Bellile, V., Hamrouni, K., Sayd, P.: Noise modelling in time-of-flight sensors with application to depth noise removal and uncertainty estimation in three-dimensional measurement. IET Comput. Vis. 9(6), 967–977 (2015)

    Article  Google Scholar 

  2. Cao, Y., Wu, Z., Shen, C.: Estimating depth from monocular images asclassification using deep fully convolutional residual networks. IEEE Trans. Circ. Syst. Video Technol. 28(11), 3174–3182 (2017)

    Article  Google Scholar 

  3. Clark, J., Trucco, E., Wolff, L.B.: Using light polarization in laser scanning. Image Vis. Comput. 15(2), 107–117 (1997)

    Article  Google Scholar 

  4. Feigin, M., Bhandari, A., Izadi, S., Rhemann, C., Schmidt, M., Raskar, R.: Resolving multipath interference in kinect: an inverse problem approach. IEEE Sens. J. 16, 3419–3427 (2015)

    Article  Google Scholar 

  5. Francesco, F.S.R., Christoph, R., Murthy, K.A.P., Vladimir, T., David, K., Shahram, I.: Determining depth from structured light using trained classifiers. https://lens.org/103-012-483-114-400

  6. Fujimura, Y., Iiyama, M., Hashimoto, A., Minoh, M.: Photometric stereo in participating media considering shape-dependent forward scatter. arXiv preprint arXiv:1804.02836 (2018)

  7. Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2414–2423 (2016)

    Google Scholar 

  8. Hartmann, W., Galliani, S., Havlena, M., Van Gool, L., Schindler, K.: Learned multi-patch similarity. In: International Conference on Computer Vision (ICCV) 2017 (2017)

    Google Scholar 

  9. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  10. Hullin, M.B., Fuchs, M., Ihrke, I., Seidel, H.P., Lensch, H.P.: Fluorescent immersion range scanning. ACM Trans. Graph. 27(3), 87–1 (2008)

    Article  Google Scholar 

  11. Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graph. (TOG) 36(4), 107 (2017)

    Article  Google Scholar 

  12. Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, pp. 448–456 (2015)

    Google Scholar 

  13. Kim, J., Reshetouski, I., Ghosh, A.: Acquiring axially-symmetric transparent objects using single-view transmission imaging. In: 30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

    Google Scholar 

  14. Kim, K., Shim, H.: Robust approach to reconstructing transparent objects using a time-of-flight depth camera. Opt. Exp. 25(3), 2666–2676 (2017)

    Article  Google Scholar 

  15. Kohler, J., Daneshmand, H., Lucchi, A., Zhou, M., Neymeyr, K., Hofmann, T.: Towards a theoretical understanding of batch normalization. arXiv preprint arXiv:1805.10694 (2018)

  16. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)

    Google Scholar 

  17. Lee, J.H., Heo, M., Kim, K.R., Kim, C.S.: Single-image depth estimation based on fourier domain analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 330–339 (2018)

    Google Scholar 

  18. Li, B., Dai, Y., Chen, H., He, M.: Single image depth estimation by dilated deep residual convolutional neural network and soft-weight-sum inference. arXiv preprint arXiv:1705.00534 (2017)

  19. Liao, J., Yao, Y., Yuan, L., Hua, G., Kang, S.B.: Visual attribute transfer through deep image analogy. arXiv preprint arXiv:1705.01088 (2017)

  20. Lysenkov, I., Eruhimov, V., Bradski, G.: Recognition and pose estimation of rigid transparent objects with a kinect sensor. Robotics 273, 273–280 (2013)

    Google Scholar 

  21. Maeno, K., Nagahara, H., Shimada, A., Taniguchi, R.I.: Light field distortion feature for transparent object recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2786–2793. IEEE (2013)

    Google Scholar 

  22. Marco, J., et al.: Deeptof: off-the-shelf real-time correction of multipath interference in time-of-flight imaging. ACM Trans. Graph. (TOG) 36(6), 219 (2017)

    Article  Google Scholar 

  23. Nah, S., Kim, T.H., Lee, K.M.: Deep multi-scale convolutional neural network for dynamic scene deblurring, vol. 3. arXiv preprint arXiv:1612.02177 (2016)

  24. Naik, N., Kadambi, A., Rhemann, C., Izadi, S., Raskar, R., Bing Kang, S.: A light transport model for mitigating multipath interference in time-of-flight sensors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 73–81 (2015)

    Google Scholar 

  25. Santurkar, S., Tsipras, D., Ilyas, A., Madry, A.: How does batch normalization help optimization? (no, it is not about internal covariate shift). arXiv preprint arXiv:1805.11604 (2018)

  26. Sarbolandi, H., Lefloch, D., Kolb, A.: Kinect range sensing: structured-light versus time-of-flight kinect. Comput. Vis. Image Underst. 139, 1–20 (2015)

    Article  Google Scholar 

  27. Seib, V., Barthen, A., Marohn, P., Paulus, D.: Friend or foe: exploiting sensor failures for transparent object localization and classification. In: International Conference on Robotics and Machine Vision, vol. 10253, p. 102530I. International Society for Optics and Photonics (2017)

    Google Scholar 

  28. Shim, H., Lee, S.: Recovering translucent objects using a single time-of-flight depth camera. IEEE Trans. Circ. Syst. Video Technol. 26(5), 841–854 (2016)

    Article  Google Scholar 

  29. Su, S., Heide, F., Wetzstein, G., Heidrich, W.: Deep end-to-end time-of-flight imaging. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6383–6392 (2018)

    Google Scholar 

  30. Tanaka, K., Mukaigawa, Y., Kubo, H., Matsushita, Y., Yagi, Y.: Recovering transparent shape from time-of-flight distortion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4387–4395 (2016)

    Google Scholar 

  31. Tieleman, T., Hinton, G.: Lecture 6.5-RmsProp: divide the gradient by a running average of its recent magnitude. COURSERA: Neural Netw. Mach. Learn. 4(2), 26–31 (2012)

    Google Scholar 

  32. Torres-Gómez, A., Mayol-Cuevas, W.: Recognition and reconstruction of transparent objects for augmented reality. In: 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 129–134. IEEE (2014)

    Google Scholar 

  33. Wang, T., He, X., Barnes, N.: Glass object localization by joint inference of boundary and depth. In: 21st International Conference on Pattern Recognition (ICPR), pp. 3783–3786. IEEE (2012)

    Google Scholar 

Download references

Acknowledgement

This research was supported by the MSIP (Ministry of Science, ICT and Future Planning), Korea, under the University Information Technology Research Center support program (IITP-2016-R2718-16-0014) supervised by the IITP, by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the MSIP (NRF-2016R1A2B4016236), and also by the MIST(Ministry of Science and ICT), Korea, under the “ICT Consilience Creative Program” (IITP-2018-2017-0-01015) supervised by the IITP(Institute for Information & communications Technology Promotion).

Author information

Authors and Affiliations

  1. School of Integrated Technology, Yonsei University, Incheon, South Korea

    Seongjong Song & Hyunjung Shim

Authors
  1. Seongjong Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hyunjung Shim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hyunjung Shim .

Editor information

Editors and Affiliations

  1. IIIT Hyderabad, Hyderabad, India

    C.V. Jawahar

  2. ANU, Canberra, ACT, Australia

    Hongdong Li

  3. Simon Fraser University, Burnaby, BC, Canada

    Greg Mori

  4. ETH Zurich, Zurich, Zürich, Switzerland

    Konrad Schindler

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Song, S., Shim, H. (2019). Depth Reconstruction of Translucent Objects from a Single Time-of-Flight Camera Using Deep Residual Networks. In: Jawahar, C., Li, H., Mori, G., Schindler, K. (eds) Computer Vision – ACCV 2018. ACCV 2018. Lecture Notes in Computer Science(), vol 11365. Springer, Cham. https://doi.org/10.1007/978-3-030-20873-8_41

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-20873-8_41

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-20872-1

  • Online ISBN: 978-3-030-20873-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation